Google has outlined plans to turn robots exclusion protocol (REP) — better known as robots.txt — into an internet standard after 25 years.

C++ and Java. For anyone writing ... If you, or your company, has plans to write or has written a crawler which parses robots.txt files for directives looking for important information (not just) ...