C++ and Java. For anyone writing ... If you, or your company, has plans to write or has written a crawler which parses robots.txt files for directives looking for important information (not just) ...