Search Engine Land » Platforms » Google » Google opens the source for its robots.txt parser in Java and testing framework ... which parses robots.txt files for directives looking for important ...