The core issue is that rules around robots.txt and donotcrawl are more of a gentleman’s agreement rather than unbreakable ...