Google has outlined plans to turn robots exclusion protocol (REP) — better known as robots.txt — into an internet standard after 25 years.