‘Robots.txt’ needs a major upgrade for AI
For a long time we have existed in a weird world where if as a human you want to use a website you agreed to a long terms-of-service either explicitly or implicitly… But if you were a robot accessing a website the rules of the road were pretty basic: 1. There was some usual blanket…