I built an AI assistant for our docs at Handsontable, and then Grok found it. 4.5 million edge requests and 87,000 page renders in only 4 hours, all from xAI's Colossus data center in Memphis. I caught it during the scan and pushed a fix to redirect traffic to our MCP, then spent a week analyzing the logs and learning how crawlers work, what they look for, and what they ignore.
I built an AI assistant for our docs at Handsontable, and then Grok found it. 4.5 million edge requests and 87,000 page renders in only 4 hours, all from xAI's Colossus data center in Memphis. I caught it during the scan and pushed a fix to redirect traffic to our MCP, then spent a week analyzing the logs and learning how crawlers work, what they look for, and what they ignore.
Another example of uselessness of llms.txt.