Tag: AI crawlers
-
Do AI crawlers read llms.txt? What three days of server logs show
Lab note 002. On a brand-new site, verified Googlebot, GoogleOther and ClaudeBot requested llms.txt or llms-full.txt 10 times in three days. Bingbot and Applebot never asked, and no verified request came from an OpenAI or Perplexity crawler. Markdown page copies drew 77 requests.
-
Lab note 001: we launched a brand-new domain today. Here’s everything we set up, and what we predict.
A brand-new domain, registered and launched on 29 October 2026. The full setup, the first crawler visits and our predictions, written down before the results.