--- title: "Do AI crawlers read llms.txt? What three days of server logs show" url: "https://aivisibilitycensus.com/do-ai-crawlers-read-llms-txt/" description: "Lab note 002. On a brand-new site, verified Googlebot, GoogleOther and ClaudeBot requested llms.txt or llms-full.txt 10 times in three days. Bingbot and Applebot never asked, and no verified request came from an OpenAI or Perplexity crawler. Markdown page copies drew 77 requests." publisher: "AI Visibility Census" date_published: "2026-11-01" date_modified: "2026-11-01" --- # Do AI crawlers read llms.txt? What three days of server logs show **Short answer:** some do. In the first three days of our brand-new site, three crawlers checked against their company’s published IP ranges asked for llms.txt or llms-full.txt: Googlebot 5 times, GoogleOther 4 times and ClaudeBot once. Bingbot and Applebot never asked, and no verified request came from an OpenAI or Perplexity crawler. The same crawlers fetched our Markdown page copies 77 times, so the files llms.txt points to got far more attention than llms.txt itself. ## What is llms.txt? llms.txt is a Markdown file at the root of a site: a title, a short summary of the site and lists of links to its key pages, each with an optional note. It was proposed on 3 September 2024 as a way to give language models a short, clean map of a site ([llmstxt.org](https://llmstxt.org/)). Many sites also publish llms-full.txt, a longer file with the full text of their pages; that is a common convention, not part of the proposal. llms.txt itself is a proposal, not a standard that search engines or AI companies have signed up to, so the useful question is whether their crawlers fetch it at all. Our site has both files, generated automatically from every published page, plus a Markdown copy of every page at the same address with .md on the end. The full setup is in [Lab note 001](https://aivisibilitycensus.com/lab-note-001-new-domain-launch/). ## What our logs show The window is our first logged request at 10:23 UTC on 29 October 2026 to 13:00 UTC on 1 November 2026, about 75 hours. The table counts only requests whose IP address matched the crawler company’s published ranges. | Crawler (company) | Verified requests | llms.txt | llms-full.txt | Markdown copies (.md) | | --- | --- | --- | --- | --- | | Googlebot (Google) | 230 | 3 | 2 | 42 | | ClaudeBot (Anthropic) | 101 | 1 | 0 | 14 | | GoogleOther (Google) | 72 | 2 | 2 | 21 | | Bingbot (Microsoft) | 13 | 0 | 0 | 0 | | Applebot (Apple) | 12 | 0 | 0 | 0 | | Google-InspectionTool (Google) | 12 | 0 | 0 | 0 | | OpenAI crawlers | 0 | 0 | 0 | 0 | | PerplexityBot | 0 | 0 | 0 | 0 | | **Total** | **440** | **6** | **4** | **77** | *Source: AI Visibility Census bot log, 29 October 10:23 UTC to 1 November 13:00 UTC 2026. Requests over http and https are counted together. Ten requests that used a crawler’s name (Googlebot 6, ClaudeBot 2, GPTBot 2) came from addresses outside that company’s published ranges and are excluded. They came from the same addresses as our own testing.* That is 10 llms file requests out of 440 verified crawler requests, about 2%. Markdown copies made up 77, or 17.5%. ClaudeBot was the first verified crawler to ask for llms.txt, at 15:38 UTC on 29 October, about five hours after launch. GoogleOther followed at 21:07 UTC the same evening, and Googlebot at 18:45 UTC on 30 October. ## Crawlers found the Markdown copies before they read llms.txt All three crawlers that read llms.txt had already fetched a Markdown copy first. ClaudeBot’s first .md request came at 15:37 UTC, about a minute before its llms.txt request. GoogleOther fetched Markdown from 17:59 UTC and llms.txt at 21:07 UTC. Googlebot fetched Markdown from 21:27 UTC on 29 October, about 21 hours before it asked for llms.txt. So llms.txt was not how they found the Markdown pages. Every HTML page on our site also points to its Markdown copy with a `Link: rel="alternate"; type="text/markdown"` header, and that is one route they could have used. Our log shows what was fetched and when, not why. ## Bing was told about llms.txt and never fetched it Every time we publish or update a page, our site submits the page, its Markdown copy and llms.txt to IndexNow, which passes them to Bing and the other engines that take part. llms.txt was in dozens of those submissions, each answered with response code 200. In the same 75 hours Bingbot made 13 verified requests, none for llms.txt, llms-full.txt or a Markdown copy. ## Should you create an llms.txt file? On this evidence, it costs little and some crawlers do fetch it. Two cautions. A fetch is not use: our log shows a crawler downloaded the file, not that any AI answer drew on it. And on our site the Markdown copies of pages drew about eight times as many verified requests as the llms files, so if you can only do one, clean Markdown versions of your pages look like the bigger draw. This is one new site over three days. Treat it as a dated example, not an average. In Lab note 001 we predicted that no verified AI crawler would request llms.txt in the first 30 days. ClaudeBot proved that wrong within five hours. ## What we are testing next We have registered a follow-up for the rest of November, LL-001, on our [experiments page](https://aivisibilitycensus.com/experiments/#ll-001). It asks whether verified AI crawlers are any more likely, within an hour of reading llms.txt or llms-full.txt than within an hour of reading robots.txt or our sitemap, to fetch a page they had not requested since our log began on 29 October. The result is due on 1 December 2026, whatever it shows. Whether a site serves llms.txt and Markdown is one of the things the [CensusIndex™](https://aivisibilitycensus.com/census-index/) looks at in its monthly check of how ready the leading sites in each software category are for AI search. Whether AI answers then name a brand is what the [Census100™](https://aivisibilitycensus.com/census-100/) counts for software buyers, and our [Editions](https://aivisibilitycensus.com/editions/) count for professional services firms, starting with Ireland. To check crawler visits on your own server logs, the free AI crawler log check skill is on our [Tools page](https://aivisibilitycensus.com/tools/). ## Frequently asked questions ### Does Google read llms.txt? Google’s crawlers fetched it on our site: Googlebot asked for llms.txt or llms-full.txt 5 times and GoogleOther 4 times in our first three days, all from Google’s published IP ranges. Fetching a file does not show that Google uses it in search results or AI answers. ### Does ChatGPT read llms.txt? We cannot tell yet. In the first three days, no request identifying as an OpenAI crawler came from OpenAI’s published IP ranges, for llms.txt or anything else. ### Where does llms.txt go? At the root of the domain, so it loads at yoursite.com/llms.txt, the same place as robots.txt. ### How can I see if AI crawlers fetch my llms.txt? Search your server or CDN logs for requests to /llms.txt, then check each request’s IP address against the ranges the crawler’s company publishes. A user agent name on its own proves nothing, because any client can send it. ## About this article Lab note 002 in our series on a brand-new domain. Counts come from our own bot log, which checks each request against the IP ranges that Google, Microsoft, Apple, Anthropic, OpenAI and Perplexity publish. Requests from Claude-User, Anthropic’s user-triggered fetcher, came from our own testing and are left out, as in Lab note 001. How we count is set out in the [methodology](https://aivisibilitycensus.com/methodology/), and the launch-day numbers, with a CSV, are on the [new site crawl and indexing statistics](https://aivisibilitycensus.com/new-site-crawl-indexing-statistics/) page. Data on this page is free to reuse under [CC BY 4.0](https://aivisibilitycensus.com/data-license/). ## Sources - [llmstxt.org: The /llms.txt file](https://llmstxt.org/), the proposal and format. - AI Visibility Census bot log and IndexNow log, 29 October to 1 November 2026.