- 49 minfrom launch to ClaudeBot’s first verified requestCensus bot log, 29 Oct 2026
- 190verified ClaudeBot requests since launchCensus bot log, 29 Oct to 8 Nov 2026
- 45web pages and Markdown copies it fetched, each onceCensus bot log, to 3 Nov 2026
- 0verified ClaudeBot requests after 16:54 UTC on 4 NovChecked to 13:30 UTC, 8 Nov 2026
Short answerClaudeBot is Anthropic’s crawler for web content that could help train its models. On our new site it arrived 49 minutes after launch, fetched 45 pages and Markdown copies in five days, then went quiet: no verified request since 16:54 UTC on 4 November.
What is ClaudeBot?
FindingAnthropic runs 3 agents with their own robots.txt names, and it describes only ClaudeBot as a training crawler.
| Agent | What Anthropic says it does | Robots.txt name |
|---|---|---|
| ClaudeBot | Collects web content that could potentially contribute to training Anthropic’s models | ClaudeBot |
| Claude-User | Fetches pages when people ask Claude questions | Claude-User |
| Claude-SearchBot | Crawls to improve the quality of search results for Claude users | Claude-SearchBot |
So blocking ClaudeBot is a decision about training, not about whether Claude can open your page for someone.
What ClaudeBot fetched on our site
Finding101 of its first 166 verified requests were for robots.txt and the sitemap index, and every page it wanted it fetched exactly once.
| What it requested | Requests | Different URLs |
|---|---|---|
| Sitemap index (wp-sitemap.xml) | 64 | 1 |
| robots.txt | 37 | 1 |
| Web pages (HTML) | 23 | 23 |
| Markdown copies of pages (.md) | 22 | 22 |
| RSS feeds | 7 | 7 |
| Category and tag archives | 5 | 5 |
| Child sitemaps | 4 | 4 |
| Error responses (two 404s, one 405) | 3 | 3 |
| llms.txt | 1 | 1 |
| Total | 166 | 67 |
- It checks the sitemap index about every two hours, but opened our post sitemap only once, on launch day.
- It stopped picking up new pages after 21:34 UTC on 1 November, and never requested our home page.
- Then it stopped coming. 24 more robots.txt and sitemap index requests to 16:54 UTC on 4 November took its total to 190. None since, to 13:30 UTC on 8 November, while Googlebot, Bingbot and Applebot kept visiting. We will update this page if it returns.
How to verify ClaudeBot in your logs
FindingIn our first five days, 2 requests that called themselves ClaudeBot came from outside Anthropic’s published addresses, because a user agent is just text.
- Copy the request’s IP from your server or CDN log (behind a CDN, the visitor IP it passes on).
- Open Anthropic’s published list at claude.com/crawling/bots.json.
- Inside a listed range: Anthropic. Outside: unverified, whatever the user agent says.
Our two fakes came from our own testing addresses and are left out of every count. The free crawler log check skill on our Tools page runs these checks for you.
How to block or slow ClaudeBot
FindingA ClaudeBot block covers 1 of Anthropic’s 3 agents; Claude-User and Claude-SearchBot need their own lines.
User-agent: ClaudeBot
Disallow: /
Put it in the robots.txt of each domain and subdomain. To slow ClaudeBot instead, Anthropic supports Crawl-delay: 1 under User-agent: ClaudeBot, and warns that blocking its IP addresses may not reliably opt you out. Our robots.txt allows every crawler, so we cannot test whether ClaudeBot obeys a block.
What about GPTBot, OAI-SearchBot and PerplexityBot?
| Crawler (company) | Verified requests | Notes |
|---|---|---|
| ClaudeBot (Anthropic) | 166 | First at 11:10 UTC on 29 October |
| OAI-SearchBot (OpenAI) | 1 | robots.txt only, at 06:21 UTC on 3 November |
| GPTBot (OpenAI) | 0 | 2 unverified requests, from our own testing addresses |
| PerplexityBot (Perplexity) | 0 | No request of any kind |
The day after this five-day window, GPTBot made its first verified visit, from 08:42 to 08:44 UTC on 4 November: 156 requests from addresses on OpenAI’s published GPTBot list, including 27 published pages, 26 Markdown copies and llms.txt once.
What to do
- Check the IP before you trust the name. 2 of the ClaudeBot requests in our first five days were not ClaudeBot.
- Decide training and answers separately. Blocking ClaudeBot alone leaves Claude-User and Claude-SearchBot free to open your pages.
- Do not count on a recrawl. ClaudeBot fetched each page it requested once, and none of the pages we published from 2 November onward.
Next: the CensusIndex™ checks whether leading sites can be crawled by AI tools; CensusRank™ and our Editions count which brands and firms AI answers name. Get each result by email.
On a brand-new site, ClaudeBot made its first verified request 49 minutes after launch and 166 verified requests in five days, 101 of them for robots.txt and the sitemap index (29 October to 3 November 2026).
Source: AI Visibility Census, “What is ClaudeBot?”, https://aivisibilitycensus.com/what-is-claudebot/ (CC BY 4.0)Frequently asked questions
Is ClaudeBot a legitimate crawler?
Yes, when the request comes from an address on Anthropic’s published list. On our site every verified ClaudeBot request was for a public URL.
How fast does ClaudeBot find a new website?
On ours, 49 minutes after the home page went live, before we had built any links to it. One data point; more launch numbers are on our New site crawl and indexing statistics page.
Does ClaudeBot read llms.txt?
It fetched ours once, at 15:38 UTC on launch day, and not again as of 8 November 2026.
Does blocking ClaudeBot stop Claude from reading my site?
Not on its own. Claude-User and Claude-SearchBot have their own robots.txt names, so each has to be blocked by name if that is what you want.
ClaudeBot’s first day, minute by minute
| Time (UTC, 29 Oct) | What ClaudeBot requested |
|---|---|
| 10:21 | Site goes live (home page published) |
| 11:10 | First request: the comments feed, then the main feed |
| 11:12 to 11:13 | Two default WordPress addresses, /uncategorized/hello-world/ and /sample-page/ (both 404), and xmlrpc.php (405) |
| 12:48 | robots.txt and the sitemap index, for the first time |
| 13:43 to 13:44 | The post, page and category sitemaps |
| 14:43 to 14:47 | 12 pages, including Methodology, Editions, About and Experiments |
| 15:37 to 15:39 | Markdown copies of 10 of those pages |
| 15:38 | llms.txt (its only request for it) |
By the end of launch day it had made 47 verified requests, including 12 for Markdown copies. On 1 November it came back for 8 more web pages (19:55 to 20:43 UTC) and their 8 Markdown copies (20:42 to 21:33 UTC), along with tag archives and feeds. The Markdown copies were all requested by their .md address, not through content negotiation. The llms.txt picture across all crawlers is in our Lab note 002, “Do AI crawlers read llms.txt?”
How we counted, and limits
Our server logs every crawler request and checks its IP address against the lists that Google, Microsoft, Apple, Anthropic, OpenAI and Perplexity publish. Only ClaudeBot requests from Anthropic’s published addresses are counted; every one used the user agent Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com). How we count is in the methodology, and every test on this site is on our experiments page.
Limits: a fetch is not use. Our log shows what ClaudeBot downloaded and when, not whether any Claude answer draws on it. This is one new site with a few dozen pages, so treat the counts as a dated example, not an average. Requests from Claude-User are left out: our log does not verify that agent, and our own testing uses it.
Data on this page is free to reuse under CC BY 4.0; please credit the AI Visibility Census with a link to this page. Spotted an error? Email hello@aivisibilitycensus.com.
Sources: Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?; Anthropic: published crawler IP addresses (bots.json); AI Visibility Census bot log, 29 October to 8 November 2026.
Published by the AI Visibility Census.
Newsletter
Get the next Census by email
Each new edition, list and experiment result, sent when it publishes. About one email a week. One click to leave.