What is ClaudeBot? What Anthropic’s crawler fetched on our brand-new site

ClaudeBot is Anthropic’s web crawler. On our brand-new site it made its first verified visit 49 minutes after launch and 166 verified requests in five days: 101 for robots.txt and the sitemap index, 23 web pages and 22 Markdown copies, and llms.txt once. How to verify it and how to block it.

On this page
  1. What is ClaudeBot?
  2. What ClaudeBot fetched on our site
  3. How to verify ClaudeBot in your logs
  4. How to block or slow ClaudeBot
  5. What to do
  6. Frequently asked questions
  7. Get the next Census by email
  • 49 minfrom launch to ClaudeBot’s first verified requestCensus bot log, 29 Oct 2026
  • 190verified ClaudeBot requests since launchCensus bot log, 29 Oct to 8 Nov 2026
  • 45web pages and Markdown copies it fetched, each onceCensus bot log, to 3 Nov 2026
  • 0verified ClaudeBot requests after 16:54 UTC on 4 NovChecked to 13:30 UTC, 8 Nov 2026

Short answerClaudeBot is Anthropic’s crawler for web content that could help train its models. On our new site it arrived 49 minutes after launch, fetched 45 pages and Markdown copies in five days, then went quiet: no verified request since 16:54 UTC on 4 November.

What is ClaudeBot?

FindingAnthropic runs 3 agents with their own robots.txt names, and it describes only ClaudeBot as a training crawler.

AgentWhat Anthropic says it doesRobots.txt name
ClaudeBotCollects web content that could potentially contribute to training Anthropic’s modelsClaudeBot
Claude-UserFetches pages when people ask Claude questionsClaude-User
Claude-SearchBotCrawls to improve the quality of search results for Claude usersClaude-SearchBot
Source: Anthropic help center, “Does Anthropic crawl data from the web, and how can site owners block the crawler?” (checked 8 November 2026).

So blocking ClaudeBot is a decision about training, not about whether Claude can open your page for someone.

What ClaudeBot fetched on our site

Finding101 of its first 166 verified requests were for robots.txt and the sitemap index, and every page it wanted it fetched exactly once.

What it requestedRequestsDifferent URLs
Sitemap index (wp-sitemap.xml)641
robots.txt371
Web pages (HTML)2323
Markdown copies of pages (.md)2222
RSS feeds77
Category and tag archives55
Child sitemaps44
Error responses (two 404s, one 405)33
llms.txt11
Total16667
Source: AI Visibility Census bot log, 29 October 2026 to 13:30 UTC on 3 November 2026. All 166 requests came from two IP addresses, 216.73.217.69 and 216.73.217.32, both inside the 216.73.216.0/22 block in Anthropic’s published list. Free to reuse under CC BY 4.0.
  • It checks the sitemap index about every two hours, but opened our post sitemap only once, on launch day.
  • It stopped picking up new pages after 21:34 UTC on 1 November, and never requested our home page.
  • Then it stopped coming. 24 more robots.txt and sitemap index requests to 16:54 UTC on 4 November took its total to 190. None since, to 13:30 UTC on 8 November, while Googlebot, Bingbot and Applebot kept visiting. We will update this page if it returns.

How to verify ClaudeBot in your logs

FindingIn our first five days, 2 requests that called themselves ClaudeBot came from outside Anthropic’s published addresses, because a user agent is just text.

  1. Copy the request’s IP from your server or CDN log (behind a CDN, the visitor IP it passes on).
  2. Open Anthropic’s published list at claude.com/crawling/bots.json.
  3. Inside a listed range: Anthropic. Outside: unverified, whatever the user agent says.

Our two fakes came from our own testing addresses and are left out of every count. The free crawler log check skill on our Tools page runs these checks for you.

How to block or slow ClaudeBot

FindingA ClaudeBot block covers 1 of Anthropic’s 3 agents; Claude-User and Claude-SearchBot need their own lines.

User-agent: ClaudeBot
Disallow: /

Put it in the robots.txt of each domain and subdomain. To slow ClaudeBot instead, Anthropic supports Crawl-delay: 1 under User-agent: ClaudeBot, and warns that blocking its IP addresses may not reliably opt you out. Our robots.txt allows every crawler, so we cannot test whether ClaudeBot obeys a block.

What about GPTBot, OAI-SearchBot and PerplexityBot?
Crawler (company)Verified requestsNotes
ClaudeBot (Anthropic)166First at 11:10 UTC on 29 October
OAI-SearchBot (OpenAI)1robots.txt only, at 06:21 UTC on 3 November
GPTBot (OpenAI)02 unverified requests, from our own testing addresses
PerplexityBot (Perplexity)0No request of any kind
Source: AI Visibility Census bot log, 29 October 2026 to 13:30 UTC on 3 November 2026, checked against the IP lists OpenAI and Perplexity publish.

The day after this five-day window, GPTBot made its first verified visit, from 08:42 to 08:44 UTC on 4 November: 156 requests from addresses on OpenAI’s published GPTBot list, including 27 published pages, 26 Markdown copies and llms.txt once.

What to do

  1. Check the IP before you trust the name. 2 of the ClaudeBot requests in our first five days were not ClaudeBot.
  2. Decide training and answers separately. Blocking ClaudeBot alone leaves Claude-User and Claude-SearchBot free to open your pages.
  3. Do not count on a recrawl. ClaudeBot fetched each page it requested once, and none of the pages we published from 2 November onward.

Next: the CensusIndex™ checks whether leading sites can be crawled by AI tools; CensusRank™ and our Editions count which brands and firms AI answers name. Get each result by email.

Cite this

On a brand-new site, ClaudeBot made its first verified request 49 minutes after launch and 166 verified requests in five days, 101 of them for robots.txt and the sitemap index (29 October to 3 November 2026).

Source: AI Visibility Census, “What is ClaudeBot?”, https://aivisibilitycensus.com/what-is-claudebot/ (CC BY 4.0)

Frequently asked questions

Is ClaudeBot a legitimate crawler?

Yes, when the request comes from an address on Anthropic’s published list. On our site every verified ClaudeBot request was for a public URL.

How fast does ClaudeBot find a new website?

On ours, 49 minutes after the home page went live, before we had built any links to it. One data point; more launch numbers are on our New site crawl and indexing statistics page.

Does ClaudeBot read llms.txt?

It fetched ours once, at 15:38 UTC on launch day, and not again as of 8 November 2026.

Does blocking ClaudeBot stop Claude from reading my site?

Not on its own. Claude-User and Claude-SearchBot have their own robots.txt names, so each has to be blocked by name if that is what you want.

ClaudeBot’s first day, minute by minute
Time (UTC, 29 Oct)What ClaudeBot requested
10:21Site goes live (home page published)
11:10First request: the comments feed, then the main feed
11:12 to 11:13Two default WordPress addresses, /uncategorized/hello-world/ and /sample-page/ (both 404), and xmlrpc.php (405)
12:48robots.txt and the sitemap index, for the first time
13:43 to 13:44The post, page and category sitemaps
14:43 to 14:4712 pages, including Methodology, Editions, About and Experiments
15:37 to 15:39Markdown copies of 10 of those pages
15:38llms.txt (its only request for it)
Source: AI Visibility Census bot log. The 14:43 to 14:47 count includes the analysis category archive.

By the end of launch day it had made 47 verified requests, including 12 for Markdown copies. On 1 November it came back for 8 more web pages (19:55 to 20:43 UTC) and their 8 Markdown copies (20:42 to 21:33 UTC), along with tag archives and feeds. The Markdown copies were all requested by their .md address, not through content negotiation. The llms.txt picture across all crawlers is in our Lab note 002, “Do AI crawlers read llms.txt?”

How we counted, and limits

Our server logs every crawler request and checks its IP address against the lists that Google, Microsoft, Apple, Anthropic, OpenAI and Perplexity publish. Only ClaudeBot requests from Anthropic’s published addresses are counted; every one used the user agent Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com). How we count is in the methodology, and every test on this site is on our experiments page.

Limits: a fetch is not use. Our log shows what ClaudeBot downloaded and when, not whether any Claude answer draws on it. This is one new site with a few dozen pages, so treat the counts as a dated example, not an average. Requests from Claude-User are left out: our log does not verify that agent, and our own testing uses it.

Data on this page is free to reuse under CC BY 4.0; please credit the AI Visibility Census with a link to this page. Spotted an error? Email hello@aivisibilitycensus.com.

Sources: Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?; Anthropic: published crawler IP addresses (bots.json); AI Visibility Census bot log, 29 October to 8 November 2026.

Published by the AI Visibility Census.

Newsletter

Get the next Census by email

Each new edition, list and experiment result, sent when it publishes. About one email a week. One click to leave.

We send a confirmation email first. If you add a website, we may tell you when your company appears in a Census result. Privacy notice