--- title: "New site crawl and indexing statistics" url: "https://aivisibilitycensus.com/new-site-crawl-indexing-statistics/" description: "How long Google takes to crawl and index a brand-new site, and which AI crawlers arrive first: IP-verified data from our own launch, free to reuse under CC BY 4.0." publisher: "AI Visibility Census" date_published: "2026-10-30" date_modified: "2026-11-01" --- # New site crawl and indexing statistics **Short answer:** on a brand-new domain with no links, Google indexed every page we checked within 24 hours, but Googlebot did not visit at all until we connected Search Console. After that, pages got their first verified Googlebot visit a median of 7.3 hours after publishing. ClaudeBot was the first AI crawler, 49 minutes after launch. OpenAI and Perplexity had not visited at all by the end of day two. These figures come from aivisibilitycensus.com, which we registered and launched on 29 October 2026 and log in full. This is one site, so read the numbers as a dated, real example, not an average. We update this page as new data comes in. Last updated: 30 October 2026, 16:00 UTC. ## Key numbers | Measure | Result | | --- | --- | | First verified crawler request | Bingbot, 22 minutes after the first pages went live (it fetched our IndexNow key file) | | Googlebot requests before Search Console was connected | None | | First Googlebot request after Search Console was connected | About 1 minute (14:50 to 14:51:48 UTC) | | Median time from publishing to first verified Googlebot visit | 7.3 hours (8 pages, range 3.6 to 13.4) | | Pages confirmed indexed within 24 hours | 5 of 5 checked in Search Console | | First verified AI crawler request | ClaudeBot, 11:10 UTC, 49 minutes after launch (for feeds and pages left over from the default WordPress install) | | First verified AI crawler visit to a Census page | ClaudeBot, 14:45 UTC on launch day | | First verified AI crawler request for llms.txt | ClaudeBot, 15:38 UTC on launch day | | Share of verified ClaudeBot requests for Markdown copies | 22% (14 of 64) | | Requests from OpenAI and Perplexity crawlers | 0 by 16:00 UTC on 30 October | *Source: AI Visibility Census bot log and Google Search Console, 29 and 30 October 2026. [Download the data (CSV)](https://aivisibilitycensus.com/wp-content/uploads/figures/census-new-site-crawl-statistics.csv).* ## How long does Google take to crawl a new site? On our site, Googlebot came only after we connected Search Console and submitted the sitemap at 14:50 UTC. Its first verified request arrived at 14:51:48, for the sitemap. Pages published that morning had waited more than four hours with no Googlebot visit at all. Bingbot and YandexBot, which take part in IndexNow, had already been. Once Google had the sitemap, each page got its first verified Googlebot visit between 3.6 and 13.4 hours after it was published. The one faster page, at 0.6 hours, had a manual indexing request in Search Console, so we leave it out of the median. ![Bar chart of hours from publishing to the first verified Googlebot visit for nine pages, from 0.6 hours after a manual indexing request to 13.4 hours](https://aivisibilitycensus.com/wp-content/uploads/figures/census-googlebot-first-crawl-hours.svg) *Hours from publishing to the first Googlebot request verified against Google’s IP ranges: Content Shape Checker 0.6 (manual indexing request), Tools 3.6, Data license 5.3, Homepage 6.5, Experiments 6.8, About 7.7, Lab note 001 7.9, Methodology 9.5, keyword research article 13.4.* ## How long does Google take to index a new site? At 09:57 UTC on 30 October, Search Console’s URL Inspection showed all five pages we checked as “URL is on Google”: the homepage, the first two articles, the Content Shape Checker and the Tools page. Search Console does not say when a page entered the index, so the accurate claim is “indexed within 24 hours of publishing”, not an exact time. On launch day we submitted the sitemap, sent every page to IndexNow and requested indexing for some URLs. These results show what happens with that basic setup done, not what happens to a site nobody tells Google about. ## Which crawlers visit a new site first? In the first day, Google sent the most crawler requests, followed by Meta, whose 75 requests all arrived within four minutes. ClaudeBot made 55, every one from Anthropic’s published IP range. OpenAI’s and Perplexity’s crawlers made none, and they had still not visited by 16:00 UTC on day two. ![Bar chart of crawler requests by company in the first day: Google 127, Meta 75, Anthropic 55, Ahrefs 45, Yandex 19, Microsoft 7, OpenAI 0, Perplexity 0](https://aivisibilitycensus.com/wp-content/uploads/figures/census-crawler-requests-first-48-hours.svg) *Crawler requests from 10:23 UTC on 29 October to 10:00 UTC on 30 October 2026: Google 127 (Googlebot 58, GoogleOther 67, Google-InspectionTool 2), Meta 75, Anthropic 55, Ahrefs 45, Yandex 19, Microsoft 7, OpenAI 0, Perplexity 0. Spoofed requests and our own test requests excluded.* ## Do AI crawlers read llms.txt and Markdown pages? Some do, sooner than we expected. We had predicted that no verified AI crawler would request our llms.txt in the first 30 days. A verified ClaudeBot requested it at 15:38 UTC on launch day, and Google’s GoogleOther crawler requested it twice that evening. Of 64 verified ClaudeBot requests by 16:00 UTC on day two, 14 (22%) were for the Markdown copies of our pages. Fetching a file is not the same as using it in an answer. Whether any of this leads to citations is what our [running experiments](https://aivisibilitycensus.com/experiments/) measure. ## The setup behind these numbers ![Diagram: publishing a page creates an HTML page, a Markdown copy and an llms.txt entry; the sitemap and IndexNow notify search engines; crawler visits are written to an IP-verified bot log](https://aivisibilitycensus.com/wp-content/uploads/figures/census-new-page-setup-diagram.svg) *WordPress on shared hosting behind Cloudflare, a sitemap read through Search Console, IndexNow on every publish, and a bot log that checks each crawler against its company’s published IP ranges. The full setup is in [Lab note 001](https://aivisibilitycensus.com/lab-note-001-new-domain-launch/).* ## Method and limits - **One site.** Every number is from aivisibilitycensus.com. Other sites, hosts and topics will differ. - **Verified crawlers only.** Where a company publishes crawler IP ranges (Google, Microsoft, OpenAI, Perplexity, Apple, Anthropic), only requests from those ranges count. Requests claiming to be a bot from anywhere else are excluded as spoofed. - **Our own tests are excluded.** Requests we made while testing the site, including every Claude-User request, are left out. - **Times are UTC** and come from the server log and Search Console. ## How to cite this data The data is free to reuse under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), with credit and a link to this page: *AI Visibility Census, “New site crawl and indexing statistics”, aivisibilitycensus.com/new-site-crawl-indexing-statistics/, updated 30 October 2026.* Each chart above has an embed code under it. The [data license](https://aivisibilitycensus.com/data-license/) page has the full terms. ## Update log | Date | Change | | --- | --- | | 30 October 2026 | Page published with data from the first 30 hours. |