---
title: "CensusIndexBot: our crawler"
url: "https://aivisibilitycensus.com/index-bot/"
description: "What the AI Visibility Census crawler requests, how to recognize it and how to opt out."
publisher: "AI Visibility Census"
date_published: "2026-10-30"
date_modified: "2026-10-30"
---

# CensusIndexBot: our crawler

**In short:** CensusIndexBot is the AI Visibility Census crawler. Once a month it reads a few public files on each site in a CensusIndex™ category, so we can report how ready that category is for AI search. It identifies itself, waits between requests and follows robots.txt. To opt out, block it in robots.txt.

## How to recognize it

Every request carries this user agent:

```
CensusIndexBot/0.1 (+https://aivisibilitycensus.com/index-bot/)
```

It runs on GitHub’s hosted servers, so its IP addresses change from run to run and we cannot publish a fixed list. Use the user agent to identify it.

## What it requests

For each site, in one visit a month:

- robots.txt
- the home page, once as plain HTML and once loaded in a headless browser, which also loads the page’s own scripts, styles and images
- the home page again with an Accept header asking for markdown, to see whether a markdown version is offered
- the XML sitemap and up to three of its child sitemaps
- /llms.txt

That is about ten requests to your files, with at least two seconds between them, plus whatever your home page loads when a browser opens it. It does not crawl other pages, fill in forms, log in, or try to get past bot protection. If a firewall blocks it, we record that the site did not load for an identified bot and move on.

## How to opt out

Add this to your robots.txt:

```
User-agent: CensusIndexBot
Disallow: /
```

It will still read robots.txt, since that is how it learns the rule, and nothing else. If you want your site left out of the Index entirely, [tell us](https://aivisibilitycensus.com/contact/) and we will remove it from future runs and from published results.

## What we do with the results

We publish category totals, such as how many sites in a category let AI search bots crawl their home page. Each site’s own results page is hidden from search engines unless the site’s owner asks us to make it public. We do not store page content beyond what each check needs, and we do not use it to train anything. Every published Index report lists the checks, what each one is based on, and how sites were chosen.

## Something wrong?

If the bot misbehaves, sends too many requests or reports something about your site that is wrong, [contact us](https://aivisibilitycensus.com/contact/) and include the date and your domain.