ClaudeBot: Crawl Mechanics, Directory Bias & Content Indexation
ClaudeBot: Crawl Mechanics, Directory Bias & Content Indexation
ClaudeBot is Anthropic’s automated crawler. It feeds training corpora and real-time retrieval context into Claude 3.5 Sonnet and future Anthropic reasoning systems.
1. Technical User-Agent Identification
ClaudeBot identifies itself via an official user-agent header containing Anthropic’s contact domain:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)Network Origin & Verification:
- Official IP Ranges JSON:https://claude.com/crawling/bots.json (unified CIDR feed for all Anthropic infrastructure).
- Autonomous System Numbers (ASNs):
AS16509 (Amazon AWS)andAS396982 (Google Cloud). - Reverse DNS Pattern: Legitimate requests resolve backwards to
*.anthropic.comhostnames. - Respects robots.txt: Anthropic strictly adheres to standard robots exclusion standards. Avoid blocking solely by IP so the crawler can parse your disallow directives.
Anthropic Crawler Disambiguation Matrix
Anthropic operates three distinct crawler tokens. Configuring them independently allows website owners to protect training IP while participating in conversational search:
| User-Agent Token | Operational Purpose | Trigger Mechanism | Impact of Disallow |
|---|---|---|---|
ClaudeBot | Model training & knowledge ingestion | Periodic directory crawl sweeps | Excludes site from future Claude training |
Claude-SearchBot | Indexing for Claude web search answers | Algorithmic indexation cycles | Reduces presence in Claude search results |
Claude-User | On-demand prompt retrieval | User asks Claude to inspect a URL | Prevents real-time page inspection |
2. Zelitho Production Telemetry: The Root-Domain Bias
Across our 110-day benchmark tracking window ($N=13,971$ total edge sessions), ClaudeBot generated 164 total visits, beginning with an initial cohort of 114 sessions. Our logs revealed a critical architectural phenomenon:
The First-Click Discovery Bias: Anthropic’s crawler enforces a conservative crawl depth budget. Before we linked research guides directly on the homepage, ClaudeBot visited the root domain 76 times without exploring deep pagination paths. Once we placed direct links on the root homepage, ClaudeBot traversed and indexed our technical subpages within 48 hours.
3. How to Optimize Your Site for ClaudeBot
To ensure Anthropic’s models index and cite your enterprise assets, engineering teams should follow two site architecture rules:
1. Keep High-Value Guides Within 1 Click of Root: Avoid burying research whitepapers, benchmark data, or API documentation behind deep category pagination. Feature primary assets in your homepage navigation or root footer.
2. Clean Semantic Heading Strings: Claude’s retrieval parser prioritizes declarative section headers (<h2> and <h3>) formatted as complete questions or entity definitions over generic 1-word headings.
4. Recommended robots.txt Configuration
Allow ClaudeBot full access to public technical pages while excluding administrative and private directories:
# Allow Anthropic Claude to index public documentation and case studies User-agent: ClaudeBot Allow: / Disallow: /admin/ Disallow: /private/ Disallow: /api/Frequently Asked Questions: ClaudeBot
Where does Anthropic publish official IP addresses for ClaudeBot?
Anthropic hosts an official, public machine-readable JSON feed at https://claude.com/crawling/bots.json. This feed covers all Anthropic crawling infrastructure, including ClaudeBot, Claude-SearchBot, and Claude-User.
Why does ClaudeBot concentrate so heavily on the root domain?
Anthropic uses a conservative link-hop depth limit to prevent crawler traps. In our benchmark, 66.7% of initial visits hit the root homepage. Featuring primary research guides or documentation links directly on your homepage triggers subpage indexation within 48 hours.
Can I block Claude from training while keeping pages eligible for Claude search?
Yes. Disallow ClaudeBot in robots.txt to opt out of foundation model training, while allowing Claude-SearchBot so your content remains indexable for Claude conversational search.
Source Data: This telemetry was collected and analyzed as part of The 2026 Global AI Search & LLM Retrieval Benchmark (13,971 Production Edge Sessions Analyzed by Zelitho Research).
Optimize Your Content for Claude 3.5 Sonnet
Zelitho helps technical B2B brands structure content for multi-agent retrieval and capture persistent citations in Claude, ChatGPT, and Perplexity.
Explore Zelitho AI Growth Plans →