The AI Crawler & Bot Directory: Telemetry, Ingestion Mechanics & Robots.txt Directives
The AI Crawler & Bot Directory: Telemetry, Ingestion Mechanics & Directives
An empirical reference guide to commercial LLM user-agents, based on verified edge logs across 110 continuous production days. Learn which bots drive live search citations versus offline training scrapes.
The Great Decoupling: Live Inference vs. Model Pre-Training
For over two decades, search engine indexing operated on a single paradigm: crawlers ingested web pages into an inverted index to return ranked URL lists for human navigation. In 2026, web infrastructure split into two distinct bot categories with opposing economic models:
1. Live Runtime Inference Crawlers: Triggered synchronously when a human user submits a query to ChatGPT Search or Claude Artifacts. These crawlers ingest live HTML to verify factual claims and attach attributed footnote links. Blocking them removes your site from real-time AI answers.
2. Offline Batch Pre-Training Scrapers: Ingest millions of documents asynchronously to build training corpora for future foundational model weights. They do not generate live search referrals.
“Blocking all AI bots blindly in robots.txt cuts off live referral traffic from active search sessions. Allowing all bots without rate-limiting consumes server bandwidth without yielding user attribution.”
Master Bot Comparison: Observed Production Telemetry
Across our 110-day dataset ($N=13,971$), AI search crawlers drove 43.6% of all production edge requests (6,091 total bot requests), outpacing traditional Googlebot (1,012 requests) by 4.85 to 1. Below is the verified breakdown across commercial crawlers:
| Bot User-Agent | Parent Lab | Operational Role | Trigger Type | Observed Hits (N=13,971) | Executes JS? | Recommendation |
|---|---|---|---|---|---|---|
| ChatGPT-User | OpenAI | Live Search Grounding | Synchronous User Prompt | 2,970 (46.8%) | No (SSR HTML) | ALLOW FOR CITATIONS |
| GPTBot | OpenAI | Base Model Training | Scheduled Batch Sweep | 0 (Dormant) | No | DISALLOW (OPTIONAL) |
| Bytespider | ByteDance | TikTok / Doubao LLM Index | Continuous High-Frequency | 1,827 (28.8%) | No | RATE LIMIT / BLOCK |
| ClaudeBot | Anthropic | Claude Retrieval & Training | Periodic Directory Crawl | 164 (2.6%) | No | ALLOW (ROOT LINKS) |
| Googlebot | Traditional Inverted Index | Algorithmic Graph Traversal | 1,012 (15.9%) | Yes (WRS Headless) | STANDARD BASELINE |
Core Architectural Takeaway: OpenAI’s training bot (GPTBot) logged zero visits throughout our entire tracking window, while its live search counterpart (ChatGPT-User) logged 2,970 visits. Enterprise teams can safely disallow GPTBot to protect IP without reducing live ChatGPT Search citations.
Detailed Crawler Profiles & Directives
Select a specific user-agent below to view its network parameters, verified IP/ASN ranges, crawl patterns, and copy-paste configuration directives:
ChatGPT-User
OpenAI’s real-time verification fetcher. Triggered when users ask ChatGPT questions requiring live web browsing. Your primary driver of generative citations.
GPTBot
OpenAI’s foundational model scraper. Gathers web datasets for future base model training. Operates in batch sweeps and does not power live search answers.
Bytespider
ByteDance’s high-frequency crawler for TikTok search and Doubao LLM grounding. Logged nearly double the volume of Googlebot without referral traffic.
ClaudeBot
Anthropic’s web crawler for Claude models. Displays a heavy discovery bias toward root domains (66.7% of hits on root) before evaluating deep links.
Googlebot
Google’s traditional search indexer. Provides the baseline benchmark to evaluate how AI crawlers compare to classic web indexing.
The 2026 Global Benchmark
Read the complete empirical study of 13,971 production sessions, mathematical formulas (LCV, CIR, RFI), and the 4-tier syntactic chunking schema.
Official Machine-Readable IP Feeds & Endpoints
To prevent scrapers from impersonating legitimate search agents, network engineers can cross-reference inbound requests against the official published IP endpoints maintained by each provider:
| Crawler Token | Provider | Official Published IP Feed | Verification Standard |
|---|---|---|---|
ChatGPT-User | OpenAI | openai.com/chatgpt-user.json | Published CIDR matching + rDNS (*.openai.com) |
GPTBot | OpenAI | openai.com/gptbot.json | Published CIDR matching + rDNS (*.openai.com) |
OAI-SearchBot | OpenAI | openai.com/searchbot.json | Published CIDR matching + rDNS (*.openai.com) |
ClaudeBot | Anthropic | claude.com/crawling/bots.json | Shared cluster CIDR list + robots.txt compliance |
Googlebot | googlebot.json (Google API) | Forward-Confirmed Reverse DNS (FCrDNS) | |
Bytespider | ByteDance | No public IP feed published | Edge WAF user-agent filtering & ASN check |
Frequently Asked Questions
Can I allow ChatGPT Search without letting OpenAI scrape my site for base model training?
Yes. OpenAI operates separate crawler tokens. You can disallow GPTBot in your robots.txt file to restrict training corpus ingestion while allowing ChatGPT-User to participate in live search queries and earn footnote citations.
Why does Bytespider consume more server bandwidth than Googlebot?
ByteDance uses high-concurrency recursive crawlers across multiple cloud providers without publishing centralized rate-control standards. In our 110-day benchmark, Bytespider generated 1,827 hits compared to Googlebot’s 1,012 hits, often running 15 to 40 requests per minute during batch ingestion cycles.
How can sysadmins distinguish legitimate AI bots from malicious scrapers?
Never whitelist crawlers solely by user-agent string. For OpenAI and Anthropic, match client IP addresses against their published JSON endpoints. For Googlebot, execute Forward-Confirmed Reverse DNS (FCrDNS) lookups to ensure the hostname resolves back to official Google domains.
Audit Your Server Logs for AI Crawler Ingestion
Discover which AI models are scraping your proprietary documentation without citing your brand. Zelitho measures your Live Citation Velocity and Crawl-to-Inference Ratio.
Explore Zelitho AI Growth Plans →