GPTBot: User-Agent Signature, Scraping Behavior & Robots.txt Directives
GPTBot: User-Agent Signature, Ingestion Behavior & Robots.txt Rules
GPTBot is OpenAI’s large-scale web scraper used to collect training datasets for future base models. It operates in asynchronous batch sweeps and does not power live real-time ChatGPT search responses.
1. Technical User-Agent Identification
GPTBot identifies itself through an explicit user-agent header. System administrators can match it at the web server layer using this signature:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.0; +https://openai.com/gptbotNetwork Origin & Validation: When active during training collection cycles, GPTBot operates across OpenAI’s egress IP ranges:
- Official IP Ranges JSON:https://openai.com/gptbot.json (machine-readable CIDR blocks published by OpenAI).
- Official Documentation:OpenAI Bots Reference Guide.
- Autonomous System Numbers (ASNs):
AS16509 (Amazon AWS)andAS14618 (Amazon.com). - Reverse DNS Pattern: Hostnames resolve backwards to
*.openai.com. - Crawl Profile: High concurrency, bulk HTML scraping without live user prompt context.
2. Zelitho Production Telemetry: The 0-Hit Phenomenon
Across our 110-day production benchmark tracking window ($N=13,971$ total edge sessions), GPTBot recorded zero requests:
Why GPTBot Was Dormant: Foundational model pre-training does not occur continuously. OpenAI gathers training corpora in massive, scheduled batch sweeps separated by months. Live runtime verification, however, runs every second of every day through ChatGPT-User.
3. The Myth of Blocking GPTBot
A frequent misconception among engineering and marketing leadership is that disallowing GPTBot prevents your website from being referenced or cited in ChatGPT.
The Technical Reality: Disallowing GPTBot only prevents your content from being used to train future foundational weights (e.g., GPT-5). It has zero effect on ChatGPT Search citations, which are driven exclusively by ChatGPT-User.
The Commercial Strategy: For B2B companies with proprietary research, technical documentation, or original data, the optimal economic setup is:
- Disallow
GPTBot: Prevents OpenAI from training future base models on your content without attribution or compensation. - Allow
ChatGPT-User: Ensures your site is actively retrieved and cited with attributed footnote links when active buyers search.
4. Recommended robots.txt Configuration
To decouple model training ingestion from live search citation traffic, configure your robots.txt file with these distinct rules:
# Block background model pre-training User-agent: GPTBot Disallow: / # Allow live runtime search verification and footnote citations User-agent: ChatGPT-User Allow: /Frequently Asked Questions: GPTBot
Does disallowing GPTBot block our content from appearing in ChatGPT Search answers?
No. Disallowing GPTBot only excludes your pages from future foundational model pre-training. Real-time search queries in ChatGPT rely on ChatGPT-User and OAI-SearchBot. Allowing those agents ensures your content remains eligible for live citations.
Where does OpenAI publish the official IP addresses for GPTBot?
OpenAI publishes all active CIDR address blocks in a machine-readable JSON feed at https://openai.com/gptbot.json. System administrators can pull this feed to configure network firewall rules.
Can we allow GPTBot access to select directories while blocking others?
Yes. Standard robots.txt path syntax applies. You can allow public case studies and blog posts while disallowing API endpoints, internal documentation, or user account portals.
Source Data: This telemetry was collected and analyzed as part of The 2026 Global AI Search & LLM Retrieval Benchmark (13,971 Production Edge Sessions Analyzed by Zelitho Research).
Audit Your Crawl-to-Inference Ratio (CIR)
Find out how much training scraping your site tolerates for every live citation earned. Zelitho audits edge traffic and configures defensive robots.txt policies.
Explore Zelitho AI Growth Plans →