How We Tracked 1,000+ AI Crawls: A Technical Case Study on AI Search Behavior
Most marketers and website owners are currently flying completely blind in the AI search era. They spend thousands of dollars optimizing their pages for AI search engine visibility, but they have absolutely no way to verify if their efforts are working. They look at their Google Analytics 4 dashboards, see standard traffic metrics, and guess. We decided to stop guessing. We built a server-level user-agent tracker to log exactly which AI search engines visit our site, what pages they read, and how often they show up.
in 7 days
in 7 days
extraction bias
source citations
Source: Zelitho production database logs and Bing Webmaster Tools (June 2026). A citation means our URL was visibly shown as a source in an AI-generated answer.
1. The Blindsight Problem: Why GA4 is Blind to AI Search
If you look at your Google Analytics 4 (GA4) or HubSpot reports to track AI bot behavior, you will see nothing. Traditional web analytics tools rely entirely on client-side JavaScript execution. When a human visitor opens your site, the browser loads the page, executes the tracking script, and sends an event payload back to the analytics server.
AI crawlers and search engine indexing spiders do not operate this way. Bots like ChatGPT-User or ClaudeBot fetch your raw HTML directly using server-side GET requests. They do not run the browser DOM, they do not execute JavaScript tag managers, and they do not fire tracking pixels. They scrape the text, extract the entities, and exit. This means that standard analytics dashboards are completely blind to bot traffic. To see the crawlers, you must log the requests at the server level, before the HTML is even rendered to the client.
2. The Architecture: Building a Server-Level Bot Wiretap
We had to intercept the crawlers at the edge. Instead of waiting for a page to load or a script to fire, we built a custom middleware layer that sits directly on our production server. Every single incoming HTTP request (every millisecond, day and night) is intercepted before the server even renders a single block of HTML.
The middleware acts as a digital wiretap. It extracts the raw connection parameters: the User-Agent string (the crawler’s digital fingerprint), the IP Address and ASN (to verify if the bot is actually who it claims to be, or a malicious scraper hiding behind a VPN), the exact URL paths, referrer strings, and timestamps. All of this raw telemetry is routed directly into relational tables in a Supabase Postgres database.
By mapping these incoming user-agent signatures (using strict regex rules to isolate ChatGPT-User, ClaudeBot, Bytespider, and OAI-SearchBot), we created an automated alarm system. For the first time, we had a window into the machine. We could see the AI engines consuming our site in real-time, matching their visits against human behavior.
3. The Crawl Dynamics: The Shallow Spammer vs. The Deep Explorer
After running this setup, we pulled the raw rollup data for the week of June 22, 2026. The database revealed a massive, structural difference in how OpenAI and Anthropic index the web. Here are the raw session and crawl totals for the period:
| Crawler User-Agent | Total Sessions | Total Page Crawls | Unique Pages Crawled |
|---|---|---|---|
| ChatGPT-User | 346 | 374 | 37 |
| ClaudeBot | 11 | 253 | 164 |
| Bytespider | 101 | 136 | 56 |
| Googlebot | 34 | 77 | 31 |
| OAI-SearchBot | 9 | 13 | 12 |
Finding 1: The Heartbeat vs. The Tidal Wave (Crawl Pacing)
When we looked at the daily crawl pacing over the week, we discovered that crawlers have completely different temporal patterns. OpenAI’s ChatGPT-User operates like a steady heartbeat. It crawled our site 72 times on Tuesday, 53 times on Wednesday, 66 times on Thursday, and 61 times on Friday. The numbers remained flat day after day. It is constantly monitoring the site, polling for tiny updates to keep its conversational interface fresh.
Anthropic’s ClaudeBot operates like a tidal wave. It had 5 crawls on Tuesday, jumped to 63 crawls on Wednesday, dropped to 27 on Thursday, and then spiked to a massive 95 crawls on Saturday. Claude does not poll. It sleeps for days, then hits your site with a massive, concentrated wave of requests to copy your entire directory tree.
Finding 2: Shallow Checklists vs. Deep Site Sweeps
The ratio of sessions to page crawls shows the depth of each bot’s interest. ChatGPT-User initiated 346 sessions but only touched 37 unique pages. It is highly transactional: it checks the same resource over and over.
ClaudeBot initiated only 11 sessions, but it crawled 164 unique pages. When Claude shows up, it conducts a massive, vertical sweep of your entire site directory. This has massive implications for site design. If your internal link structure is broken, Claude will get lost and exit during its rare deep sweeps, leaving your site unindexed. OpenAI, however, will simply check the same top pages repeatedly.
(For a companion analysis on how these crawl behaviors translate into actual search citations and daily traffic, read our Bing AI Performance Case Study.)
4. The 149x Extraction Bias: What AI Bots Actually Read
Perhaps the most important discovery was the extreme bias in what page types AI search engines choose to crawl. Traditional search engines crawl everything to map your site. AI engines crawl to extract information. We grouped our pageviews by URL structure: Educational (blogs, guides, tutorials) vs. Commercial (pricing, features, booking pages).
| Page URL Type | Total AI Bot Crawls |
|---|---|
Educational Guides (/blog/* or /guide/*) | 598 crawls |
Commercial Feature Pages (/feature/*) | 4 crawls |
This represents a 149x bias toward educational content. Looking at the specific page hotspots in our logs, we can see exactly which resources the bots targeted:
/blog/ai-agents-in-2025-a-comprehensive-guide/was crawled 133 times./blog/what-is-keyword-stuffing-and-why-should-you-avoid-it/was crawled 45 times./blog/how-googles-helpful-content-update-works/was crawled 37 times./feature/agency-seats/was crawled 5 times./feature/seo-analysis/was crawled 4 times.
AI search crawlers are extraction engines. They ignore marketing copy, landing page slogans, and pricing tables because they cannot easily package them into conversational answers. They look for structured headings, data tables, bullet points, and definition blocks inside educational guides. If you only publish product pages, you are practically invisible to AI search.
5. The Visual Index: AI Bots are Scraping Your Image Assets
When we analyzed the raw session paths, we noticed a pattern we hadn’t anticipated. A single session from Bytespider hit our homepage (/), and then immediately downloaded four image assets in rapid succession: hero-llm-chatgpt.png, hero-llm-claude.png, hero-llm-perplexity.png, and hero-llm-gemini.png.
AI bots are not just reading your text. They are actively indexing your images and visual assets to display them as visual citation cards inside chatbot interfaces like Google’s AI Overviews, Perplexity cards, or ChatGPT Search. If your images lack descriptive filenames or descriptive ALT text, the bots cannot match them to user queries, and you lose the visual citation.
6. Technical Takeaways & The Zelitho Solution
Building this telemetry setup changed how we publish content on our site. We implemented three immediate updates based on these observations:
- Keep content in subdirectories: Because bots like Claude crawl vertically down directory trees during deep sweeps, we avoid subdomains. Keeping everything under
zelitho.com/blog/ensures that deep sweepers read the guides and connect them directly to our root domain entity. - Optimize for extraction units: Since bots ignore standard landing copy, we structure our articles with clear tables, bulleted lists, and FAQ sections. We make it as easy as possible for a crawler to extract a fact.
- Optimize Image Assets: We ensure all our diagram assets have highly descriptive, keyword-rich filenames and alt attributes so AI search bots can pair them with visual answer blocks.
If you want your website to stand out in the AI search era, you cannot rely on traditional keyword strategies. You need a system that builds structured, research-backed, extraction-ready content that bots actually want to read.
That is why we built Zelitho. Zelitho is a content automation platform that handles the entire pipeline for you. It researches trending topics, defines intent-aware titles, structures drafts with real citations, and publishes the finished pages directly to your Webflow or WordPress directories automatically.
You do not need to spend weeks setting up server trackers or writing manual outlines. You can test the platform output for your own business right now. Get your first article written by our automated pipeline for $5 with no complex setup at app.zelitho.com.
Get your first AI-structured article for just $5
Write content that search engines index and AI chatbots cite. No complex setup required.