Bytespider: Crawl Frequency, Bandwidth Impact & Rate-Limiting Guide
Bytespider: Crawl Frequency, Bandwidth Impact & Rate-Limiting
Bytespider is ByteDance’s high-frequency web crawler. Ingesting content for TikTok search, Doubao LLM grounding, and ByteDance algorithmic models, it is one of the most aggressive crawlers on the modern web.
1. Technical User-Agent Identification
Bytespider identifies itself with mobile and desktop user-agent signatures containing its feedback email:
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)Network Origin & Autonomous Systems:
- Official IP Feeds:None published. Unlike Google and OpenAI, ByteDance does not maintain an official machine-readable IP ranges JSON.
- Autonomous System Numbers (ASNs):
AS138699 (ByteDance Singapore)andAS396982 (Google Cloud egress). - Reverse DNS Pattern: Must resolve to
*.bytedance.com. - Crawl Pattern: Recursive pagination sweeps, deep nested asset requests, and multiple concurrent threads across regional cloud nodes.
Important Disambiguation: Bytespider vs. TikTokSpider
ByteDance operates two separate crawler signatures. Conflating them can disrupt your brand’s presence on social media:
Bytespider: Ingests bulk text for search indexing and LLM training (Doubao, Toutiao). Safe to block or rate-limit.TikTokSpider: Fetches OpenGraph tags and link previews when users share URLs inside TikTok messages or profile bios. Blocking this user-agent breaks rich previews on TikTok.
2. Zelitho Production Telemetry: 1,827 Ingestion Requests
Across our 110-day benchmark tracking window ($N=13,971$ total edge sessions), Bytespider accounted for 28.8% of verified bot sessions, outpacing traditional Googlebot by 1.8 to 1:
The Economic Asymmetry: While ChatGPT-User logged 2,970 hits that directly corresponded to live search citations and referral visits, Bytespider consumed significant server CPU and bandwidth without returning meaningful referral traffic to western technical sites.
3. Engineering Decision: Allow, Rate-Limit, or Block?
Depending on your infrastructure compute budget and commercial market focus, engineering teams should choose between two approaches:
Option A: Block Completely (Recommended for Western B2B SaaS): If your target buyers operate in North America and Europe and your server compute budget is limited, blocking Bytespider eliminates uncompensated server overhead with zero impact on Google or ChatGPT citations.
Option B: Edge Rate-Limiting (Recommended for Global Enterprise): If your organization values presence in Asian LLM ecosystems (ByteDance’s Doubao and Lark platforms), allow Bytespider but enforce a strict edge rate limit (e.g., 2 requests per second) to prevent server throttling.
4. Configuration & Defense Directives
Apply these rules to restrict or block Bytespider at the server or edge layer:
Robots.txt: Block Bytespider While Preserving TikTok Previews
# Disallow ByteDance bulk scraper User-agent: Bytespider Disallow: / # Allow TikTok social sharing link previews User-agent: TikTokSpider Allow: /Nginx Edge Rate-Limiting Configuration
# Limit Bytespider to 2 requests per second with a burst buffer of 5 limit_req_zone $binary_remote_addr zone=bytespider_limit:10m rate=2r/s; if ($http_user_agent ~* "Bytespider") { limit_req zone=bytespider_limit burst=5 nodelay; }Frequently Asked Questions: Bytespider
How do I block Bytespider without breaking TikTok card previews?
Specify User-agent: Bytespider with Disallow: / while explicitly maintaining User-agent: TikTokSpider with Allow: /. Do not use blanket keyword blocks against ByteDance in your web application firewall.
Does ByteDance publish official IP address ranges for Bytespider?
No. ByteDance does not provide public machine-readable IP lists. Because Bytespider routes traffic through commercial cloud platforms, blocking entire IP blocks risks rejecting legitimate cloud traffic. Filter by verified user-agent or edge rate limits instead.
Does Bytespider respect the crawl-delay directive in robots.txt?
Industry monitoring shows Bytespider compliance with crawl-delay is inconsistent. Enforcing request limits at the edge via Nginx or Cloudflare WAF HTTP 429 rate limiting provides reliable protection against server resource exhaustion.
Source Data: This telemetry was collected and analyzed as part of The 2026 Global AI Search & LLM Retrieval Benchmark (13,971 Production Edge Sessions Analyzed by Zelitho Research).
Protect Your Server from Uncompensated AI Scraping
Zelitho helps enterprise teams analyze edge telemetry, block abusive model scrapers, and focus crawl budgets exclusively on high-conversion AI search engines.
Explore Zelitho AI Growth Plans →