Retrieval Friction Index (RFI): Why LLMs Drop 99.9% of Retrieved Pages

Overview

Having an AI bot crawl your site does not mean your content will appear in the generated answer. In production retrieval pipelines, roughly 99.9% of crawled pages are discarded during re-ranking and Natural Language Inference (NLI) verification. The Retrieval Friction Index (RFI) measures this candidate drop-off rate. Understanding why systems reject pages helps convert bot visits into verified citations.

The 10,000-to-3 Candidate Attrition Funnel

When an AI crawler like ChatGPT-User, ClaudeBot, or PerplexityBot visits a URL, the content must pass several filtering stages before earning a citation:

1. Candidate Retrieval (approx. 10,000 documents): When an orchestrator runs query fan-out, BM25 indices and dense vector stores gather thousands of candidate pages.
2. Reciprocal Rank Fusion (approx. 50 to 100 chunks): The engine merges vector similarity scores with lexical matches, dropping candidates with weak alignment.
3. Late Interaction Re-Ranking (approx. 10 to 15 chunks): Models like ColBERT evaluate token-level MaxSim matches, removing irrelevant sections.
4. Natural Language Inference Grounding (1 to 3 citations): The generation model checks whether the remaining passages directly support the factual statement required. Only passages with high entailment confidence receive footnote links.

What Causes High Retrieval Friction in AI Search?

If your edge logs show regular AI bot traffic but low Live Citation Velocity (LCV), your content is encountering one of three friction bottlenecks:

1. DOM Bloat and Hydration Delays

AI search engines use headless HTTP runtimes with per-page latency limits, typically between 800ms and 2.5 seconds. Pages requiring client-side JavaScript rendering, heavy tracking scripts, or deeply nested markup often hit crawler timeouts.

When an engine encounters HTML where the core text is buried behind large CSS blocks, tracking tags, and layout wrappers, the parser truncates the payload before reaching the main argument.

2. Token Dilution and Low Information Density

Retrieval systems operate within finite context windows. To manage compute costs, document parsers split pages into chunks of 150 to 300 tokens.

If an article opens with generic filler, those chunks receive low cosine similarity scores when evaluated against specific technical questions. The system drops the filler chunks, discarding the page.

3. Weak Syntactic Entailment

Even when a passage reaches the final generation step, the model cites it only if it provides clear evidence for the answer. When content relies on speculative phrasing, the NLI model marks it as neutral rather than entailed, citing a competitor that states concrete figures instead.

Identifying Retrieval Friction Bottlenecks

3 Ways to Reduce Retrieval Friction

1. Use 4-Tier Syntactic Chunking

Format technical sections into self-contained 150-word units. Make sure each section begins with an explicit premise and contains numerical data, avoiding reliance on earlier paragraphs. Read our guide on syntactic content chunking for RAG.

2. Clean Semantic Microdata

Add structured JSON-LD schema (such as TechArticle and Dataset) and clean HTML5 elements (<article>, <section>, <table>). This helps parsers extract entity relationships without navigating custom CSS layouts.

3. Place Verifiable Data Early

Put research findings, pricing information, and architectural specs near the top. Search crawlers give the highest attention weights to the first 1,500 words; ensure key insights appear early.