REVIEW 3 major objections 3 minor
The Attribution Crisis in LLM Search Results
T0 review · 3 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Web-enabled LLMs routinely answer queries without citing the relevant web pages they consumed.
desk verdict A real-world measurement of LLM attribution gaps with a valuable dataset, but the headline numbers hinge on a log-completeness assumption the paper itself concedes is violated for one model. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the attribution gap, defined as the difference between relevant URLs fetched by the model and URLs it actually cites. The measurement is carried by a negative binomial hurdle model, which first models whether a response produces any citations and then models how many citations it produces; this lets the authors separate 'no citation at all' from 'cites too few.' The data are roughly 14,000 LMArena conversation logs with search-enabled systems, with fetched URLs labelled as relevant or not from conversation context. This machinery turns an abstract concern about borrowing without credit into a comparable, per-query number across models.
What would settle it
If a provider released its own server-side traces for the same conversations and the attribution gap for GPT-4o was still near zero—and similarly small for Gemini and Sonar—the paper's central conclusion that large uncited gaps are hidden by selective disclosure would fail. Conversely, if server-side traces reveal even larger uncited fetch counts, the gap estimates in the paper are conservative.
Extended reading notes
Core claim
The paper's central claim is that an attribution gap—the difference between the relevant URLs a model reads and the URLs it actually cites—is large and systematic in current web-enabled LLM search engines. Using LMArena conversation logs, the paper finds that Gemini and Sonar leave roughly three relevant websites uncited on the average query, while GPT-4o's much smaller apparent gap is best explained by its selective log disclosures rather than by superior attribution. A negative binomial hurdle model separates the decision to cite anything at all from the number of citations given, showing that citation efficiency varies from 0.19 to 0.45 across models on identical queries. The paper concludes that high-volume, low-credit retrieval—fetching many pages while crediting few—is the dominant pattern for several major systems.
Load-bearing premise
The findings assume the conversation logs accurately record every URL each model fetched and cited, and that the authors can correctly label which fetched pages were relevant from the conversation context.
Editorial extensions
If this is right
- If the claim holds, head-to-head benchmarks should not rank GPT-4o as the best-attributing model, because its small uncited gap may be an artifact of selective logging.
- Citation efficiency becomes a design metric: models range from 0.19 to 0.45 extra citations per additional relevant page, so large gains are available without new retrieval ability.
- Content producers can expect high-volume, low-credit systems to keep sending disproportionate traffic value to a few cited pages while many read pages go uncredited.
- Standardized telemetry and full disclosure of search traces would let users and regulators audit which pages a model actually used, making attribution gaps independently verifiable.
Reading between the lines
- The paper's selective-disclosure caveat about GPT-4o may generalize: if logs are incomplete for any model, the measured gaps and model ordering are best read as lower bounds on the true attribution problems.
- One testable extension is to measure whether uncited fetch volume predicts downstream harms like traffic diversion or content scraping, not just credit loss.
- Citation-efficiency scores could become a practical consumer signal if providers published them per query type, nudging systems toward fewer high-credit citations rather than many invisible fetches.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper analyzes approximately 14,000 LMArena conversation logs of search-enabled LLMs to measure an 'attribution gap' between the relevant web pages a model fetches and the citations it actually provides. The authors report that Gemini frequently answers without citations (92% of answers with no clickable source), Sonar fetches many pages but cites only a fraction, and a negative binomial hurdle model estimates that the average Gemini or Sonar query leaves about 3 relevant sites uncited, whereas GPT-4o's small uncited gap is attributed to selective log disclosure rather than better attribution. Citation efficiency is reported to vary from 0.19 to 0.45 across models. The paper concludes with a recommendation for transparent LLM search architectures with standardized telemetry. The review is based on the abstract only, as the full text is not available.
Significance. If the findings withstand scrutiny, this paper would provide a valuable empirical baseline for a real and growing problem: the lack of attribution to sources in AI-generated search answers. The use of naturalistic LMArena logs is a strength because it captures real user behavior rather than synthetic benchmarks, and the paper makes a practical contribution by quantifying the gap in a way that could inform platform accountability. The reported cross-model variation in citation efficiency is a potentially important result, suggesting that attribution behavior is a design choice rather than a technical necessity. However, the identification of the gap depends heavily on the completeness and representativeness of the logs and on the validity of the 'relevant URLs read' measure, both of which are not verifiable from the abstract. The analysis is therefore potentially significant but currently not fully substantiated.
major comments (3)
- [Abstract, second paragraph] The claim that GPT-4o's tiny uncited gap 'is best explained by its selective log disclosures' creates an identification problem: if logs can omit real citations or fetches for GPT-4o, the same omission could affect Gemini and Sonar. The paper treats absence of a fetch/citation in the log as evidence that no fetch/citation occurred, yet for GPT-4o it treats absence as possibly a logging artifact. Without a per-model false-negative rate for logged fetches and citations, or an independent validation of log completeness, the reported uncited-gap magnitudes and the citation-efficiency ranking (0.19 vs 0.45) conflate logging policy with retrieval design. This is load-bearing for the central claim of cross-model differences, and the abstract does not indicate how the issue is addressed.
- [Abstract, first paragraph] The central measure is 'relevant URLs read,' but the abstract does not describe how relevance is determined from conversation context. If relevance is labeled by the authors without a validated protocol, inter-annotator agreement, or a defined boundary between relevant and irrelevant fetched pages, then the uncited-gap estimates may reflect labeling bias rather than model behavior. The full text must provide a concrete annotation scheme and reliability statistics; the current abstract gives no basis to assess whether the outcome variable is measured consistently across models.
- [Abstract, second paragraph] The negative binomial hurdle model is mentioned but not specified in terms of covariates or identification. In particular, how is the 'selective log disclosure' effect for GPT-4o estimated? If the model includes a logging-artifact parameter for GPT-4o that is not identified from external data, then the subsequent comparison of uncited gaps across models is circular: the conclusion that GPT-4o's small gap is an artifact would be built into the model rather than tested. The abstract needs to clarify the model specification and the source of identification for any disclosure parameter.
minor comments (3)
- [Abstract, first paragraph] The phrase 'exploitation patterns' has a normative tone; consider using a more neutral term such as 'observed patterns' or 'usage patterns' to avoid implying intentionality by the models.
- [Abstract, first sentence] The scope of the dataset ('approximately 14,000 conversation logs') should be dated and described in terms of model versions, retrieval tool configurations, and time period, since LLM behavior changes quickly and the findings may not generalize beyond the specific systems tested.
- [Abstract, final sentence] The recommendation of 'full disclosure of search traces and citation logs' would be more actionable if paired with a measurable standard, such as a minimum set of telemetry fields or a machine-readable citation format; the abstract currently states the goal without operationalization.
Circularity Check
No significant circularity: the attribution-gap estimates are directly measured from conversation logs, and the hurdle-model parameters are fitted to observed citation counts rather than defining the target quantity.
full rationale
The paper's central quantities—percentage of responses without search, percentage without clickable citations, and relevant pages visited versus cited—are presented as direct empirical measurements from approximately 14,000 LMArena logs. The negative binomial hurdle model is used to estimate an adjusted 'uncited gap' from observed fetch and citation counts; nothing in the abstract indicates that the model's dependent variable is defined by, or fitted to, the same quantity it is claimed to predict. The claim that GPT-4o's small gap may reflect selective log disclosure is an interpretive caveat about data completeness, not a circular reduction: it does not define the estimated gap as equal to a fitted parameter. No self-citation chain, uniqueness theorem, or ansatz-smuggling is visible in the abstract. Potential measurement bias from incomplete logs is a validity threat, not a circularity, because the paper's derivation would still fail or succeed depending on external ground truth about fetched and cited URLs. With abstract-only evidence, no specific equation or construction can be quoted to demonstrate that a prediction is equivalent to its input. Therefore the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (1)
- negative binomial hurdle model parameters (mean count, overdispersion, zero-inflation probability) =
not reported in abstract
assumptions (4)
- domain assumption LMArena conversation logs accurately record which web pages each model retrieved and cited.
- domain assumption The relevance of each URL visited can be reliably determined from the log context.
- domain assumption Negative binomial hurdle model assumptions (for example, appropriate zero-inflation and count distribution) hold for this data.
- domain assumption The LMArena sample is representative of real-world LLM search use.
Cite this review
Pith. "Pith review of The Attribution Crisis in LLM Search Results." pith.science (2026). https://pith.science/paper/TD7BCKQ7
@misc{pith2026250800838,
author = {Pith},
title = {Pith review of: The Attribution Crisis in LLM Search Results},
year = {2026},
howpublished = {\url{https://pith.science/paper/TD7BCKQ7}},
note = {Machine review of arXiv:2508.00838}
}
read the original abstract
Web-enabled LLMs frequently answer queries without crediting the web pages they consume, creating an "attribution gap" - the difference between relevant URLs read and those actually cited. Drawing on approximately 14,000 real-world LMArena conversation logs with search-enabled LLM systems, we document three exploitation patterns: 1) No Search: 34% of Google Gemini and 24% of OpenAI GPT-4o responses are generated without explicitly fetching any online content; 2) No citation: Gemini provides no clickable citation source in 92% of answers; 3) High-volume, low-credit: Perplexity's Sonar visits approximately 10 relevant pages per query but cites only three to four. A negative binomial hurdle model shows that the average query answered by Gemini or Sonar leaves about 3 relevant websites uncited, whereas GPT-4o's tiny uncited gap is best explained by its selective log disclosures rather than by better attribution. Citation efficiency - extra citations provided per additional relevant web page visited - varies widely across models, from 0.19 to 0.45 on identical queries, underscoring that retrieval design, not technical limits, shapes ecosystem impact. We recommend a transparent LLM search architecture based on standardized telemetry and full disclosure of search traces and citation logs.
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.