Pith. sign in

REVIEW 3 major objections 4 minor 30 references

How Does Response Length Affect Long-Form Factuality

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Large language models produce less factually precise answers as responses get longer, and the paper attributes the decline to facts exhaustion: models exhaust reliable knowledge and shift to speculative details.

desk verdict BAFE is a solid, citable evaluator and the length-bias trend is credible; the 'facts exhaustion' mechanism is oversold and needs a softer claim. read the letter →

arxiv 2505.23295 v1 pith:4RYRROCN submitted 2025-05-29 cs.CL cs.AIcs.LG

James Xu Zhao , Jimmy Z.J. Liu , Bryan Hooi , See-Kiong Ng This is my paper
classification cs.CLcs.AIcs.LG
keywords largelanguagemodelslong-formfactualitylengthbiasfactualprecisionatomicfactverificationfactsexhaustionhallucinationresponse
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether asking a large language model to write a longer answer makes the answer less factual, and answers yes. Using a new fact-checking pipeline, it shows that factual precision falls steadily as requested length grows from 100 to 500 words. It then tests three explanations for the drop and argues the main driver is facts exhaustion: the model spends its more reliable knowledge early and compensates with speculative or invented details. Error buildup and long conditioning context are shown to play at most a minor role. If this is right, response length is a causal handle on factuality, and improving long-form reliability means helping models find new knowledge rather than only cleaning up after errors.

What carries the argument

The load-bearing object is BAFE, Bi-Level Atomic Fact Evaluation, an evaluator that decomposes a response into atomic facts, statements each carrying one piece of information, verifies each against a retrieved Wikipedia page, and sends only unsupported facts to a second check against a single Google Search query. It supplies the factual precision measurements on which the length-bias claim rests, and the paper validates it against human annotations with 89.31% agreement. Around it, the causal argument is carried by three controlled designs: length-anchored prompts for the existence of bias, autocorrelation and a counterfactual error injection for error propagation, a fixed evaluation section after varied context for long context, and a single-topic versus multiple-topic comparison for facts exhaustion.

What would settle it

Measure factual precision separately for the first and second halves of a 400-word single-topic response. Facts exhaustion predicts a decline within the response; if precision is flat across both halves, the length bias exists but the exhaustion mechanism is not what produces it.

Watch

Extended reading notes

Core claim

The central finding is a quantified length bias in long-form factuality. On a biography generation task, factual precision measured by the paper's evaluator drops from about 94.5% at 100 words to 90.5% at 500 words; on long descriptions of non-person entities it drops from 98.1% to 96.9%. The paper attributes this to facts exhaustion: when the model is forced to keep generating within one topic, it exhausts its more reliable knowledge and shifts to less certain details, whereas the same total content spread across multiple topics keeps precision higher. The authors rule out error propagation through lagged autocorrelation of binary error series and a counterfactual first-sentence flip, and rule out long context by varying preceding context length while evaluating a fixed target section. The conclusion is that the model is not going wrong because it compounds mistakes or loses track in a long context, but because the supply of well-grounded knowledge it can draw on runs out.

Load-bearing premise

The attribution to facts exhaustion assumes the single-topic versus multiple-topic comparison isolates the model's internal knowledge depletion, rather than confounds such as prompt structure, segmentation, or the model's tendency to pad a long single-topic request with filler.

Editorial extensions

If this is right

  • Factual precision declines as requested output length rises from 100 to 500 words on both studied tasks, so length alone can shift an answer's reliability.
  • Error propagation is weak and short-range: the autocorrelation of errors is only significant at lag 1 and near zero afterward, and deliberately falsifying the first sentence does not lower the factuality of later sentences.
  • Long prior context does not reduce the factuality of newly generated text: holding the evaluated topic fixed while varying context length leaves precision essentially unchanged.
  • Generating 400 words on a single biography topic lowers factual precision by 2.25 to 2.86 percentage points compared with the same total content split across two topics, supporting facts exhaustion as the primary mechanism.
  • A practical consequence is that long-form factuality work should shift from error-correction strategies toward knowledge supplementation or retrieval, and evaluation metrics should weigh factual coverage alongside precision.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of facts exhaustion would measure factual precision of the first and second halves of a single long response; the hypothesis predicts an intra-response decline, while instruction-following or filler accounts predict flat precision.
  • The paper's mechanistic conclusion could be sharpened by comparing two separate 200-word single-topic prompts against one 400-word single-topic prompt; if separate turns stay precise, the mechanism is about continuous same-topic generation, not word count alone.
  • If facts exhaustion generalizes, retrieval-augmented generation should flatten the length-precision curve, and the same length manipulation on weaker instruction-following models may show a different pattern because they are less able to hit the requested length.
  • The findings connect to judge-side length bias: if humans or model judges prefer longer answers while longer answers are factually sparser, preference optimization that rewards length could be trading perceived quality for factual precision.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper investigates whether response length affects factual precision in long-form LLM generation and, if so, what mechanism drives the effect. The authors introduce BAFE, a bi-level atomic-fact evaluation framework that verifies facts first against Wikipedia and then, for unsupported facts, against Google Search; they report 89.31% agreement with human annotations, outperforming FACTSCORE and SAFE at lower cost. Using BAFE, they run controlled prompt experiments on GPT-4o across two datasets (biography generation and long fact description) with requested lengths from 100 to 500 words, observing a monotonic decline in factual precision. They then test three hypotheses for this decline: error propagation, long context, and facts exhaustion. Autocorrelation and counterfactual analyses suggest error propagation is weak and short-term; a context-length manipulation shows no effect of prior context on evaluation-section precision; and a single-topic versus multiple-topic comparison shows lower precision in the single-topic setting. The paper concludes that facts exhaustion is the primary cause of length bias.

Significance. If the findings hold, the paper makes two useful contributions. First, BAFE is a practical evaluation tool: it is validated against human annotations with higher agreement than existing automatic factuality metrics while being substantially cheaper and faster than SAFE, and the authors release code and data. Second, the length-bias finding—longer requested outputs yield lower factual precision—is an important empirical observation for the long-form factuality literature, and the controlled prompt manipulation is a clean way to vary length. The main weakness is that the causal mechanism claim (facts exhaustion) rests on a single confounded experiment, and the length-bias trend itself lacks error bars and significance tests. The paper is therefore valuable as a demonstration of a phenomenon and a reusable evaluator, but the explanatory story needs to be either better tested or substantially toned down.

major comments (3)
  1. [Section 4.3, Figure 7; Appendix G, Tables 17–18] The experiment intended to isolate 'facts exhaustion' is confounded. In the single-topic setting, the model produces one 400-word passage per topic; in the multiple-topic setting, it produces two 200-word passages per topic, each embedded in a two-topic response with explicit section headers. The conditions therefore differ simultaneously in per-topic segment length (400 vs 200 words), in whether a topic switch occurs, and in the surface structure of the prompt. The observed 2.25–2.86 percentage point precision gap is compatible with an alternative explanation that does not involve internal knowledge depletion: when instructed to produce a 400-word passage on a single topic, the model may pad the latter half with plausible but unsupported filler, while the multiple-topic prompts elicit shorter, more conservative sub-passages. Because the Limitation section concedes that the internal knowledge state cannot be directly observed, the causal claim that 'facts exhaustion ... is the primary cause of factual degradation' (Abstract, Section 4.3, Conclusion) is not supported by the reported experiment. I recommend either reframing the conclusion as evidence consistent with facts exhaustion, or adding a control that holds segment length and response structure constant—for example, comparing two 200-word single-topic responses per topic with two 200-word segments within a single multi-topic response.
  2. [Section 3.2, Figures 1 and 3] The central length-bias trend is reported as point estimates without error bars or significance tests. The experiments use greedy decoding but vary the entity, so the decline from roughly 94.5% to 90.5% (biography) and from 98.1% to 96.9% (long fact description) could in principle be driven by a few difficult entities or by random variation. Please report bootstrap confidence intervals over entities—as is already done for the autocorrelation analysis in Figure 4—and a formal trend test, such as a linear mixed-effects model with a random intercept per entity or a permutation test. In addition, all experiments are on GPT-4o (Section 3.1); at least one open-source model would help assess whether the trend generalizes. These additions are important because the paper's headline claim is the length-bias phenomenon itself.
  3. [Abstract and Limitation] The Abstract and Conclusion state facts exhaustion as the established 'primary cause' of factual degradation, while the Limitation section says that 'due to the black-box nature of LLMs, it is hard to directly examine the facts exhaustion problem at the internal knowledge level.' This is an internal inconsistency between the strength of the claim and the admitted evidentiary basis. Even with the confounds in Section 4.3 addressed, the appropriate language would be 'consistent with' or 'suggests,' not 'reveals.' I flag this as a major issue because it affects how readers interpret the central scientific contribution.
minor comments (4)
  1. [Section 5.2] The reference to Zhou et al. (2024) appears mismatched: the cited paper is about object hallucination in large vision-language models, not about long-form text descriptions. Please verify the citation and replace it if the intended work is different.
  2. [Figure 4 caption] The phrase 'only the coefficient at lag 1 is slightly higher than 0 with statistical significance' is imprecise; it should read 'only the coefficient at lag 1 is significantly different from zero' or 'only the lag-1 confidence interval excludes zero.'
  3. [Table 2] The definition of 'Subsequent sentences' as 'all sentences after the first' is confusing because the second sentence is also part of that set. Please clarify whether the 'Second sentence' row and the 'Subsequent sentences' row overlap or whether 'Subsequent sentences' excludes the second sentence.
  4. [Section 3.2] The precision values reported in the text (e.g., 94.5%, 92.2%, 90.5%) are given without the number of responses or atomic facts per condition; including these sample sizes would help the reader assess the stability of the point estimates.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: BAFE is externally validated against human annotations, and the length-bias and mechanism findings come from separate controlled experiments rather than from the evaluator's construction.

full rationale

The paper's derivation chain is not circular. BAFE, the evaluation framework used throughout, is validated against 786 atomic facts with three independent human annotators, achieving 89.31% agreement and a Fleiss kappa of 0.7655 (Section 2.3). This is external grounding, not a self-defined target. The RQ1 length-bias result is an empirical measurement: only the requested output length is varied, and factual precision is computed by BAFE on the resulting responses (Section 3.1). The three RQ2 hypotheses are tested by separate experiments: error propagation via autocorrelation and a counterfactual first-sentence flip (Section 4.1), long context via fixed evaluation sections after varying context lengths (Section 4.2), and facts exhaustion via single-topic versus multiple-topic generation with facts aggregated across prompts (Section 4.3). The facts-exhaustion experiment has a potential confound, since the single-topic and multiple-topic conditions differ in response structure and topic continuity, but that is a threat to internal validity, not circularity. The paper's own Limitation section concedes that the internal knowledge-level mechanism cannot be directly examined, which weakens the causal attribution but does not make any result equivalent to its inputs by construction. No parameter is fitted and then renamed as a prediction, no equation identifies an output with an input, and no load-bearing self-citation chain is present. The central length-bias finding is self-contained, so the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper relies on several domain assumptions rather than fitted parameters. BAFE is an evaluation method, not a fitted model, and the experimental conditions are design choices. The main assumptions concern the sufficiency of atomic fact decomposition and external knowledge sources, the reliability of GPT-4o's length control, and the interpretability of the single-topic versus multiple-topic contrast.

assumptions (5)
  • domain assumption Atomic fact decomposition preserves the factual content of long-form responses and supports precision-based evaluation.
    Used throughout; inherited from FActScore and SAFE. If decomposition splits or merges facts improperly, precision estimates are distorted.
  • domain assumption Wikipedia and Google Search together provide sufficient coverage of established knowledge for the tested entities.
    BAFE's second-level verification relies on this; the authors acknowledge in the Limitation that specialized knowledge may be missed.
  • domain assumption GPT-4o's instruction following allows reliable control of response length via 'around X words' prompts.
    Used in all controlled experiments (Section 3.1); the authors note in Limitations that weaker models may not follow these instructions.
  • domain assumption Factual precision differences of 2 to 3 percentage points are meaningful and not dominated by BAFE's measurement error.
    BAFE has 89.31% agreement with humans, leaving roughly 10% evaluation error; the paper does not quantify how this error propagates into the reported precision gaps.
  • domain assumption Autocorrelation and counterfactual manipulation isolate error propagation.
    The lag-1 autocorrelation and first-sentence flip are interpreted causally, but the counterfactual flip also changes response structure, which is acknowledged in Section 4.1.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How Does Response Length Affect Long-Form Factuality." pith.science (2026). https://pith.science/paper/4RYRROCN

@misc{pith2026250523295,
  author       = {Pith},
  title        = {Pith review of: How Does Response Length Affect Long-Form Factuality},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4RYRROCN}},
  note         = {Machine review of arXiv:2505.23295}
}
read the original abstract

Large language models (LLMs) are widely used for long-form text generation. However, factual errors in the responses would undermine their reliability. Despite growing attention to LLM factuality, the effect of response length on factuality remains underexplored. In this work, we systematically investigate this relationship by first introducing an automatic and bi-level long-form factuality evaluation framework, which achieves high agreement with human annotations while being cost-effective. Using this framework, we conduct controlled experiments and find that longer responses exhibit lower factual precision, confirming the presence of length bias. To explain this phenomenon, we empirically examine three hypotheses: error propagation, long context, and facts exhaustion. Our results reveal that facts exhaustion, where the model gradually exhausts more reliable knowledge, is the primary cause of factual degradation, rather than the other two hypotheses.

Figures

Figures reproduced from arXiv: 2505.23295 by the authors.

Figure 1
Figure 1. Factual precision gradually decreases as re [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The pipeline of BAFE, an automatic bi-level long-form factuality evaluator. It first decomposes long responses into atomic facts. Each atomic fact is compared with a retrieved Wikipedia page at first-level verification. Atomic facts that are not supported at first-level, will reach second-level verification, where they are revised to be self-contained and checked against Google Search results. Human evaluation shows… view at source ↗
Figure 3
Figure 3. Factual precision decreases as response length [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Autocorrelation coefficient at different lags. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Factual precision in the evaluation section [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: Factual precision in single-topic and multiple [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Human evaluation interface for fact-checking. Instructions are provided at the top of the interface. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 10
Figure 10. Figure 10: Autocorrelation coefficient on GPT-4o in the [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 9
Figure 9. Figure 9: Autocorrelation coefficient at different lags [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

30 extracted references · 27 canonical work pages

  1. [3]

    title” and “snippet

    Long-form factuality in large language mod- els. In Advances in Neural Information Processing Systems, volume 37, pages 80756–80827. Curran As- sociates, Inc. Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi. 2023. A critical evaluation of evaluations for long-form question answering. In Proceedings of the 61st Annual Meeting of the Association for ...

    arXiv 1992

  2. [5]

    Instruction: Tell me a bio of <entity>

    Return ONLY the bio, and nothing else. Instruction: Tell me a bio of <entity>. Table 11: Prompt used for generating people biographies with varying requested output length. <Length> is set to {100, 200,..., 500} in our experiments. The generated responses are used to compute factual precision across different response lengths. The results are shown in Fig...

    work page

  3. [6]

    Instruction: Tell me about <entity>

    Return the information in paragraph form using plain text, not in markdown or any other format. Instruction: Tell me about <entity>. Table 12: Prompt used for generating long fact descriptions with varying requested output length. <Length> is set to {100, 200,..., 500} in our experiments. The generated responses are used to compute factual precision acros...

    work page

  4. [10]

    The bio should be around <Length> words

    work page

  5. [14]

    Return ONLY the bio, and nothing else

    work page

  6. [19]

    Instruction: Tell me a bio of <entity>

    Return ONLY the bio, and nothing else. Instruction: Tell me a bio of <entity>. Table 13: Prompt used for generating people biographies with the model’s default output length. The generated responses are used for investigating the effect of error propagation with autocorrelation analysis in Section 4.1.1. 22 Prompt for Flipping the Factual Correctness Syst...

    work page

  7. [20]

    It should introduce small perturbations rather than major shifts in context

    Keep it plausible: The new unsupported facts should NOT alter the main point of the original bio. It should introduce small perturbations rather than major shifts in context

    work page

  8. [21]

    **Small factual errors (e.g., places, dates, or minor career details) are suitable**

    The overall meaning should NOT change dramatically. **Small factual errors (e.g., places, dates, or minor career details) are suitable**

    work page

Show all 30 references
  1. [22]

    You can generate unsupported facts by slightly altering the supported facts, referring to the original unsupported facts, or generating plausible but unsupported details, or in other ways

  2. [23]

    Keep the provided unsupported facts in the new bio

  3. [24]

    You need to first give new unsupported facts

    The inserted unsupported fact should relate to the broader biography and fit into the narrative. You need to first give new unsupported facts. Then you need to give a new bio including the new unsupported facts. The new bio should match the format of the original bio as closel...

  4. [25]

    Be sure to only include accurate, factual information in the completed bio

  5. [26]

    The completed bio should be comprehensive and detailed

  6. [27]

    The completed bio should start with the given first sentence bio

    Do NOT change the given one-sentence bio. The completed bio should start with the given first sentence bio

  7. [28]

    <Topic A>

    Return ONLY the completed bio, and nothing else. Instruction: Complete the following bio of <entity>. The first sentence in the bio: <the first sentence> Table 15: Prompt used for continuing generation from the first sentence, which is either the original or a factually flippe...

  8. [29]

    <Topic A>

    Firstly generate a bio relates to "<Topic A>" with around <Context Length> words

  9. [35]

    Instruction: Tell me a bio of <entity>

    Return ONLY the bio, and nothing else. Instruction: Tell me a bio of <entity>. Table 16: Prompt used to investigate the effect of long context in Section 4.2.<Topic A> and<Context Length> are used in the context section. <Context Length> is set to {100, 200, ..., 600} in our e...

  10. [36]

    Generate a bio relates to "<Topic>" with around <Length> words

  11. [37]

    The response format should be like: ### Topic ### <Bio for Topic>

  12. [41]

    Early life

    Return ONLY the bio, and nothing else. Instruction: Tell me a bio of <entity>. Table 17: Prompt of single-topic setting in Section 4.3, for the investigation of facts exhaustion.<Topic> is set to either “Early life”, “Personal life” or “Career”. <Length> is set to 400 in our e...

  13. [42]

    <Topic A>

    Firstly generate a bio relates to "<Topic A>" with around <Length> words

  14. [43]

    <Topic B>

    Then generate a bio relates to "<Topic B>" with around <Length> words

  15. [44]

    The response format should be like: ### Topic A ### <Bio for Topic A> ### Topic B ### <Bio for Topic B>

  16. [45]

    Be sure to only include accurate, factual information in the response

  17. [46]

    The bio should be comprehensive and detailed

  18. [47]

    Do not include any controversial, disputable, or inaccurate factual claims in the response

  19. [48]

    Early life

    Return ONLY the bio, and nothing else. Instruction: Tell me a bio of <entity>. Table 18: Prompt of multiple-topic setting in Section 4.3, for the investigation of facts exhaustion.<Topic A> and <Topic B> are set to either “Early life”, “Personal life” or “Career”. <Length> is ...

  20. [2011]

    His tenure is noted for his commitment to justice and legal reform

    Prior to this, he was a judge at the Jammu and Kashmir High Court, where he was appointed in 1997 and later became the acting Chief Justice. His tenure is noted for his commitment to justice and legal reform. 200 Nisar Ahmad Kakru is a distinguished Indian jurist known for his...

    1997

  21. [2020]

    In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY , USA

    Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY , USA. Curran Associates Inc. Liang Chen, Yang Deng, Yatao Bian, Zeyu Qin, Bingzhe Wu, Tat-Seng Chua, and Kam-Fai Wong. 2...

    2023 arXiv

  22. [2024]

    In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 6626–6642, Bangkok, Thailand

    Advancing large language models to capture varied speaking styles and respond properly in spoken conversations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 6626–6642, Bangkok, Thailand. Association...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.