Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

"Lost-in-the-Later": Framework for Quantifying Contextual Grounding in Large Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Large language models systematically underuse later parts of their input, a positional bias the authors call 'lost-in-the-later,' which appears even in contexts of only 50 atomic sentences and is worse for reasoning models and…

desk verdict Useful framework and dataset, and the lost-in-the-later pattern is worth taking seriously, but the central measurement rests on NLI judgments that are validated only on synthetic responses, so the effect is conditional pending a human check. read the letter →

arxiv 2507.05424 v1 pith:3D5G7KVT submitted 2025-07-07 cs.CL cs.AI

Yufei Tao , Adam Hiatt , Rahul Seetharaman , Ameeta Agrawal This is my paper
classification cs.CLcs.AI
keywords contextualknowledgeparametricpositionalbiaslost-in-the-laterchain-of-thoughtpromptingmultilingualevaluationnaturallanguageinferencehallucination
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that large language models carry a systematic positional bias, dubbed 'lost-in-the-later,' under which they underuse relevant information that appears later in their input, even when the input contains only about 50 atomic sentences and is far below typical long-context limits. To demonstrate this, the authors introduce CoPE, an evaluation framework that scores a response's contextual knowledge (sentences entailed by the provided context) versus parametric knowledge, and a new multilingual dataset of 15,000 atomic sentences in English, Spanish, and Danish. Across six models and three languages, the framework shows that recall falls steadily from the first to the last quartile of the context, that reasoning models and chain-of-thought prompting use context even less, and that a context-only 'CK prompt' raises grounding and lowers hallucination in summarization. If the claim is right, the failure to ground in late context is a general property of current LLMs rather than a long-context artifact, and prompting strategies matter for fixing it.

What carries the argument

The machinery is CoPE, a sentence-level evaluation framework. It atomizes both the input context and the model's response into minimal 'atomic sentences,' then uses a bidirectional NLI model (mDeBERTa-v3-based INFUSE) with a threshold of t=0.7 to classify each response sentence as contextual knowledge (CK) if it is entailed by the context, or parametric knowledge (PK) otherwise. From these labels it computes the CK score (fraction of response sentences grounded in context) and a context-recall distribution (fraction of sentences recalled from each segment of the context). The MultiWikiAtomic dataset, built by extending WikiAtomic with Spanish and Danish articles, supplies the controlled contexts of 0–50 atomic sentences, and the recall-by-quartile curves are the instrument that exposes the lost-in-the-later bias.

What would settle it

Take a sample of the paper's response sentences across all three languages and all context quartiles, have human annotators (or an independently validated NLI model) decide entailment, and recompute the context-recall curves; if the late-context deficit shrinks to zero or reverses under the alternative judge, the lost-in-the-later effect is an artifact of the mDeBERTa-v3 classifier rather than a genuine property of the models.

Watch

Extended reading notes

Core claim

The central discovery is the 'lost-in-the-later' effect: when answering open-ended questions about a supplied context, LLMs incorporate progressively less information from later segments of the context and rely instead on parametric memory, even when the later information is relevant and the context is short. The paper reports that across six models (GPT-4o, Gemini 1.5 Pro, LLaMA 3.2 90B, LLaMA 3.2 3B, GPT-o3, Qwen 3 235B) and three languages, context recall is highest in the first quartile and lowest in the last, with CK usage peaking near 70%; the pattern persists when context sentences are shuffled, ruling out the original document order as the cause. The authors further find that reasoning models and CoT prompting reduce CK further, that parametric content tends to appear at the end of responses, and that higher CK is associated with less hallucination. On the basis of these measurements, they show that a prompt combining strict context-only instruction with balanced-use instruction increases CK by several points and improves factual grounding in multi-document summarization.

Load-bearing premise

The whole measurement chain hinges on the untested assumption that the mDeBERTa-v3 NLI model, at threshold 0.7, correctly decides in English, Spanish, and Danish whether a response sentence is entailed by a context sentence; if that classifier is biased by sentence position or language, the lost-in-the-later curves would be artifacts of the measurement rather than properties of the LLMs.

Editorial extensions

If this is right

  • If the lost-in-the-later effect is real, positional bias is not only a long-context problem: even 50-sentence inputs produce declining recall, so short-context applications such as question answering and summarization need explicit recall awareness.
  • Reasoning models and chain-of-thought prompting do not cure grounding and can make it worse, so users should not assume that more reasoning steps mean better context adherence.
  • A simple CK prompt that forbids outside information and demands balanced use of all context segments raises CK scores and produces more factually grounded summaries with less hallucination.
  • CK score and FActScore move together, so measuring contextual grounding gives a practical proxy for hallucination risk across languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because the recall decline appears even after shuffling context order, the bias likely lives in how pretrained transformers weight earlier input positions rather than in document structure; this could be tested by probing attention patterns or by training small models on position-randomized data.
  • Beyond the paper: the same CoPE measurement could be applied to retrieval-augmented generation pipelines, where late-placed retrieved passages may be systematically ignored, and the CK prompt could be adapted as a retrieval-aware generation instruction.
  • Beyond the paper: the finding that CoT shortens responses and lowers recall suggests reasoning and grounding compete for a limited output budget; structured reasoning formats that separate reasoning from the final answer might preserve grounding where free-form CoT loses it.
  • Beyond the paper: the CK-score/FActScore correlation suggests CoPE could serve as a lightweight, language-agnostic hallucination monitor in production systems where running full factuality checks is too expensive.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces CoPE, a framework that classifies each atomic sentence of a model response as contextual knowledge (CK) or parametric knowledge (PK) using a multilingual NLI model, together with a new dataset, MultiWikiAtomic, containing 15,000 atomic sentences in English, Spanish, and Danish. Using this framework, the authors measure CK scores and context recall (CR) for six LLMs and three languages, and report a 'lost-in-the-later' effect: models increasingly underutilize later parts of the context even when the context is only 50 atomic sentences. They further report that reasoning models and chain-of-thought prompting reduce contextual grounding, that PK increases toward the end of responses, that counterfactual contexts lower CK scores, that higher CK is associated with lower hallucination, and that a CK-informed prompt improves grounding in a summarization case study.

Significance. If the central measurement is valid, the paper makes a useful empirical contribution: it provides a reusable multilingual evaluation framework, a new dataset, and evidence that positional bias in context use appears even in short, knowledge-consistent contexts, which would strengthen and extend the 'lost in the middle' literature. The framework is model-agnostic and applicable to closed models, the dataset and code are released, and the threshold-sensitivity ablation in Appendix B is a good-faith robustness check. However, the main empirical claims are only as strong as the NLI-based entailment judgments, and the paper currently validates those judgments only on synthetic responses. The missing validation on real model outputs is a load-bearing weakness, and the CoT comparison is confounded by response format and length.

major comments (3)
  1. [§2 and Appendix B] The CK/CR measurements rest entirely on mDeBERTa-v3 NLI entailment judgments at threshold t=0.7, but the accuracy evaluation in Appendix B validates the classifier only on synthetically constructed responses (10 CK sentences, 5 unrelated PK sentences) and on alignment with the authors' prior WikiAtomic setup. Actual model responses contain paraphrase, synthesis, cross-sentence inference, and partial entailment, and the NLI model is known to be weaker in Spanish and Danish; the threshold-sensitivity ablation in Appendix B only filters scores within the same classifier and cannot detect a systematic under-detection of entailment for later context sentences. If such a bias exists, the monotonic decline in CR across quartiles in Figures 9-11 would be a measurement artifact rather than evidence for 'lost-in-the-later.' The authors should validate CoPE on human-annotated samples of real responses, stratified by language, model, and context position, and report per-language agreement and error rates.
  2. [§4.2, §4.3, and Figure 3] The central quantitative claims—CK plateaus near 70%, CR decreases across quartiles, and the effect persists under randomization—are presented as single aggregate curves without confidence intervals, error bars, or significance tests. With per-topic variability and only six models, the '~5%' shift in the randomization test (Section 4.3) should be accompanied by per-model distributions and a paired test; the qualitative stability of the threshold ablation does not supply this. Reporting bootstrap confidence intervals or mixed-effects models would materially strengthen the claim that the effect is structural rather than noise.
  3. [Section 5 and Appendix J] The CoT comparison is confounded by response format and length. The CoT prompt instructs the model to return a JSON object with separate 'reasoning' and 'answer' fields; if the full generated text is atomized and scored by CoPE, the reasoning portion is counted in the denominator and is likely to be classified as PK, directly lowering CK and recall independently of the model's actual context use. In addition, Table 5 shows CoT responses are roughly one-third to one-half the length of original responses, so the lower recall in Figure 7 may reflect token-budget constraints rather than a grounding deficit. The authors should either score only the extracted answer field, or analyze the full-output and answer-only conditions separately, and should include a length-matched or length-controlled comparison before concluding that CoT degrades contextual grounding.
minor comments (6)
  1. [Table 2] The table lists 'CoT + CK Prompt' twice with different CK values (73.10 and 75.10 for English); the second row appears to be mislabeled and should be corrected.
  2. [§2 and Appendix C] The definition of CR_q is underspecified when a segment Q_q contains multiple sentences: the indicator 'I(S_i is entailed by Q_q)' should state whether entailment is checked against each sentence in the segment and aggregated as a maximum, a union, or some other policy.
  3. [Throughout] The phenomenon is sometimes written as 'lost in the later' (Section 7, Conclusion) and sometimes as 'lost-in-the-later'; the terminology should be made consistent.
  4. [§5] There is a duplicated word in 'we explore whether whether better prompting strategies' that should be removed.
  5. [Appendix J] The text refers to 'CoPEval' in the opening sentence, but the framework is called CoPE elsewhere; the name should be unified.
  6. [Table 4] The header 'Fasle First' is a typo for 'False First'.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: CoPE's measurements rest on an external NLI model; the only self-referential element is threshold calibration against the authors' prior WikiAtomic setup, which is not load-bearing.

full rationale

The claimed derivation chain is empirical rather than deductive: contexts and responses are atomized, each response atom is labeled CK/PK by mDeBERTa-v3 bidirectional NLI at threshold t=0.7, and context recall (CR) is computed as per-quartile entailment recall. The lost-in-the-later effect is an observed monotonic decline in CR over input quartiles, not a quantity defined in terms of the paper's own fitted parameters. The central measurement is anchored by an externally trained multilingual NLI model, and the threshold is not fitted to the actual target responses. The only self-referential element is Appendix B's threshold calibration: "We selected t = 0.7 because it yielded distributions most aligned with the original evaluation" from Tao et al. (2024), the authors' prior WikiAtomic work. This is a minor self-citation, but it is not load-bearing: the paper reports a threshold-sensitivity ablation showing "shifts were minimal and the main trends held across languages and models," and Section 4.3 adds a randomization control, so the main positional trend is not forced by the particular threshold choice. The skeptic concern that mDeBERTa-v3 may systematically mislabel actual model responses, especially in Danish, is a measurement-validity risk rather than a circularity: no equation in the paper reduces CK or CR to the NLI model's own outputs by construction, and no fitted parameter is renamed as a prediction. PK = 100 - CK is a complementary definition, not a derived result. The CK-prompt evaluation is a prompt-policy comparison rather than a first-principles prediction, so it does not reduce to its own inputs. Overall, there is no significant circularity; the score of 2 reflects only the minor self-referential calibration step, with the substantive findings relying on external measurement and robustness checks.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claims rest on a chain of measurement choices, mainly entailment-based attribution, GPT-4o atomization, and the NLI threshold, rather than on free parameters fitted to the headline result. No new physical or architectural entities are introduced. The threshold t=0.7 is a calibrated tuning parameter, but the sensitivity analysis suggests the qualitative trends persist across threshold choices.

free parameters (1)
  • CK-PK entailment threshold t = 0.7
    Selected in Appendix B to match prior WikiAtomic distributions and to minimize error on synthetic English, Spanish, and Danish sets; all CK scores and context recall distributions depend on this threshold.
assumptions (4)
  • domain assumption A response sentence is either entailed by the input context (CK) or not (PK), and this dichotomy captures contextual grounding.
    Section 2 defines CK as entailment by context and PK as the complement; the framework and all conclusions use this behavioral dichotomy.
  • domain assumption GPT-4o prompt-based atomization splits context and responses into faithful minimal propositions across English, Spanish, and Danish.
    Appendix A; inconsistent atomization would change sentence counts and distort CK and recall scores.
  • domain assumption mDeBERTa-v3 multilingual NLI scores approximate semantic entailment for the three languages at threshold 0.7.
    Section 2 and Appendix B; no human validation is reported on the actual response corpus.
  • domain assumption FActScore with a multilingual Wikipedia knowledge base and XLM-R tokenizer measures hallucination in Spanish and Danish adequately.
    Appendix G and the Limitations section; the authors concede that FActScore was originally designed for English.

how reviews work

0 comments
Cite this review

Pith. "Pith review of "Lost-in-the-Later": Framework for Quantifying Contextual Grounding in Large Language Models." pith.science (2026). https://pith.science/paper/3D5G7KVT

@misc{pith2026250705424,
  author       = {Pith},
  title        = {Pith review of: "Lost-in-the-Later": Framework for Quantifying Contextual Grounding in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3D5G7KVT}},
  note         = {Machine review of arXiv:2507.05424}
}
read the original abstract

Large language models are capable of leveraging both contextual and parametric knowledge but how they prioritize and integrate these sources remains underexplored. We introduce CoPE, a novel evaluation framework that systematically measures contextual knowledge (CK) and parametric knowledge (PK) across models and languages. Using our MultiWikiAtomic dataset in English, Spanish, and Danish, we analyze how large language models (LLMs) integrate context, prioritize information, and incorporate PK in open-ended question answering. Our analysis uncovers a phenomenon we call lost-in-the-later, where LLMs tend to overlook or deprioritize information that appears later in a given context, revealing a strong positional bias that affects contextual grounding. We further find that reasoning models, as well as non-reasoning models prompted with chain-of-thought (CoT), use context even less than non-reasoning models without CoT and fail to mitigate the lost-in-the-later effect. CoT prompting, in particular, results in lower recall and shorter responses, leading to degraded contextual grounding. Based on these insights, we design prompt-based methods to effectively leverage input context. A case study applying CoPE to summarization demonstrates that CK-informed prompting improves factual grounding and reduces hallucination.

Figures

Figures reproduced from arXiv: 2507.05424 by the authors.

Figure 1
Figure 1. Overview of the MultiWikiAtomic dataset creation and CoPE framework. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Example of CoPE identifying CK, PK, and context recall in a model response. Contextual Knowledge score This metric quan￾tifies the proportion of a response that is directly entailed by the provided context versus information not directly entailed from the context. Formally, given a model-generated response R consisting of n atomic sentences, the CK score is computed as: CK = Pn i=1 I(Si ∈ C) n × 100 where C represen… view at source ↗
Figure 3
Figure 3. Contextual Knowledge (CK) scores across context lengths for different models and languages. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (20 more)
Figure 4
Figure 4. Figure 4: Context Recall (CR) distribution of Llama 3.2 3B across English, Spanish, and Danish. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: PK distribution across response quartiles in [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Average FActScore for each model in English, Spanish, and Danish, grouped by different CK score ranges. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Context recall (at 50 contexts) across dif [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Threshold ablation results across all languages and models. Each plot shows CK/PK score trends with and [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Context recall results for English (a) GPT-4o (b) Gemini 1.5 Pro (c) Llama 3.2 90B (d) Llama 3.2 3B (e) GPT-o3 (f) Qwen 3 235B [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Context recall results for Spanish 13 [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Context recall results for Danish (a) GPT-4o (b) Gemini 1.5 Pro (c) Llama 3.2 90B (d) Llama 3.2 3B (e) GPT-o3 (f) Qwen 3 235B [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: PK distribution in English responses edge base by using a Wikipedia dataset4 that in￾cludes Spanish and Danish articles. To better ac￾commodate the multilingual setting, we also re￾placed the original RoBERTa tokenizer (Liu et al., 2019) with the XLM-RoBERTa tokenizer…
Figure 13
Figure 13. Figure 13: PK distribution in Spanish responses (a) GPT-4o (b) Gemini 1.5 Pro (c) Llama 3.2 90B (d) Llama 3.2 3B (e) GPT-o3 (f) Qwen 3 235B [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: PK distribution in Danish responses J Summarization Task Implementation Detail Here we will go through the implementation details using CoPEval for the summarization task with DivSum dataset. The exact prompt we used to for simple (vanilla) summaries is: The following…
Figure 15
Figure 15. Figure 15: GPT-4o average number of CK/PK in responses (Top) and context recall (Bottom) across Different [PITH_FULL_IMAGE:figures/full_fig_p017_15.png]
Figure 16
Figure 16. Figure 16: GPT-4o average number of CK/PK in responses (Top) and context recall (Bottom) across Different [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]
Figure 17
Figure 17. Figure 17: GPT-4o average number of CK/PK in responses (Top) and context recall (Bottom) across Different [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]
Figure 18
Figure 18. Figure 18: Llama 3.2 90B average number of CK/PK in responses (Top) and context recall (Bottom) across Different [PITH_FULL_IMAGE:figures/full_fig_p020_18.png]
Figure 19
Figure 19. Figure 19: Llama 3.2 90B average number of CK/PK in responses (Top) and context recall (Bottom) across Different [PITH_FULL_IMAGE:figures/full_fig_p021_19.png]
Figure 20
Figure 20. Figure 20: Llama 3.2 90B average number of CK/PK in responses (Top) and context recall (Bottom) across Different [PITH_FULL_IMAGE:figures/full_fig_p022_20.png]
Figure 21
Figure 21. Figure 21: Qwen 3 235B average number of CK/PK in responses (Top) and context recall (Bottom) across Different [PITH_FULL_IMAGE:figures/full_fig_p023_21.png]
Figure 22
Figure 22. Figure 22: Qwen 3 235B average number of CK/PK in responses (Top) and context recall (Bottom) across Different [PITH_FULL_IMAGE:figures/full_fig_p023_22.png]
Figure 23
Figure 23. Figure 23: Qwen 3 235B average number of CK/PK in responses (Top) and context recall (Bottom) across Different [PITH_FULL_IMAGE:figures/full_fig_p024_23.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMs

    cs.CL 2026-01 conditional novelty 5.0 of 10

    On a new extended needle-in-a-haystack benchmark, explicit anti-hallucination prompts and dispersed fact placement cause some long-context LLMs to over-refuse or collapse in accuracy, while others remain robust.

Reference graph

Works this paper leans on

32 extracted references · 9 canonical work pages · cited by 1 Pith paper

  1. [1]

    Jordi Bayarri-Planas, Ashwin Kumar Gururajan, and Dario Garcia-Gasulla. 2024. https://arxiv.org/abs/2409.15127 Boosting healthcare llms through retrieved context . Preprint, arXiv:2409.15127

    work page Pith review arXiv 2024

  2. [2]

    Baolong Bi, Shenghua Liu, Yiwei Wang, Yilong Xu, Junfeng Fang, Lingrui Mei, and Xueqi Cheng. 2025. https://arxiv.org/abs/2503.15888 Parameters vs. context: Fine-grained control of knowledge reliance in language models . Preprint, arXiv:2503.15888

    arXiv 2025

  3. [3]

    Chaoran Chen, Daodao Zhou, Yanfang Ye, Toby Jia jun Li, and Yaxing Yao. 2025. https://arxiv.org/abs/2410.13387 Clear: Towards contextual llm-empowered privacy policy analysis and risk generation for large language model applications . Preprint, arXiv:2410.13387

    work page Pith review arXiv 2025

  4. [4]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://arxiv.org/abs/1911.02116 Unsupervised cross-lingual representation learning at scale . Preprint, arXiv:1911.02116

    arXiv 2020

  5. [5]

    Mehrdad Farahani and Richard Johansson. 2024. https://arxiv.org/abs/2410.05162 Deciphering the interplay of parametric and non-parametric memory in retrieval-augmented language models . Preprint, arXiv:2410.05162

    work page Pith review arXiv 2024

  6. [6]

    Yu Fu, Haz Sameen Shahgir, Hui Liu, Xianfeng Tang, Qi He, and Yue Dong. 2025. https://arxiv.org/abs/2504.08202 Harnessing the unseen: The hidden influence of intrinsic knowledge in long-context language models . Preprint, arXiv:2504.08202

    work page arXiv 2025

  7. [7]

    Muhan Gao, TaiMing Lu, Kuai Yu, Adam Byerly, and Daniel Khashabi. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.447 Insights into LLM long-context failures: When transformers know but don`t tell . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7611--7625, Miami, Florida, USA. Association for Computational Linguistics

    doi:10.18653/v1/2024.findings-emnlp.447 2024

  8. [8]

    Yair Gat, Nitay Calderon, Amir Feder, Alexander Chapanin, Amit Sharma, and Roi Reichart. 2023. https://arxiv.org/abs/2310.00603 Faithful explanations of black-box nlp models using llm-generated counterfactuals . Preprint, arXiv:2310.00603

    arXiv 2023

Show all 32 references
  1. [9]

    Reshmi Ghosh, Rahul Seetharaman, Hitesh Wadhwa, Somyaa Aggarwal, Samyadeep Basu, Soundararajan Srinivasan, Wenlong Zhao, Shreyas Chaudhari, and Ehsan Aghazadeh. 2024. https://arxiv.org/abs/2410.00857 Quantifying reliance on external information over parametric knowledge during...

    2024 arXiv

  2. [10]

    Yuxiao Hu, Qian Li, Dongxiao Zhang, Jinyue Yan, and Yuntian Chen. 2025. https://arxiv.org/abs/2501.03747 Context-alignment: Activating and enhancing llm capabilities in time series . Preprint, arXiv:2501.03747

    2025

  3. [11]

    Evgenii Kortukov, Alexander Rubinstein, Elisa Nguyen, and Seong Joon Oh. 2024. https://arxiv.org/abs/2404.16032 Studying large language model behaviors under context-memory conflicts with real documents . Preprint, arXiv:2404.16032

    2024 arXiv

  4. [12]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

    2004

  5. [13]

    Smith, Sophie Lebrecht, Yejin Choi, Hannaneh Hajishirzi, Ali Farhadi, and Jesse Dodge

    Jiacheng Liu, Taylor Blanton, Yanai Elazar, Sewon Min, YenSung Chen, Arnavi Chheda-Kothary, Huy Tran, Byron Bischoff, Eric Marsh, Michael Schmitz, Cassidy Trier, Aaron Sarnat, Jenna James, Jon Borchardt, Bailey Kuehl, Evie Cheng, Karen Farley, Sruthi Sreeram, Taira Anderson, D...

    2025 arXiv

  6. [14]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023 a . https://arxiv.org/abs/2307.03172 Lost in the middle: How language models use long contexts . Preprint, arXiv:2307.03172

    2023 arXiv

  7. [15]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023 b . https://arxiv.org/abs/2303.16634 G-eval: Nlg evaluation using gpt-4 with better human alignment . Preprint, arXiv:2303.16634

    2023 arXiv

  8. [16]

    Yi Liu, Lianzhe Huang, Shicheng Li, Sishuo Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. 2023 c . https://arxiv.org/abs/2311.08147 Recall: A benchmark for llms robustness against external counterfactual knowledge . Preprint, arXiv:2311.08147

    2023 arXiv

  9. [17]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1907.11692 Roberta: A robustly optimized bert pretraining approach . Preprint, arXiv:1907.11692

    2019 arXiv

  10. [18]

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. https://arxiv.org/abs/2305.14251 Factscore: Fine-grained atomic evaluation of factual precision in long form text generation . Preprint, ...

    2023 arXiv

  11. [19]

    Olubusayo Olabisi, Aaron Hudson, Antonie Jetter, and Ameeta Agrawal. 2022. https://aclanthology.org/2022.coling-1.542/ Analyzing the dialect diversity in multi-document summaries . In Proceedings of the 29th International Conference on Computational Linguistics, pages 6208--62...

    2022

  12. [20]

    Chen, and Shafiq Joty

    Mathieu Ravaut, Aixin Sun, Nancy F. Chen, and Shafiq Joty. 2024. https://arxiv.org/abs/2310.10570 On context utilization in summarization with large language models . Preprint, arXiv:2310.10570

    2024 arXiv

  13. [21]

    Kuniaki Saito, Kihyuk Sohn, Chen-Yu Lee, and Yoshitaka Ushiku. 2024. https://arxiv.org/abs/2402.12170 Where is the answer? investigating positional bias in language model knowledge extraction . Preprint, arXiv:2402.12170

    2024 arXiv

  14. [22]

    Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. 2024. To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning. arXiv preprint arXiv:2409.12183

    2024 arXiv

  15. [23]

    Jetter, and Ameeta Agrawal

    Yufei Tao, Adam Hiatt, Erik Haake, Antonie J. Jetter, and Ameeta Agrawal. 2024. https://arxiv.org/abs/2409.08435 When context leads but parametric memory follows in large language models . Preprint, arXiv:2409.08435

    2024 arXiv

  16. [24]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  17. [25]

    Rongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei Xu. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.486 Knowledge conflicts for LLM s: A survey . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, page...

    2024 doi

  18. [26]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  19. [27]

    Hao Zhang, Yuyang Zhang, Xiaoguang Li, Wenxuan Shi, Haonan Xu, Huanshuo Liu, Yasheng Wang, Lifeng Shang, Qun Liu, Yong Liu, and Ruiming Tang. 2024 a . https://arxiv.org/abs/2405.19010 Evaluating the external and parametric knowledge fusion of large language models . Preprint, ...

    2024 arXiv

  20. [28]

    Huajian Zhang, Yumo Xu, and Laura Perez-Beltrachini. 2024 b . https://aclanthology.org/2024.eacl-long.102/ Fine-grained natural language inference based faithfulness evaluation for diverse summarisation tasks . In Proceedings of the 18th Conference of the European Chapter of t...

    2024

  21. [29]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://arxiv.org/abs/1904.09675 Bertscore: Evaluating text generation with bert . Preprint, arXiv:1904.09675

    2020 arXiv

  22. [30]

    Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. https://arxiv.org/abs/2104.05938 Qmsum: A new benchmark for query-based multi-domain meeting summarization . Preprint,...

    2021 arXiv

  23. [31]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  24. [32]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.