Pith. sign in

REVIEW 4 major objections 7 minor 36 references

Fact omission in air-gapped LLM agents is a whole-pipeline failure: this paper shows that under a 75,476-trial controlled benchmark, 73.4% of omitted facts are lost in deterministic software layers (L0–L3) before the model ever sees them, a

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 04:42 UTC pith:Y5XKHU76

load-bearing objection Worth reading for the taxonomy and diagnostic harness; the headline 73.4% waterfall number is a model-based benchmark decomposition, not a measured end-to-end attribution, and raw data is still missing. the 4 major comments →

arxiv 2607.22448 v2 pith:Y5XKHU76 submitted 2026-07-24 cs.MA

Where Facts Go Missing: A Layerwise Taxonomy and Per-Layer Attribution of Information Omission in Air-Gapped LLMAgent Pipelines

Santhiya Rajan This is my paper
classification cs.MA
keywords omissionLLM agentslayerwise attributionair-gapped deploymentpipeline diagnosislong-context retrievalKV cache quantizationtool protocol faults
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Fact omission in air-gapped LLM agents is usually a whole-pipeline failure, not a model cognitive failure. The paper defines a nine-layer taxonomy (L0–L8) from ingestion to the agent loop, instruments every boundary with checkpoint taps and unique canary needles, and composes per-layer losses into a conditional omission waterfall that assigns each lost fact to the first layer where it disappears. Under a deliberately weighted 75,476-trial synthetic benchmark, the overall omission rate is 0.574 and deterministic software layers L0–L3 account for 73.4% of the loss; context length is the strongest behavioral association (OR 7.43 for 32k vs 2k). The practical reason to care: operators of on-prem agents should verify transport, pagination, truncation, and loop coverage before attributing a missing fact to — or retraining — the model.

Core claim

The paper's central claim is that omission in a deployed agent can be a property of the whole pipeline, not only of the model. Output-only evaluation cannot distinguish mechanisms: a needle can be deleted by a PHI redactor before inference, dropped because pagination was not followed, evicted under context pressure, suppressed during generation, or lost when the orchestrator compacts history. To make this measurable, the paper defines nine pipeline layers, treats L0–L3 as deterministic (audited at checkpoint taps) and L4–L8 as behavioral (estimated by controlled contrasts and a forced-logprob probe), and composes them into a conditional omission waterfall. Applying the waterfall to the commi

What carries the argument

The central object is the conditional omission waterfall (Eq. 1): with N_i evaluation opportunities entering layer i and L_i first lost there, the conditional layer rate is ω_i = L_i/N_i and the unconditional contribution to total omission is c_i = ω_i ∏_{j<i}(1−ω_j); total omission O_R = 1 − ∏(1−ω_i). This makes omission attribution pipeline-level and mutually exclusive: each fact is charged to the first layer where it disappears, and deterministic layers are verified exactly by checkpoint taps (T0–T7) with unique alphanumeric canaries, while behavioral layers are estimated from controlled experiments and a heuristic forced-logprob decomposition.

Load-bearing premise

The load-bearing premise is that each omitted fact can be attributed to exactly the first layer where it disappears, and that layer failures do not interact — if upstream truncation moves where surviving facts sit in the context and therefore changes downstream retrieval, the waterfall's per-layer shares are not clean causal contributions.

What would settle it

Disable the pagination-following step (L1) in the same protocol and rerun the waterfall: if total omission O_R drops by roughly the L1 share (0.175 of those entering L1), the fixed-order attribution holds; if the drop is materially smaller or larger, layers interact and the conditional-independence assumption fails. A second check: take needles that survive all deterministic layers and compare the forced-logprob heuristic labels against blind human labels of why each failure occurred; agreement would validate the behavioral layer split, disagreement would call the L5/L6/L7 rates into question.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Verify deterministic layers first: 73.4% of benchmark omission loss is assigned to ingestion, tool protocol, orchestrator, and tokenizer layers, so fixing pagination, schema truncation, and string-slicing should be the first debugging step in on-prem agents.
  • Output-level detectors alone cannot localize failure; checkpoint taps with canary needles give operators an actionable target, and the same instrumentation can be turned into a runtime alarm when a canary does not survive to the final prompt.
  • Configuration choices are omission drivers: q4 KV cache versus f16 (OR 2.25) and scaled versus native RoPE (OR 1.99) are both associated with higher omission, so hardware-budget and context-stretch decisions should be tested with needles rather than assumed safe.
  • Long context is a behavioral risk even with perfect transport: omission odds at 32k are 7.43 times those at 2k, so widening the window without needle-testing can quietly reduce reliability.
  • End-to-end agent reliability is low in the pilot: 57.8% of real-source traces are unsuccessful, and after excluding execution errors 50.9% remain unsuccessful, suggesting the pipeline-level problem is not confined to the synthetic stress test.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the waterfall's checkpoint-first attribution is right, per-layer shares are localization, not causal decomposition: a double-deletion or interaction experiment (deleting a needle at two layers and seeing where it is counted) would test how much the layers truly interact.
  • The canary-injection technique could be turned into a production runtime omission monitor, alarming whenever an injected canary is absent at a checkpoint or in the final answer, giving real-time coverage alerts without a judge model.
  • Because the raw main-sweep trial files are not included in this checkout, the committed aggregates can only be checked for internal arithmetic; archiving the trials with model hashes and engine commits is required for the attribution result to be independently regenerated.
  • The benchmark's 73.4% L0–L3 share is allocation-driven; a field study with natural workload weights is the direct next test, and the paper explicitly defers that to future scope.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes a nine-layer taxonomy (L0–L8) for information omission in air-gapped LLM agent pipelines, spanning ingestion, tool protocol, orchestration, tokenization, engine, attention, priors, decoding, and agent loop. It describes an instrumented harness with checkpoint taps and a conditional omission waterfall (Eq. 1) to attribute each failure to the first layer where a needle disappears. The main empirical claims are: (i) a weighted synthetic benchmark of 75,476 trials yields O_R = 0.574 (95% CI 0.571–0.578), with deliberately injected L0–L3 deterministic faults accounting for 73.4% of waterfall loss; (ii) context length shows the largest behavioral association (OR = 7.43, 32k vs. 2k); (iii) server-profile follow-ups associate q4 KV cache and scaled RoPE with higher omission; (iv) a 372-trial real-agent pilot suggests 50.9–57.8% unsuccessful traces. The authors repeatedly disclaim that the benchmark allocation is deliberately weighted and that cross-model and engine comparisons are descriptive/confounded. The paper candidly lists limitations, including unvalidated Phase-D logprob labels, interaction analyses deferred to future work, and absent raw trial files.

Significance. If the results hold, the paper makes a useful methodological contribution: a concrete, instrumented attribution harness and a nine-layer taxonomy that operationalizes the claim that omission in deployed agents can be a pipeline-level property, not just a model-level failure. The deterministic L0–L3 checkpoint audit is exact and reproducible in principle, and the waterfall formula (Eq. 1) is a transparent definitional decomposition. The paper is also commendably candid about the conditional and non-prevalence nature of the 73.4% figure, the heuristic nature of behavioral labels, the confounding in cross-model comparisons, and the missing raw artifacts. However, the headline O_R and layer shares rest on the disclosed fault injection and on a product-of-independent-rates assumption that the paper does not test; the behavioral layer labels (L5–L7) depend on a Phase-D rule that the paper itself calls unvalidated. The real-pilot numbers are end-to-end reliability measures with execution errors included, not clean omission rates. Overall the contribution is a promising diagnostic framework and a controlled stress-test decomposition, not an estimate of production prevalence or causal archi

major comments (4)
  1. [Eq. 1 and 'The omission waterfall' (Attribution Methodology)] The waterfall multiplies conditional rates ω0–ω8 estimated from separate sub-experiments (Phase A for L0–L3, Phase C/D for L5–L7, Phase E for L8) as if layer losses were conditionally independent given survival of upstream layers. No trial in the 75,476 sweep passes through all layers with all faults simultaneously. The paper explicitly fixes layer order to avoid double counting, but that does not make the conditional rates independent of upstream interventions. Example: if L1 pagination drops a needle that would also have been missed at L5 because of lost-in-the-middle, checkpoint-first counts it as L1; this is a valid first-failure statistic only if the benchmark's L1 faults are exactly the ones that would occur first in a natural pipeline. Moreover, truncation at L2 can change context length and needle depth for surviving needles, directly altering L5 retrieval. The paper's own OR=7.4
  2. [Phase D and Table 5] Table 5 reports ω5 = 0.189, ω6 = 0.012, ω7 = 0.018 as if they were measured layer rates, but these are produced by a Phase-D rule-based heuristic (mean forced logprob above −2.0 → pruned_L7; else conflict needle → L6; else → L5). The paper states Phase D is 'a rule-based diagnostic heuristic, not a causal per-token decomposition' and that the implementation 'scores the expected continuation' but 'the recorded rank does not currently determine the label.' The Limitations section acknowledges the −2.0 threshold 'should be validated against labeled attention, prior, and decoding failures.' Given that L5 is the largest behavioral conditional rate and the second-largest contributor after L0–L3, the headline decomposition is not independent of an unvalidated heuristic. This should be clearly flagged as a key uncertainty in the abstract/conclusion, not only in the limitations.
  3. [Table 6 and Results: 'Context length has the largest reported behavioral association'] The OR = 7.43 for length is presented as the strongest association, but it is derived from gold-path Phase-C rows where deterministic layers are lossless. The paper uses outcome labels based on case-normalized expected-answer substring matching, including paraphrase trials. The paper notes the paraphrase contrast is 'partly a measurement effect rather than pure semantic retrieval,' but the same concern applies to the length effect insofar as longer contexts make substring-match failures more likely even with faithful answers. The odds ratios are 'associations conditional on the included factors, not percentages of omission caused,' which is good, but the causal framing in the abstract and introduction should be tempered by the measurement caveat.
  4. [Results and Table 8 (real-data pilot)] The pilot's headline numbers (57.8% unsuccessful, 50.9% after excluding execution errors) are end-to-end framework-conditioned reliability, not omission rates. The paper correctly says the measure 'is broader than literal omission' and includes recursion-limit errors and filing-not-found errors. However, the abstract states '57.8% of traces are unsuccessful overall and 50.9% remain unsuccessful after excluding execution errors' as if this were a well-defined omission result. The source/provider rates are severely confounded (domain, source, sample construction, framework, model), as the paper notes. These numbers should be reported only as pilot reliability descriptors, and the abstract should not invite readers to interpret them as omission prevalence.
minor comments (7)
  1. [Abstract and Introduction] The abstract says 'The committed aggregate artifacts describe 75,476 synthetic stress-test trials' but the raw files are absent from the checkout. The phrase 'committed aggregate artifacts' is confusing; better to say 'aggregate reports describe.'
  2. [Section 3, Table 1] L4 is labeled 'Engine runtime & memory' and classified 'Behav.', but L4 is partially deterministic (context shifts, preemption, quantized KV). The text acknowledges this in Phase B, but the taxonomy table's binary split (Deterministic vs. Behav.) is slightly misleading; the text should clarify that L4 contains both deterministic and behavioral sub-mechanisms.
  3. [Phase B description] Phase B contains B1/B2/B3, but the main sweep's L4 conditional rate in Table 5 is reported as 0.000. The paper says 'Evicted (L4) 0.000' in Table 8, suggesting no L4 events fired in the main sweep, yet the waterfall includes an L4 rate of exactly zero. This is surprising given the reported B2 quantized-KV effect (OR=2.25) and the noted SWA checkpoint-restoration bug class. Please clarify whether L4 = 0 arises because all L4 contrasts were run as separate server-profile follow-ups rather than in the main sweep, or because no context-shift events were observed in the main sweep.
  4. [Phase E / L8 attribution] ω8 is set to 'end-to-end omission minus the single-turn omission predicted by the B–D model on matched tasks,' which is an attribution by subtraction. The paper does not provide the matched-task details or the single-turn baseline value. This should be reported with the same transparency as the other rates.
  5. [Statistical reporting] Wilson intervals are given for binomial rates, but the paper acknowledges that 'repeated needles and heterogeneous model-specific effects remain sources of dependence.' The OR=7.43 CI (5.44–10.15) is model-based rather than cluster-robust; the paper says so in the Analysis section, but the abstract's '95% CI' for O_R should perhaps carry the same caveat, since the intervals do not capture cell weighting or model-selection uncertainty.
  6. [Typography / notation] The paper uses 'O_R' with a subscript and also 'OR' for odds ratios; the two are visually confusable. Rename the benchmark omission rate (e.g., ρ or μ) to avoid collision with 'odds ratio (OR).' Also, the conclusion says 'completing the experiments in Section is necessary'—the section reference is missing.
  7. [Table 5 note] The label sources ('checkpoint', 'telemetry', 'heuristic', 'task contrast') are useful, but the Wilson intervals for L0–L3 are described as 'from the committed aggregate.' Since the raw trials are absent, it is unclear how the intervals were computed (e.g., binomial n per cell). State the per-cell n or total n for each phase.

Circularity Check

0 steps flagged

No circularity: the waterfall is an explicit accounting identity, the 73.4% L0-L3 share is a disclosed benchmark-allocation descriptor, and the behavioral labels are declared heuristics rather than hidden fits or self-citation chains.

full rationale

The paper's derivation chain is self-contained and does not reduce to its own inputs. Equation (1) is an explicit accounting identity: c_i = ω_i ∏(1-ω_j) is just the checkpoint-first survival decomposition, and O_R = 1-∏(1-ω_i) is the algebraically implied total. The paper labels this "implied by rates" in Table 5 and does not present it as a fitted prediction. The headline 73.4% L0-L3 share is explicitly conditional on the deliberately weighted fault allocation; the text repeatedly disclaims generalization ("This is a controlled benchmark decomposition, not an estimate of omission prevalence in production"). Phase A rates are checkpoint counts from injected faults, not parameters fitted to produce O_R. Phase D's labeling rule (conflict needle → unsurfaced_prior_L6, other families → unsurfaced_positional_L5) is disclosed as a "rule-based diagnostic heuristic, not a causal per-token decomposition," and it feeds the behavioral ledger rather than the core waterfall identity. The reference list contains no prior work by the present author, so no self-citation is load-bearing; the cited long-context, quantization, tool-layer, and conflict literature is external. The Limitations section explicitly defers interaction analyses, raw-trial archiving, and validation of Phase-D labels, and the conclusion states that the evidence does not establish production prevalence or causal architectural effects. These are validity and reproducibility limitations, not circular reductions; no step is equivalent by construction to its input.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The paper's headline rates are conditional on the authors' hand-chosen fault injection and heuristic labeling rule, not on a production workload. The waterfall independence assumption and the Phase-D logprob rule are the two load-bearing premises for the attribution numbers. No new physical or conceptual entities are introduced beyond the taxonomy layers, which are descriptive categories with no independent falsifiable handles.

free parameters (3)
  • benchmark fault-injection rates for L0-L3 = ω0=0.160, ω1=0.175, ω2=0.165, ω3=0.000
    Hand-chosen to stress deterministic middleware; directly determines the 73.4% L0-L3 share. Disclosed as 'deliberately weighted fault allocation'.
  • Phase-D forced-logprob threshold = -2.0
    Hand-chosen boundary separating L7 pruning labels from L5/L6; changing it changes behavioral layer attribution.
  • benchmark cell weights across context length/depth/family = not fully reported
    O_R=0.574 is allocation-weighted; exact trial mix is not given in the paper, so the aggregate rate is conditional on unspecified weights.
axioms (4)
  • domain assumption Layer losses are conditionally independent and strictly ordered: N_{i+1}=N_i-L_i.
    Underpins Eq. 1 and the checkpoint-first attribution; not tested for interactions. Section 'The omission waterfall'.
  • domain assumption Checkpoint taps and canary checks do not perturb the pipeline and exactly capture needle presence at each boundary.
    Instrumentation must be non-invasive; needle canaries checked by exact match at T1-T5. Section 'Instrumentation'.
  • domain assumption The Phase-D rule (mean forced logprob threshold -2.0 and needle-family labels) correctly identifies the causal layer for behavioral failures.
    The paper calls it 'a rule-based diagnostic heuristic'; L5/L6/L7 rates rest on it. Section 'Phase D — logit decomposition'.
  • domain assumption Expected-answer substring matching is a valid retrieval label including for paraphrase/conflict needles.
    The paper notes the paraphrase contrast is partly a measurement effect; the OR=3.65 depends on this scoring choice. Section 'Sources, domains, and offline evaluation'.

pith-pipeline@v1.3.0-alltime-deepseek · 12456 in / 16663 out tokens · 169307 ms · 2026-08-01T04:42:01.988743+00:00 · methodology

0 comments
read the original abstract

Air-gapped and on-premises language-model agents can silently omit decision-critical facts at any boundary between source ingestion and final answer generation. We present a nine-layer taxonomy (L0-L8), an instrumented attribution harness, and a conditional omission waterfall that distinguishes deterministic software loss from behavioral non-retrieval. We analyze 75,476 controlled synthetic trials spanning five open-weight model configurations and two inference engines, together with a separate 372-trial real-agent pilot covering FHIR, PubMed, and SEC-EDGAR sources with LangChain and ADK orchestration. The weighted synthetic benchmark yields an omission rate of 0.574 (95% CI: 0.571-0.578); deliberately injected deterministic faults at L0-L3 account for 73.4% of weighted loss under the benchmark allocation. Increasing context length is most strongly associated with omission (odds ratio 7.43, 95% CI: 5.44-10.15). Completed server-profile analyses associate q4 KV cache and scaled RoPE with higher omission. In the real-agent pilot, 57.8% of traces are unsuccessful overall and 50.9% remain unsuccessful after excluding execution errors. These results establish pipeline-level attribution in a controlled stress test, but benchmark allocations, confounded model comparisons, and heuristic behavioral labels do not measure production prevalence or causal architectural effects.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 24 linked inside Pith

  1. [3]

    arXiv:2606.22528

    Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agents. arXiv:2606.22528. Colaco, A. G.; and Lahjouji, N

    Pith/arXiv arXiv

  2. [4]

    arXiv:2607.08032

    What to Keep, What to Forget: A Rate–Distortion View of Memory Compaction in LLMs and Agents. arXiv:2607.08032. Croxford, E.; Gao, Y.; Pellegrino, N.; Wong, K.; Wills, G.; First, E.; Liao, F.; Goswami, C.; Patterson, B.; and Afshar, M

    Pith/arXiv arXiv

  3. [5]

    arXiv:2510.21118

    The Gray Zone of Faithfulness: Taming Ambiguity in Unfaithfulness Detection. arXiv:2510.21118. Du, Y.; Tian, M.; Ronanki, S.; Rongali, S.; Bodapati, S.; Galstyan, A.; Wells, A.; Schwartz, R.; Huerta, E. A.; and Peng, H

    arXiv

  4. [6]

    arXiv:2510.05381

    Context Length Alone Hurts LLM Performance Despite Perfect Retrieval. arXiv:2510.05381. Ebrahimzadeh, A.; and Salili, S. M

    arXiv

  5. [7]

    arXiv:2601.02023

    Not All Needles Are Found:HowFactDistributionandDon’tMakeItUpPromptsShape Retrieval, Reasoning, and Hallucination in Long-Context LLMs. arXiv:2601.02023. Fei, X.; Zheng, X.; and Feng, H

    Pith/arXiv arXiv

  6. [8]

    arXiv:2506.01056

    MCP-Zero: Active Tool Discovery for Autonomous LLM Agents. arXiv:2506.01056. Feng, Y.; Guo, H.; Lv, J.; Zhou, S. K.; and Xie, X

    Pith/arXiv arXiv

  7. [9]

    arXiv:2510.13334

    Taming the Fragility of KV Cache Eviction in LLM Inference. arXiv:2510.13334. Gao,Y.;Xiong,Y.;Wu,W.;Huang,Z.;Li,B.;andWang,H.2025. U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-in-a-Haystack.ACM Transactions on Information Systems. https://doi.org/10.1145/3786609. ggml-org. 2026a. Context Shift Not Working with Gemma-4 andQuantizedKVCache...

    arXiv 2025

  8. [10]

    arXiv:2511.13900

    What Works for ’Lost-in-the-Middle’ in LLMs? A Study on GM-Extract and Mitigations. arXiv:2511.13900. Hadeliya, T.; Jauhar, M. A.; Sakpal, N.; and Cruz, D

    arXiv

  9. [11]

    arXiv:2512.02445

    When Refusals Fail: Unstable Safety Mechanisms in Long-Context LLM Agents. arXiv:2512.02445. Hasan, M. M.; Li, H.; Rajbahadur, G. K.; Adams, B.; and Hassan, A.E.2026. ModelContextProtocol(MCP)ToolDescriptionsAre Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions. arXiv:2602.14878. Hooper,C.;Kim,S.;Mohammadzadeh,H.;Mahoney,M...

    arXiv 2026

  10. [13]

    Jiang, R.; Wu, T.; Wang, Y.; Zhu, B.; and Huang, L

    RULER: What’s the Real Context Size of Your Long-Context Language Models? arXiv:2404.06654. Jiang, R.; Wu, T.; Wang, Y.; Zhu, B.; and Huang, L

    Pith/arXiv arXiv

  11. [14]

    arXiv:2606.10298

    From Context-AwaretoConflict-Aware:GeneralizingContrastiveDecod- ing for Knowledge Conflict in LLMs. arXiv:2606.10298. Lee, H.; Park, S.; Lee, S.; Seo, J.; Lee, J.; Yoo, S.; Kim, M.; Na, C.; Wang, Z.; Golkhou, Z.; Kim, M.; Sabanis, S.; Lopez-Lira, A.; Mehta, D.; Lee, S.; Choi, C.; Ahn, W.; and Lee, Y

    Pith/arXiv arXiv

  12. [15]

    arXiv:2606.29251

    WhenSummariesDistortDecisions:InformationFidelityinLLM- Compressed Financial Analysis. arXiv:2606.29251. Li, S.; Ning, X.; Wang, L.; Liu, T.; Shi, X.; Yan, S.; Dai, G.; Yang, H.; and Wang, Y

    Pith/arXiv arXiv

  13. [16]

    arXiv:2402.18158

    Evaluating Quantized Large Language Models. arXiv:2402.18158. Li, Y.; Hou, Z.; Jing, Y.; Tang, J.; and Dong, Y

    Pith/arXiv arXiv

  14. [17]

    arXiv:2607.05378

    Com- pactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents. arXiv:2607.05378. Liu, N. F.; Lin, K.; Hewitt, J.; Paranjape, A.; Bevilacqua, M.; Petroni, F.; and Liang, P

    Pith/arXiv arXiv

  15. [19]

    arXiv:2605.12366

    Classifier Context Rot: Monitor Performance Degrades with Context Length. arXiv:2605.12366. Microsoft

    Pith/arXiv arXiv

  16. [20]

    https://microsoft.github.io/mcscatblog/posts/mcp-resources-as- tool-inputs/

    Pass MCP Resources, Not Raw Data. https://microsoft.github.io/mcscatblog/posts/mcp-resources-as- tool-inputs/. Accessed: 2026-07-16. Modarressi, A.; Deilamsalehy, H.; Dernoncourt, F.; Bui, T.; Rossi, R. A.; Yoon, S.; and Schütze, H

    2026

  17. [21]

    arXiv:2502.05167

    NoLiMa: Long-Context Evaluation Beyond Literal Matching. arXiv:2502.05167. Morph

    Pith/arXiv arXiv

  18. [22]

    https://www.morphllm

    MCP Output Too Large. https://www.morphllm. com/mcp-output-too-large. Accessed: 2026-07-16. Muller, L. K.; Bich, P.; Boretti, C.; Chang, H.-M.; Zhuang, J.; and Cavigelli, L

    2026

  19. [23]

    arXiv:2606.03458

    KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks. arXiv:2606.03458. Normand, O.; Borsi, E.; Fruin, M.; Walker, L. E.; Heagerty, J.; Holmes, C. C.; Avery, A. J.; Buchan, I. E.; and Coppock, H

    Pith/arXiv arXiv

  20. [24]

    arXiv:2512.21127

    A Real-World Evaluation of LLM Medication Safety Reviews in NHS Primary Care. arXiv:2512.21127. Oukelmoun, A.; Semmar, N.; de Chalendar, G.; Cormi, C.; Oukel- moun,M.;Vibert,E.;andAllard,M.-A.2025. DetectingOmissions inLLM-GeneratedMedicalSummaries.InProceedingsofthe2025 ConferenceonEmpiricalMethodsinNaturalLanguageProcessing: Industry Track, 325–337. htt...

    arXiv 2025

  21. [25]

    arXiv:2606.05339

    A Taxonomy of Runtime Faults in Model Context Protocol Servers. arXiv:2606.05339. Peng,H.;Tang,J.;Zeng,W.;Xu,H.;andZhao,X.2026. Navigating UnreliableParametricandContextualKnowledge:ExplicitKnowl- edge Conflict Resolution for LLM Inference. arXiv:2606.20245. Piya, F. L.; and Beheshti, R

    Pith/arXiv arXiv 2026

  22. [26]

    arXiv:2602.20040

    AgenticSum: An Agentic Inference-Time Framework for Faithful Clinical Text Summariza- tion. arXiv:2602.20040. Schumacher, E.; Rosenthal, D.; Naik, D.; Nair, V.; Price, L.; Tso, G.; and Kannan, A

    arXiv

  23. [28]

    arXiv:2512.20182

    FaithLens: Detecting and Explaining Faithfulness Hallucination. arXiv:2512.20182. Su,Z.;Zhang,J.;Qu,X.;Zhu,T.;Li,Y.;Sun,J.;Li,J.;Zhang,M.; and Cheng, Y

    Pith/arXiv arXiv

  24. [29]

    arXiv:2408.12076

    ConflictBank: A Benchmark for Evaluating theInfluenceofKnowledgeConflictsinLLMs. arXiv:2408.12076. Sun, K.; Bai, F.; and Dredze, M

    Pith/arXiv arXiv

  25. [30]

    arXiv:2506.06485

    Task Matters: Knowledge RequirementsShapeLLMResponsestoContext-MemoryConflict. arXiv:2506.06485. Taraghi, M.; Morovati, M. M.; and Khomh, F

    Pith/arXiv arXiv

  26. [31]

    arXiv:2603.05637

    Real Faults in Model Context Protocol (MCP) Software: a Comprehensive Tax- onomy. arXiv:2603.05637. Wang, W.; Zhang, S.; Ren, Y.; Duan, Y.; Li, T.; Liu, S.; Hu, M.; Chen, Z.; Zhang, K.; Lu, L.; Zhu, X.; Luo, P.; Qiao, Y.; Dai, J.; Shao, W.; and Wang, W

    arXiv

  27. [32]

    arXiv:2406.11230

    Needle in a Multimodal Haystack. arXiv:2406.11230. Wu,K.;Wu,E.;andZou,J.2024. ClashEval:QuantifyingtheTug- of-War Between an LLM’s Internal Prior and External Evidence. arXiv:2404.10198. Xia, S.; Wang, Y.; Huang, Z.; and Liu, P

    Pith/arXiv arXiv 2024

  28. [33]

    Zeng,W.;Huang,Y.;andHe,J.2026.LOCA-bench:Benchmarking LanguageAgentsUnderControllableandExtremeContextGrowth

    Diagnosing and MitigatingContextRotinLong-horizonSearch.arXiv:2606.29718. Zeng,W.;Huang,Y.;andHe,J.2026.LOCA-bench:Benchmarking LanguageAgentsUnderControllableandExtremeContextGrowth. arXiv:2602.07962. Zhang,Q.;Xiang,Z.;Xiao,Y.;Wang,L.;Li,J.;Wang,X.;andSu, J

    Pith/arXiv arXiv 2026

  29. [34]

    arXiv:2506.08938

    FaithfulRAG: Fact-Level Conflict Modeling for Context- Faithful Retrieval-Augmented Generation. arXiv:2506.08938. Zhang, X.; Chen, Y.; Hu, S.; Xu, Z.; Chen, J.; Hao, M. K.; Han, X.; Thai, Z. L.; Wang, S.; Liu, Z.; and Sun, M

    Pith/arXiv arXiv

  30. [35]

    arXiv:2402.13718

    ∞Bench: Extending Long Context Evaluation Beyond 100K To- kens. arXiv:2402.13718. Zhao,T.;Chen,J.;Zhang,S.;Zhu,H.;Lin,Q.;andLiu,J.2026. Ex- ploring Knowledge Conflicts for Faithful LLM Reasoning: Bench- mark and Method. arXiv:2604.11209. Zhao, Y.; Chen, W.; Xu, Z.; Patwardhan, M.; Wang, C.; Liu, Y.; Vig, L.; and Cohan, A

    Pith/arXiv arXiv 2026

  31. [36]

    InProceedings of the 63rd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), 12479–12491

    AbGen: Evaluating Large Language Models in Ablation Study Design and Evaluation for Scientific Research. InProceedings of the 63rd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), 12479–12491. https://aclanthology.org/2025.acl-long.611/

    2025

  32. [2021]

    InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

    Entity-Based Knowledge Conflicts in Question Answering. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. ArXiv:2109.05052. Martin, S.; and Roger, F

    Pith/arXiv arXiv 2021

  33. [2023]

    arXiv:2311.08303

    Extrinsically-Focused Evaluation of Omissions in Medical Summarization. arXiv:2311.08303. Shukla,A.;Yuan,Y.;Tamo,B.;Wang,Y.;Nnamdi,M.;Tan,S.;Li, J.;Marteau,B.;Willingham,B.;andWang,M.2026.Benchmarking LLM Summaries of Multimodal Clinical Time Series for Remote Monitoring. arXiv:2603.01557. Si,S.;Wang,Q.;Zhao,H.;Bai,Y.;Chen,G.;Luo,K.;Chen,G.;Qi, F.; Zhang,...

    Pith/arXiv arXiv 2026

  34. [2024]

    arXiv:2401.18079

    KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantiza- tion. arXiv:2401.18079. Hsieh, C.-P.; Sun, S.; Kriman, S.; Acharya, S.; Rekesh, D.; Jia, F.; and Ginsburg, B

    Pith/arXiv arXiv

  35. [2025]

    arXiv:2510.18691

    In- vestigating LLM Capabilities on Long Context Comprehension for Medical Question Answering. arXiv:2510.18691. Anthropic.2026. ToolResultsCappedat ∼25KTokenswitha2KB Preview. https://github.com/anthropics/claude-code/issues/45770. GitHub issue #45770. Accessed: 2026-07-16. Bedi,S.;Lin,B.;Zhou,A.Y.;Stanwyck,C.O.;Jindal,J.A.;Koyejo, S.; Stutz, D.; and Shah, N. H

    arXiv 2026

  36. [2026]

    arXiv:2606.08969

    CARE: A Conformal Safety Layer for Medical Summarization. arXiv:2606.08969. Chen, S

    Pith/arXiv arXiv