arxiv: 2605.10828 · v1 · submitted 2026-05-11 · 💻 cs.AI

Recognition: 2 theorem links

· Lean Theorem

The First Drop of Ink: Nonlinear Impact of Misleading Information in Long-Context Reasoning

Muhan Gao , Zih-Ching Chen , Kuan-Hao Huang

Authors on Pith no claims yet

Pith reviewed 2026-05-12 03:39 UTC · model grok-4.3

classification 💻 cs.AI

keywords long-context reasoninghard distractorsmisleading informationnonlinear effectsattention mechanismsretrieval-augmented generationcontext filteringperformance degradation

0 comments

The pith

A small proportion of misleading documents causes most of the performance drop in long-context language model reasoning.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that performance in long-context reasoning degrades nonlinearly as the share of hard distractors rises, with the steepest losses occurring from the first small fraction and only marginal further decline afterward. This pattern matters for retrieval-augmented generation and agent systems that routinely pull in semantically relevant but incorrect material alongside correct context. The authors demonstrate the effect by holding total context length fixed while varying the distractor proportion, and they tie it to attention patterns that over-weight the misleading items even when they are rare. Experiments also indicate that context shortening accounts for most filtering gains, while meaningful recovery demands bringing the hard-distractor share close to zero.

Core claim

As the proportion of hard distractors increases in fixed-length contexts, performance drops sharply within the first small fraction, while the remainder of the range yields only marginal additional decline. Hard distractors capture disproportionate attention even at small proportions, with diminishing marginal impact as their proportion grows. Filtering gains come mainly from context-length reduction rather than distractor removal, and substantial recovery requires reducing the hard-distractor proportion to near zero.

What carries the argument

The First Drop of Ink effect, which describes the nonlinear performance degradation caused by hard distractors capturing disproportionate attention at low proportions.

If this is right

Filtering gains come mainly from shortening overall context length rather than from removing the distractors themselves.
Substantial performance recovery requires bringing the hard-distractor proportion close to zero.
Attention mechanisms allocate disproportionate focus to semantically relevant misleading content even when that content is rare.
Upstream retrieval systems must achieve high precision to keep hard distractors out of the context.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Retrieval pipelines may need to favor precision over recall to avoid introducing even small numbers of hard distractors.
The effect could compound in multi-turn agent workflows where context accumulates across steps.
Models might be trained or prompted to down-weight content that is semantically close but factually inconsistent.

Load-bearing premise

That the controlled construction of hard distractors isolates their misleading effect without introducing uncontrolled changes in task difficulty or attention behavior.

What would settle it

If performance declines linearly rather than sharply at low proportions when the same models are tested on naturally retrieved documents instead of synthetically inserted hard distractors, the nonlinear pattern would be falsified.

Figures

Figures reproduced from arXiv: 2605.10828 by Kuan-Hao Huang, Muhan Gao, Zih-Ching Chen.

**Figure 1.** Figure 1: THE FIRST DROP OF INK effect. Left: Conventional linear assumption (top, red dashed line) versus empirically observed nonlinear degradation (bottom, blue curve): a small fraction of hard distractors is sufficient to severely degrade accuracy. Middle: Hard distractors receive similar attention logits as gold documents (8 ≈ 9 ≫ 1), dominating the softmax competition even at low proportions. Right: With 100 d… view at source ↗

**Figure 2.** Figure 2: Accuracy as a function of hard distractor proportion at 128K context length across three models (Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, and Qwen3-Next-80B-Instruct) on Natural Questions, TriviaQA, PopQA and HotpotQA. Across all configurations, introducing the first 10% of hard distractors (shaded region) causes steep performance degradation, while further increases yield only marginal decline. Despite… view at source ↗

**Figure 3.** Figure 3: Drop ratio of accuracy degradation across different context lengths, models, and datasets. The drop ratio measures the fraction of total performance loss that occurs in the first 10% of hard distractors. A linear degradation would yield 0.1. Negative values indicate the first 10% of hard distractors does not further degrade performance. Darker green indicates higher drop ratios (stronger nonlinearity), whi… view at source ↗

**Figure 4.** Figure 4: Two controlling factors of the theoretical attention curve (Remark 4.3). (a) When the margin gap ∆e − ∆h is fixed at 4, all curves share identical shape (same b/a = e 4 ) but differ in vertical position, controlled by 1/a. Faded lines extend into p < 0 to illustrate the shape equivalence. (b) When ∆h is fixed at 2, increasing ∆e enlarges the ratio b/a, producing more convex curves and amplifying the “First… view at source ↗

**Figure 5.** Figure 5: Empirical measurement of logit margins ∆e and ∆h on retrieval heads for Llama-3.1-8B-Instruct. Green bars show ∆e (margin to easy distractors) and brown bars show ∆h (margin to hard distractors). The gap ∆e − ∆h (annotated values) remains substantial across all hard proportions, with an average of 5.83. This validates the theoretical assumption ∆h ≪ ∆e. 5. Validation of Theoretical Explanation One natural … view at source ↗

**Figure 7.** Figure 7: Effect of temperature scaling on attention distribution. From left to right: pre-softmax logits, attention weights at τ = 1, and attention weights at τ < 1. Colors denote easy distractors, hard distractors, and target passage. Lower temperature sharpens the softmax, suppressing hard distractors while maintaining attention on the target. 6.2. Incremental Filtering of Hard Distractors Our main experiments v… view at source ↗

**Figure 8.** Figure 8: Filter Hard vs. Filter Random. Both strategies yield similar gains from removing the first 80K tokens, indicating that performance recovery comes from context length reduction rather than filtering strategy. The two strategies begin to diverge below 47K tokens (shaded region), where Filter Hard has a near-zero hard distractor proportion. This suggests that the gains from partial filtering are largely attri… view at source ↗

**Figure 9.** Figure 9: Proportional Reduction. Context is reduced from 131K to 27K while maintaining a fixed hard distractor ratio (20%, 50%, or 80% hard) by removing documents proportionally from each distractor category. Across both models and all datasets, the three curves follow similar trajectories: reducing the context length consistently improves performance, while varying the fixed hard ratio within this moderate-to-high… view at source ↗

**Figure 10.** Figure 10: Empirical measurement of logit margins ∆e and ∆h on retrieval heads for Llama-3.2-1B-Instruct. Green bars show ∆e (margin to easy distractors) and brown bars show ∆h (margin to hard distractors). The gap ∆e − ∆h remains substantial across all hard proportions, with an average of 6.52, which is more significant than the gap of the 8B model. This validates the theoretical assumption ∆h ≪ ∆e [PITH_FULL_IMAG… view at source ↗

read the original abstract

As large language models are increasingly deployed in retrieval-augmented generation and agentic systems that accumulate extensive context, understanding how distracting information affects long-context performance becomes critical. Prior work shows that semantically relevant yet misleading documents degrade performance, but the quantitative relationship between the proportion of distractors and performance remains unstudied. In this work, we systematically vary the hard-distractor proportion in fixed-length contexts, revealing a striking nonlinear pattern: as the proportion of hard distractors increases, performance drops sharply within the first small fraction, while the remainder of the range yields only marginal additional decline. We term this ''The First Drop of Ink'' effect, analogous to how a single drop of ink contaminates water. Our theoretical and empirical analyses grounded in attention mechanics show that hard distractors capture disproportionate attention even at small proportions, with diminishing marginal impact as their proportion grows. Controlled experiments further show that filtering gains mainly come from context-length reduction rather than distractor removal; substantial recovery requires reducing the hard-distractor proportion to near zero, highlighting the importance of upstream retrieval precision.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper maps a nonlinear performance drop from rising hard-distractor shares in fixed-length contexts and ties it to attention capture, but the length constraint creates a plausible confound with reduced relevant content.

read the letter

The key point is that performance falls sharply with the first small addition of hard distractors and then flattens, which the authors call the First Drop of Ink effect. They support this with proportion sweeps, attention measurements, and a filtering experiment showing that gains come mostly from shorter contexts rather than cleaner ones. That pattern and the upstream-precision takeaway are the main things worth noting for RAG and agent work.

Referee Report

1 major / 2 minor

Summary. The paper claims that in fixed-length long contexts for LLM reasoning, increasing the proportion of hard distractors (semantically relevant but misleading documents) produces a nonlinear 'First Drop of Ink' effect: performance drops sharply at small proportions and then flattens, explained by hard distractors capturing disproportionate attention even at low densities. This is supported by theoretical attention analysis and controlled experiments showing that filtering gains come mainly from context-length reduction rather than distractor removal, with substantial recovery only when hard-distractor proportion approaches zero.

Significance. If the central nonlinear pattern and attention-based mechanism hold after controlling for confounds, the result would be significant for retrieval-augmented generation and long-context systems, as it implies that even small fractions of misleading content can be disproportionately damaging and that upstream retrieval precision is critical. The attention-grounded explanation offers a concrete mechanistic account that could inform model design.

major comments (1)

[Experimental Setup] Experimental Setup (fixed-length context construction): Raising the hard-distractor proportion in fixed-length contexts necessarily reduces the count or density of relevant documents. This signal reduction alone can generate a steep early performance drop (loss of the first few critical pieces) followed by diminishing marginal losses, independent of any attention-capture mechanism. The theoretical and empirical attention analyses must therefore either hold relevant content constant across proportions or provide direct measurements that isolate disproportionate capture from this confound.

minor comments (2)

[Results] The abstract and results sections mention 'controlled experiments' and attention observations but do not report error bars, statistical tests, or data-split details; these should be added for reproducibility.
[Methods] Notation for 'hard distractors' vs. other distractor types is introduced without a clear table or definition in the methods; a dedicated table would improve clarity.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their valuable feedback, particularly on the experimental setup. We clarify below how our analyses address the potential confound raised.

read point-by-point responses

Referee: [Experimental Setup] Experimental Setup (fixed-length context construction): Raising the hard-distractor proportion in fixed-length contexts necessarily reduces the count or density of relevant documents. This signal reduction alone can generate a steep early performance drop (loss of the first few critical pieces) followed by diminishing marginal losses, independent of any attention-capture mechanism. The theoretical and empirical attention analyses must therefore either hold relevant content constant across proportions or provide direct measurements that isolate disproportionate capture from this confound.

Authors: We appreciate the referee's identification of this potential confound in the fixed-length context construction. Although raising the hard-distractor proportion reduces the count of relevant documents, our theoretical attention analysis models the softmax attention allocation explicitly as a function of the proportion, revealing that hard distractors receive attention weights far exceeding their share due to their semantic relevance, causing a sharp initial drop in attention to relevant content. This mechanism produces the observed nonlinear performance pattern. Empirically, the attention analyses include direct measurements of attention weights for hard distractors and relevant documents at each proportion level. These show disproportionate capture at small proportions, with the effect saturating as proportion increases, which we correlate with the performance metrics. This goes beyond simple signal reduction, as a pure loss-of-relevant model would not predict the specific attention reallocation patterns we observe and measure. We believe our existing theoretical and empirical results isolate the attention-capture mechanism as requested. revision: no

Circularity Check

0 steps flagged

No circularity: central claim is direct empirical observation of nonlinear performance drop under controlled distractor variation.

full rationale

The paper reports an observed nonlinear pattern ('First Drop of Ink') from systematically varying hard-distractor proportion in fixed-length contexts, supported by performance metrics and attention measurements. No derivation chain, fitted parameters renamed as predictions, or self-referential definitions are present; the effect is presented as an experimental result rather than a mathematical consequence of its own inputs. Attention-mechanics analyses are described as grounded in the same empirical setups without reducing to tautology. The setup is self-contained against external benchmarks via direct measurement.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The paper introduces no new free parameters or invented entities. It relies on standard transformer attention assumptions and empirical observation of distractor effects.

axioms (1)

domain assumption Hard distractors capture disproportionate attention even at small proportions in transformer models.
Invoked to explain the nonlinear performance pattern.

pith-pipeline@v0.9.0 · 5492 in / 1208 out tokens · 58198 ms · 2026-05-12T03:39:32.861040+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

Lemma 4.2 (Monotonicity and Convexity). Let f(p)=αi,J∗(p)=1/(1+(1−p)a+pb+c). Then f′(p)<0 (strictly decreasing) and f′′(p)>0 (strictly convex) for all p∈[0,1].
IndisputableMonolith/Foundation/AlphaCoordinateFixation.lean alpha_pin_under_high_calibration unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

hard distractors capture disproportionate attention even at small proportions, with diminishing marginal impact as their proportion grows

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

12 extracted references · 12 canonical work pages

[1]

Math-Shepherd: Verify and reinforce LLMs step-by-step without human annotations

Association for Computational Linguistics, 2024. doi: 10.18653/V1/2024.ACL- LONG.172. URL https://doi.org/10.18653/v1/2024.acl -long.172. Bianchi, O., Koretsky, M. J., Willey, M., Alvarado, C. X., Nayak, T., Asija, A., Kuznetsov, N., Nalls, M. A., Faghri, F., and Khashabi, D. Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Find, 202...

work page doi:10.18653/v1/2024.acl- 2024
[2]

URL https: //doi.org/10.1145/3626772.3657834

doi: 10.1145/3626772.3657834. URL https: //doi.org/10.1145/3626772.3657834. Ding, Y ., Zhang, L. L., Zhang, C., Xu, Y ., Shang, N., Xu, J., Yang, F., and Yang, M. LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens. In Salakhut- dinov, R., Kolter, Z., Heller, K. A., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.),Forty-first Int...

work page doi:10.1145/3626772.3657834 2024
[3]

doi: 10.18653/V1/2024.FINDINGS-EMNLP.447

Association for Computational Linguistics, 2024. doi: 10.18653/V1/2024.FINDINGS-EMNLP.447. URL https://doi.org/10.18653/v1/2024.fin dings-emnlp.447. Glass, M. R., Rossiello, G., Chowdhury, M. F. M., Naik, A., Cai, P., and Gliozzo, A. Re2G: Retrieve, Rerank, Gener- ate. In Carpuat, M., de Marneffe, M., and Ru´ız, I. V . M. (eds.),Proceedings of the 2022 Co...

work page doi:10.18653/v1/2024.findings-emnlp.447 2024
[4]

R e2 G : Retrieve, rerank, generate

doi: 10.18653/V1/2022.NAACL-MAIN.194. URL https://doi.org/10.18653/v1/2022.naa cl-main.194. Guha, N., Nyarko, J., Ho, D. E., R´e, C., Chilton, A., Aditya, K., Chohlas-Wood, A., Peters, A., Waldon, B., Rockmore, D. N., Zambrano, D., Talisman, D., Hoque, E., Surani, F., Fagan, F., Sarfaty, G., Dickinson, G. M., Porat, H., Hegland, J., Wu, J., Nudell, J., Ni...

work page doi:10.18653/v1/2022.naacl-main.194 2022
[5]

T rivia QA : A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

Association for Computational Linguistics. doi: 10.18653/v1/P17-1147. URL https://aclantho logy.org/P17-1147/. Kamradt, G. Needle In A Haystack - pressure testing LLMs,

work page doi:10.18653/v1/p17-1147
[6]

GitHub reposi- tory

URL https://github.com/gkamradt/ LLMTest_NeedleInAHaystack . GitHub reposi- tory. Ke, W., Zheng, Y ., Li, Y ., Xu, H., Nie, D., Wang, P., and He, Y . Large Language Models in Document Intelligence: A Comprehensive Survey, Recent Advances, Challenges, and Future Trends.ACM Trans. Inf. Syst., 44(1):18:1– 18:64, 2026. doi: 10.1145/3768156. URL https: //doi.o...

work page doi:10.1145/3768156 2026
[7]

Peng, B., Quesnelle, J., Fan, H., and Shippole, E

Accessed: 2025-01-20. Peng, B., Quesnelle, J., Fan, H., and Shippole, E. YaRN: Efficient Context Window Extension of Large Language Models. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: //openreview.net/forum?id=wHBfxhZu1u. Petroni, F., Piktus, A., Fan, A., ...

work page doi:10.18653/v1/2021.naa 2025
[8]

URL https://doi.org/10.48550/arXiv.2 511.06818. Ryan, N. Introducing a learnable temperature value into the softmax self-attention scores, 8 2024. Schick, T., Dwivedi-Yu, J., Dess`ı, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language Models Can Teach Themselves to Use Tools. In Oh, A., Naumann, T....

work page doi:10.48550/arxiv.2 2024
[9]

Yen, H., Gao, T., Hou, M., Ding, K., Fleischer, D., Izsak, P., Wasserblat, M., and Chen, D

URL https://openreview.net/forum ?id=WE_vluYUL-X. Yen, H., Gao, T., Hou, M., Ding, K., Fleischer, D., Izsak, P., Wasserblat, M., and Chen, D. HELMET: How to evaluate long-context models effectively and thoroughly. InThe Thirteenth International Conference on Learning Representations, 2025. URL https://openreview .net/forum?id=293V3bJbmE. Yoran, O., Wolfso...

work page doi:10.48550/arxiv.250 2025
[10]

It matches or is semantically equivalent to the reference answer

work page
[11]

It accurately answers the question using information from the documents

work page
[12]

Steven Weber

It does not contain extra hallucinated or incorrect information The answer should be consideredCORRECTeven if: • It uses slightly different wording but conveys the same meaning • It uses synonyms or alternative names for the same entity • It is a shorter or longer form of the reference answer (e.g., “Steven Weber” vs “Steven Robert Weber”) Respond withONL...

work page