REVIEW 4 major objections 6 minor 10 references
Small Edits, Big Consequences: Telling Good from Bad Robustness in Large Language Models
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Frontier LLMs stay correct after 90% of the prompt is deleted, yet ignore a single quantifier flip that reverses the task; a descriptive function name is the strongest anchor.
desk verdict Good empirical idea and a clear case study, but the headline quantifier-flip result is unverifiable as written because Appendices B and C ship the same generator prompt. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is a three-by-two perturbation protocol with an execution-based oracle. Progressive deletion removes 10% of the remaining tokens per step (benign noise); lexical flip swaps a pivotal quantifier such as $\max\to\min$ (meaning-critical); jargon inflation replaces a common noun with an obscure term (ambiguous grey zone). Each family runs with the original descriptive function name and with that name masked to the neutral token `solved`, which isolates what the paper calls identifier anchoring—the reliance on a single high-salience cue. The oracle is the original LeetCode test suite: passing it after a mutation is taken as regression to the canonical solution, while failing it counts as a reaction to the edit. The authors also use a composite semantic-shift score $\Delta$ and embedding-based chain-of-thought distances to separate textual drift from behavioural change.
What would settle it
Regrade every generated program against freshly written test suites for each mutated prompt—for a $\max\to\min$ flip, check that the code returns a minimum rather than a maximum. If many programs that 'fail' the original suite in fact solve the mutated task, the reported insensitivity is inflated; if they fail both suites, the double asymmetry stands as stated.
Extended reading notes
Core claim
The paper's core claim is that today's LLMs are over-robust to underspecification and under-sensitive to meaning-changing edits, and that a single descriptive function name is the strongest cue determining which behaviour they show. The evidence: after ten steps of deleting tokens, unmasked models keep a median pass-rate above 85% on the original test suite, but swapping one quantifier ($\max\to\min$) pushes the median unmasked pass-rate to 46%, meaning only 54% of generations visibly adapt; jargon inflation sits in between at 56% pass-through. Masking the function name (replacing it with the neutral token `solved`) lowers pass-rates by 10–21 percentage points and is a larger manipulation than deleting large amounts of text. Reasoning-optimised variants track their baselines almost point-for-point; on jargon edits they are actually more likely (+4.0 percentage points) to return to the canonical answer. The authors interpret this as regression to memorised templates: models follow high-probability, well-rehearsed solutions unless a constraint is concise and explicitly foregrounded.
Load-bearing premise
The load-bearing premise is that passing the original LeetCode test suite after a prompt edit proves the model ignored the edit and reused the canonical solution; a genuinely adapted program could also pass those same tests.
Editorial extensions
If this is right
- Benchmarks that keep descriptive function names intact will overstate instruction-following, because a model can solve a heavily stripped prompt by anchoring on the identifier rather than reading the text.
- Minimal, semantics-altering edits—quantifier flips, polarity swaps, unit changes—deserve a permanent place in robustness testing, since they expose regressions that deletion and jargon miss.
- Reasoning-optimised endpoints should not be assumed to improve sensitivity to subtle prompt drift; the paper's measurements show them reusing the canonical answer at least as often as their baselines.
- Grading on the original test suite conflates correct adaptation to the mutated task with regression to the original one, so perturbation-aware oracles are a direct consequence of the paper's design.
- Masking salient identifiers during training and evaluation would reduce false robustness and is the paper's advocated next step.
Reading between the lines
- If identifier anchoring is as strong as reported, a cheap adversarial attack on coding LLMs would be to keep a descriptive function name while changing the semantics elsewhere in the prompt; the model would likely stay on the memorised solution.
- The same double asymmetry should be testable outside code generation—for example, mutating a single quantifier in a math word problem or a medical instruction and checking whether the final behaviour changes.
- The chain-of-thought drift result implies that interpretability tools relying on rationale text should be validated against output behaviour, since explanations can diverge sharply while the code stays identical.
- A direct causal check of the anchor story would be to fine-tune a small model on the same problems with randomised function names and compare its sensitivity to lexical flips; the paper's account predicts higher sensitivity than standard training.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper studies selective robustness in LLM code generation. The authors take 50 LeetCode problems, apply three perturbation families (progressive token deletion, jargon inflation, and lexical flip), and evaluate six frontier LLMs (including three reasoning-tuned variants) by running generated Python code against the original LeetCode test suites. They report a double asymmetry: models stay correct in 85% of cases after 90% of the prompt is deleted, yet only 54% react to a single quantifier flip that should reverse the task; masking the function name reduces pass-rates by 15–21 percentage points. They also report that reasoning-tuned variants are no more sensitive than their baselines, and that inter-model agreement drops sharply when anchored and anchor-free prompts are pooled. The paper frames these results as evidence that contemporary LLMs privilege memorized templates over prompt content and advocates evaluation protocols that reward differential sensitivity.
Significance. If the central findings hold, the paper makes a valuable empirical contribution to robustness evaluation of code-generation LLMs. The three-way perturbation design (benign noise, ambiguous jargon, and meaning-critical lexical flips) is a useful operationalization of 'good vs. bad' sensitivity, and the identifier-anchoring result—masking a single function name has a larger effect than deleting 90% of the prompt—is striking and practically relevant. The study covers a non-trivial scale (11,853 generations, six models, 50 problems) and includes a concrete case study (LeetCode #434) that vividly illustrates the proposed failure mode. The authors also explicitly acknowledge several limitations, including the indirectness of the evaluation oracle, which is a welcome sign of scientific honesty. However, as detailed below, the headline quantifier-flip claim is not currently supported by the documented generation protocol, and the oracle limitation directly weakens the central double-asymmetry statistics.
major comments (4)
- [Section 3.1 and Appendix C] The central claim that 'only 54% react to a single quantifier flip' is not substantiated by the documented Lexical Flip generation procedure. Section 3.1 defines LF as swapping a quantifier or polarity word (e.g., max↔min), but Appendix C, the LF meta-prompt, is nearly identical to Appendix B's Jargon Inflation prompt: it instructs the generator to 'Substitute EXACTLY ONE WORD' in a 'subtle and non-obvious way' and does not mention quantifiers, polarity, or any constraint that would force a semantics-reversing edit. Indeed, Appendix C lacks even the 'HIGHLY ADVANCED TECHNICAL and OBSCURE concepts' clause that appears in Appendix B. As written, the LF track could contain arbitrary one-word substitutions, so the reported 54% cannot be attributed specifically to a quantifier flip. The authors must either supply the actual LF-generation prompt used, or provide a distribution of the generated edits demonstrating that they were predominantly quantifier/polarity swaps.
- [Section 5 and Abstract] The evaluation oracle is the original LeetCode test suite, and the paper itself concedes (Section 5) that 'A passing score therefore demonstrates compatibility with the original task, not correctness for the mutated one.' This is load-bearing because the paper operationalizes 'react' as failing the original suite. A model that correctly adapts to the mutated prompt but whose solution also passes the original tests is counted as non-reaction, which inflates every insensitivity statistic in the abstract (85% under deletion, 54% under lexical flip, 56% under jargon). The double-asymmetry claim therefore conflates 'regression to the canonical solution' with 'passing the original suite.' To support the headline numbers, the authors need to validate the oracle on a sample—for instance, by human annotation of whether a passing generation actually reflects the mutated constraint, or by constructing fresh test suites for the mutated tasks. The single case study in §4.2 is illustrative but does not validate the aggregate statistics.
- [Section 4.1 and Table 2] The claim that 'reasoning-tuned variants are even less sensitive than their bases' is overgeneralized relative to the experimental coverage. Table 2 shows that reasoning variants were run only on the Jargon Inflation and Lexical Flip tracks, not on Progressive Deletion, and Section 5 states they were run 'only in the masked tracks.' Moreover, the LF difference (+1.6 pp) is explicitly reported as non-significant; only the JI difference (+4.0 pp) reaches p<0.05. The abstract's wording suggests a broad finding across perturbation families, whereas the evidence is limited to two families in masked mode, with one of the two differences being statistically indistinguishable from zero. The authors should restrict the claim to the JI–masked condition or extend the reasoning-variant evaluation to all tracks.
- [Data Availability and Reproducibility] The manuscript releases a dataset link ('https://huggingface.com/level8/leetp') but does not include the per-step mutated prompts or the generated outputs, and the URL appears to contain a domain typo (huggingface.com rather than huggingface.co). Because the LF-track generation prompt is the crux of the headline result, the absence of the actual mutated prompts makes it impossible for a reader to verify whether the 'lexical flips' were indeed single quantifier substitutions. The authors should release the exact prompts used for each track and the per-step mutated problem descriptions, or at least a representative sample, to enable independent verification of the central claim.
minor comments (6)
- [Section 4.4] The definition of 'strong agreement' is confusing for the Combined case: each non-reasoning model contributes two evaluations (masked and unmasked) and each reasoning model one, yet the text describes 'all models' giving the same verdict. Please clarify how the per-model duplicates are aggregated in the denominator and why a single count implies both anchor dependence and inter-model divergence.
- [Section 4.6] The composite Semantic Distance Index weights in Equation (2) are presented without derivation; the weights do not sum to 1 and appear hand-picked. This is acceptable for an exploratory factor analysis, but the authors should state that the index is a modeling choice and should provide a sensitivity check (e.g., uniform weights) so readers can assess the robustness of the reported effect sizes.
- [Figure 10 caption] Typo: 'accross' should be 'across'.
- [Data Availability] The dataset URL uses 'huggingface.com'; the correct domain is 'huggingface.co'.
- [Table 2] Column header 'o4-mini high' should be 'o4-mini-high' for consistency with the model name used in the text.
- [Section 2 and Table 2] Equation (1) gives 3,000 distinct prompts per model, but the total observation count of 11,853 implies an average of about 1,975 per model after early stopping; please state explicitly that early stopping reduces the per-model count and specify how many steps were completed on average per track.
Circularity Check
No circularity: the paper is an empirical stress-test study with no fitted parameter renamed as a prediction and no self-citation chain.
full rationale
The paper's claims are empirical measurements, not derivations from assumed inputs. The central statistics—85% pass under 90% deletion, 54% reaction to lexical flip, 56% pass under jargon inflation—are computed from 11,853 model generations against original LeetCode test suites, not fitted from any model or equation. The composite Semantic Distance Index in Section 4.6 uses hand-chosen weights (0.40 CLS–EMD, 0.60 BERTScore term, 0.40 NLI, 0.70 span delta), but this index is a secondary factor-analysis feature, not a parameter fitted to the outcome it later 'predicts'; the pass-rate results do not depend on it. There are no self-citations: the references are external datasets and prior works, and the paper invokes no uniqueness theorem or author-derived ansatz. The stated limitation in Section 5—'A passing score therefore demonstrates compatibility with the original task, not correctness for the mutated one'—is a construct-validity caveat about the evaluation oracle, not a circular reduction: the paper explicitly acknowledges that a valid adaptation to a mutated prompt could still pass the original tests. A separate methodological concern, not circularity, is that Appendix C's Lexical Flip generation prompt is textually identical to Appendix B's Jargon Inflation prompt, both instructing substitution of 'EXACTLY ONE WORD' with 'HIGHLY advanced TECHNICAL and OBSCURE concepts' and neither mentioning quantifiers or polarity; this threatens the distinctness of the LF track but does not make any result equivalent to its inputs by definition. Overall, the derivation chain is self-contained against external benchmarks, and no load-bearing step reduces to its own input.
Assumptions & free parameters
free parameters (3)
- Composite Semantic Distance Index weights =
0.40, 0.60, 0.40, 0.70
- Semantic drift threshold =
0.30
- Perturbation budget =
10 steps at 10% deletion per step
assumptions (4)
- domain assumption LeetCode test suites are a valid oracle for whether a model regressed to the baseline solution
- domain assumption The Jargon Inflation and Lexical Flip meta-prompts generate the intended distinct perturbation families
- domain assumption SBERT embeddings and MNLI-based NLI scores measure semantic distance in a way relevant to prompt drift
- domain assumption Masking the function name isolates identifier anchoring from other cues
Cite this review
Pith. "Pith review of Small Edits, Big Consequences: Telling Good from Bad Robustness in Large Language Models." pith.science (2026). https://pith.science/paper/UASZMZX7
@misc{pith2026250715868,
author = {Pith},
title = {Pith review of: Small Edits, Big Consequences: Telling Good from Bad Robustness in Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/UASZMZX7}},
note = {Machine review of arXiv:2507.15868}
}
read the original abstract
Large language models (LLMs) now write code in settings where misreading a single word can break safety or cost money, yet we still expect them to overlook stray typos. To probe where useful robustness ends and harmful insensitivity begins, we compile 50 LeetCode problems and craft three minimal prompt perturbations that should vary in importance: (i) progressive underspecification deleting 10 % of words per step; (ii) lexical flip swapping a pivotal quantifier ("max" to "min"); and (iii) jargon inflation replacing a common noun with an obscure technical synonym. Six frontier models, including three "reasoning-tuned" versions, solve each mutated prompt, and their Python outputs are checked against the original test suites to reveal whether they reused the baseline solution or adapted. Among 11 853 generations we observe a sharp double asymmetry. Models remain correct in 85 % of cases even after 90 % of the prompt is missing, showing over-robustness to underspecification, yet only 54 % react to a single quantifier flip that reverses the task, with reasoning-tuned variants even less sensitive than their bases. Jargon edits lie in between, passing through 56 %. Current LLMs thus blur the line between harmless noise and meaning - changing edits, often treating both as ignorable. Masking salient anchors such as function names can force re - evaluation. We advocate evaluation and training protocols that reward differential sensitivity: stay steady under benign noise but adapt - or refuse - when semantics truly change.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[1]
LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs
Yunhui Xia, Wei Shen, Yan Wang, Jason Klein Liu, Huifeng Sun, Siyue Wu, Jian Hu, and Xiaolong Xu. LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs. arXiv preprint arXiv:2504.14655, 2025. https: //arxiv.org/abs/2504.14655
arXiv 2025
- [2]
-
[3]
Adversarial ex- amples for evaluating reading comprehen- sion systems
Robin Jia and Percy Liang. Adversarial ex- amples for evaluating reading comprehen- sion systems. In Proceedings of EMNLP , pages 2021–2031, 2017
work page 2021
-
[4]
On faith- fulness and factuality in abstractive sum- marization
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faith- fulness and factuality in abstractive sum- marization. In Proceedings of ACL , pages 1906–1919, 2020
work page 1906
-
[5]
Xinyi Ye, Mingfeng Ou, Tianyi Li, Xuetao Ma, Yifan Yanggong, Sai Wu, Jie Fu, Gang Chen, and Junbo Zhao. Assessing hidden risks of large language models: An empirical study on robustness, consistency and credi- bility. arXiv:2305.10235, 2023
arXiv 2023
-
[6]
Chain-of-thought prompting elicits reasoning in large lan- guage models
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models. arXiv:2201.11903, 2022
arXiv 2022
-
[7]
Univer- sal adversarial triggers for attacking and an- alyzing NLP
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Univer- sal adversarial triggers for attacking and an- alyzing NLP. In Proceedings of EMNLP- IJCNLP, pages 2155–2167, 2019
work page 2019
-
[8]
Don’t take the easy way out: Ensemble based methods for avoid- ing known dataset biases
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. Don’t take the easy way out: Ensemble based methods for avoid- ing known dataset biases. In Proceedings of EMNLP , pages 4069–4083, 2019
work page 2019
Show all 10 references
-
[9]
Beyond the imitation game: Quantifying and examining the ca- pabilities of language models (BIG-Bench)
Ananya Srivastava, Justin Gilmer, Anselm Levskaya, et al. Beyond the imitation game: Quantifying and examining the ca- pabilities of language models (BIG-Bench). arXiv:2206.04615, 2022
2022 arXiv
-
[10]
Jargon Inflation
Percy Liang, Angela Jiang, et al. Holis- tic evaluation of language models (HELM). arXiv:2211.09110, 2022. Appendix Appendix A – General LLM prompt template You are an expert Python programmer. You will be given a question and will generate a correct Python program. ### Questi...
2022 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.