REVIEW 4 major objections 5 minor 1 cited by
Impact of Noise on LLM-Models Performance in Abstraction and Reasoning Corpus (ARC) Tasks with Model Temperature Considerations
T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read The paper claims that perturbing as little as 0.05% of the cells in ARC demonstration grids drives LLM exact-match accuracy toward zero, even on tasks the same model solves perfectly when clean.
desk verdict The paper's headline fragility result is contradicted by its own noise-injection arithmetic, and its zero-noise success is an artifact of task selection. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The operative object is the corrupted few-shot prompt: one or more ARC input-output pairs in which a fraction n of cells in either the input or output grid is randomly changed to another value from the same grid's palette while the paired grid stays fixed. This object carries the argument because it isolates whether the model can recover a rule when the demonstrations themselves are unreliable. The supporting machinery is exact-match accuracy computed over 30 trials per condition, model temperature as a second knob, and a modified prompt that explicitly warns the model that noise has been added to the examples.
What would settle it
Give the same noisy example grids to a reasoner that already knows the official ARC rule for the task but is not told which cells were altered; if that reasoner also cannot solve the 0.05% noise prompt, the corrupted demonstrations are ambiguous and the accuracy collapse is not evidence of LLM-specific fragility.
Extended reading notes
Core claim
The central claim is that noise in the demonstrations, not in the test input, is enough to break LLM abstraction on ARC. The paper defines noise as replacing chosen cells of an example grid by other values already present in the grids, at stated fractions such as 0.05%, while leaving the paired grid untouched. Under exact-match evaluation over 30 trials, GPT-4o's 30/30 zero-noise performance dropped sharply, often to near zero, at the lowest noise level, for both input-noise and output-noise variants; adding more k-shot examples did not reliably help, and temperature 1 reduced accuracy even with clean prompts. The authors conclude that the drop is a shared vulnerability of current LLMs to input perturbation, not a peculiarity of one model.
Load-bearing premise
The conclusions assume that a few corrupted example grids still reveal the original transformation rule well enough that a capable reasoner could ignore the altered cells and solve the task, even though the model is never told which cells are noise.
Editorial extensions
If this is right
- If a few changed cells in demonstrations cause near-total collapse, then success on clean ARC tasks cannot be taken as evidence of robust rule induction.
- Deploying current LLMs on visual or spatial reasoning with noisy input data will require noise-awareness in the prompt or training, since adding more examples alone does not restore accuracy.
- Deterministic decoding at temperature 0 is the appropriate setting for grid-reasoning tasks, while temperature 1 consistently sacrifices accuracy.
- Explicitly telling the model that examples contain noise is a partial mitigation, since the recovered accuracy remains modest at higher noise levels.
Reading between the lines
- A direct extension would be to mark the actual noise locations in the prompt rather than merely warning that noise exists; the paper's partial-recovery result predicts that locating the altered cells would restore much more of the accuracy if the failure is attention-based.
- The 0.05% collapse may be amplified by exact-match scoring, since a model that finds the rule but misplaces one cell scores zero; the paper's partial-match statistics are the right check, and a future study could compare partial-match trends to separate rule loss from output transcription error.
- The same protocol could be run on an oracle or human baseline that knows the true transformation rule; without that floor, the 'fragility' label is relative to an unspecified robust reasoner.
- The paper's evidence under noise is concentrated on GPT-4o, so the claim that the vulnerability is architecture-independent is an extrapolation rather than a directly measured comparison across all tested models.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript reports an empirical study of GPT-4o, DeepSeek R1, and LLaMA 3.2 on a subset of seven Abstraction and Reasoning Corpus (ARC) tasks under injected grid noise and varying sampling temperature. The authors select tasks by requiring that GPT-4o solve them in more than half of 30 trials at zero noise, then inject noise into either the input or output grids of the few-shot examples, run 30 trials per condition, and report accuracy under correct- and partial-match criteria. The paper's central claims, stated in the abstract and Section 4.1, are that GPT-4o solves all ARC tasks at zero noise, that a minimal stated noise level of 0.05% causes accuracy to collapse to near zero, that higher temperature degrades performance, and that the introduction of noise consistently impairs performance across architectures.
Significance. If established, the reported collapse at a 0.05% noise level would be a striking finding about the fragility of LLM reasoning on grid-abstraction tasks, and the study's structure has some meritorious components: the separation of input-grid versus output-grid noise, the use of a modified-prompt control condition (Section 4.5), the temperature sweep, and the provision of a public code repository link. However, the significance is not realized in the manuscript as written: the headline effect is arithmetically inconsistent with the paper's own noise-injection formula, the zero-noise result is guaranteed by the task-selection rule rather than discovered, the abstract's 'all ARC tasks' claim exceeds the seven-task scope, the promised cross-model comparison is not presented (Sections 4-5 report only GPT-4o results), and the quantitative results are not verifiable because no numeric tables are given and the results figures render as unreadable font-encoded text. The paper identifies a worthwhile question but does not currently provide credible evidence bearing on it.
major comments (4)
- [3.2, 4.1] The central quantitative claim is internally inconsistent with the stated noise-injection protocol. Section 3.2 defines the number of altered elements as T = floor(n x T), where T is the total element count, and the noise levels are described throughout as percentages (0.05% to 0.3%). Under that formula, 0.05% (n = 0.0005) yields floor(0.0005 x 342) = 0 altered cells for the largest grid in the study (19 x 18) and zero for every other grid in the seven-task set (6 x 6, 8 x 8, 7 x 7, 5 x 5, 15 x 17, 14 x 12); even the 0.125% level depicted in Figures 2-3 and the top level of 0.3% alter at most one cell on the largest grid and zero cells on most others. The 0.05% condition is therefore identical to the zero-noise condition under the stated formula, so the sharp accuracy decline attributed to that level in Section 4.1 cannot be caused by the described injection. Conversely, the noisy prompts reproduced in the Appendix contain dozens of altered cells, which is irreconcilable with any of the stated percentages; the executed protocol and the documented protocol cannot both be correct as written. The paper provides no per-task numeric tables or readable figures with which to determine which protocol was actually used, so this is not a presentation issue but a load-bearing inconsistency in the paper's headline result.
- [3.1, 5, Abstract] The zero-noise result reported in Section 5 is an artifact of the task-selection rule in Section 3.1. Tasks were admitted to the study only if GPT-4o solved them in more than 50% of 30 trials at noise level 0, and Section 5 then reports that GPT-4o solves exactly those seven tasks under noise-free conditions as though it were a finding. The abstract compounds this by stating that GPT-4o solves 'all ARC tasks under zero-noise conditions,' which contradicts the seven-task scope acknowledged in Section 5. In addition, the abstract and title claim that noise impairs performance 'regardless of architecture' and 'across all models,' but no experimental results for DeepSeek R1 or LLaMA 3.2 appear anywhere in Sections 4-5; the only information about these models is the zero-noise screening failure stated in the selection criteria. The claims as framed in the abstract are therefore unsupported by the presented experimental design and evidence.
- [4.1, 4.3, 5, Figures 5-17] The quantitative evidence for the headline results is not verifiable. The claims of a 'perfect score of 30/30,' a decline 'often reducing it to near-zero' at 0.05% noise, and 'significantly more incorrect predictions' at temperature 1.0 are asserted in prose without any numeric table of per-task, per-condition accuracy. The results figures (Figures 5, 7, 9, 11, 13, 15, 17) render in the manuscript as corrupted font-encoded text rather than readable plots, and even the caption of Figure 5 does not consistently describe the stated axes (the right y-axis is said to show both counts and percentages). Because the paper's central claims are quantitative collapses at specific noise levels, the absence of any readable data point, mean, or standard deviation is a substantive gap that prevents the results from being checked.
- [4.5, 4.1] The interpretation that the accuracy drop reflects 'pattern memorization rather than generalizable rule learning' is not established, because the noisy prompts are underdetermined. The model is never told which cells are noise, so a demonstration pair with altered cells is consistent with many alternative transformation rules; a robust reasoner could not be expected to infer the original rule from it. The paper's own new-prompt experiment (Section 4.5) partially restores accuracy when the model is told that different input grids may map to the same output, which is direct evidence that the original noisy prompts were ambiguous rather than merely perturbed. The fragility conclusion should therefore be framed as an ambiguity effect unless a condition is reported in which an oracle or human reasoner can solve the same corrupted prompts.
minor comments (5)
- [3.2] The notation in the noise-injection formula is confusing: T denotes both the total element count and the number of altered elements in 'T = floor(n x T)'; distinct symbols should be used for the two quantities.
- [3.3, 3.1] Section 3.3 describes experimental configurations as (1 x k)-, (3 x k)-, and (9 x k)-shot, while Section 3.1 states that the seven selected tasks consist of three 2-shot and four 3-shot examples; the relationship between the k-shot scheme and the task set is never explained.
- [4.6, 4.7, Figures 2-3] Sections 4.6 and 4.7 contain identical text, both claiming to present 'Figure 3,' and Figures 2 and 3 appear to be duplicates; the cross-referencing and figure numbering need to be corrected.
- [References] Several bibliography entries lack sufficient detail to be verified (e.g., references [2] and [3] give no archive or DOI identifiers, and [4] cites only a workshop URL); the Mitchell entries are also inconsistently formatted.
- [Throughout] The text contains many typographical and stylistic errors that should be corrected in any revision, including 'Franois Chollet' (missing cedilla), 'introdusing,' 'Orignal' in the Appendix headings, 'T ask ID' in figure headings, and the sentence fragment in Section 5 beginning 'We demonstrating how noise is introduced.'
Circularity Check
Zero-noise solve claim is the Section 3.1 selection filter restated; the noise-sensitivity conclusion itself is not derived by construction.
-
self definitional
[Section 3.1, item 2; Section 5; Abstract]
"Each task was evaluated over 30 trials using the GPT-4o model. Only those tasks which the model solved correctly in more than 50% of the trials were considered. ... GPT-4o initially exhibited strong problem-solving capabilities on the ARC challenge, accurately solving the following 7 tasks under noise-free conditions and a deterministic setup (temperature = 0)."
The selected set is defined by the condition that GPT-4o solves each task in more than 50% of 30 zero-noise trials (Section 3.1). Section 5 then reports as a result that GPT-4o 'accurately solving the following 7 tasks under noise-free conditions.' That report is the inclusion rule itself: every task in the set satisfies the criterion by construction, so the zero-noise solve list conveys no new information. The abstract's 'solving all ARC tasks under zero-noise conditions' generalizes the filter to the whole 400-task corpus, which was never evaluated; it is a paraphrase of the selection criterion presented as a capability result.
full rationale
The main noise study is an empirical measurement: take tasks GPT-4o can solve clean, inject noise as specified, and measure accuracy. That chain does not derive its conclusion from its inputs by definition; a robust model could in principle maintain accuracy under perturbation, so the observed degradation is an independent finding. The clear circular step is the zero-noise claim: the subset of seven tasks was constructed using GPT-4o's zero-noise solve rate, and the paper then reports that rate as a result, with the abstract expanding it to 'all ARC tasks.' This is selection-criterion-as-finding. The paper's own noise formula T = floor(n x T) is also internally inconsistent with the reported 0.05% and 0.125% collapse: for every grid size used (largest 19x18 = 342 cells), floor(0.0005 x 342) = 0, so the stated low-noise conditions would be identical to the clean baseline under the formula as written. That is a validity problem under the paper's own equations, but it is not an example of a result reducing to its inputs by construction, so it does not contribute to the circularity score as a separate step. Reference [4] is a self-citation used to motivate noise sensitivity, but the paper's own experiments supply the evidence, so it is not load-bearing.
Assumptions & free parameters
free parameters (4)
- task inclusion threshold =
solve rate > 50% over 30 trials
- noise levels =
reported inconsistently as 0.05%, 0.125%, and 0 to 0.3%
- trial count =
30
- temperature settings =
0 and 1
assumptions (5)
- domain assumption ARC tasks are solvable by inferring a transformation rule from two or three examples without task-specific priors.
- ad hoc to paper A grid with randomly altered cells still encodes the original rule well enough that a robust solver could produce the correct output.
- domain assumption GPT-4o output is deterministic at temperature 0.
- domain assumption The 7 selected tasks are representative of the ARC benchmark's 400 tasks.
- standard math Exact cell-by-cell match is the correct success metric for noisy-task evaluation.
Cite this review
Pith. "Pith review of Impact of Noise on LLM-Models Performance in Abstraction and Reasoning Corpus (ARC) Tasks with Model Temperature Considerations." pith.science (2026). https://pith.science/paper/ATXDJUBQ
@misc{pith2026250415903,
author = {Pith},
title = {Pith review of: Impact of Noise on LLM-Models Performance in Abstraction and Reasoning Corpus (ARC) Tasks with Model Temperature Considerations},
year = {2026},
howpublished = {\url{https://pith.science/paper/ATXDJUBQ}},
note = {Machine review of arXiv:2504.15903}
}
read the original abstract
Recent advancements in Large Language Models (LLMs) have generated growing interest in their structured reasoning capabilities, particularly in tasks involving abstraction and pattern recognition. The Abstraction and Reasoning Corpus (ARC) benchmark plays a crucial role in evaluating these capabilities by testing how well AI models generalize to novel problems. While GPT-4o demonstrates strong performance by solving all ARC tasks under zero-noise conditions, other models like DeepSeek R1 and LLaMA 3.2 fail to solve any, suggesting limitations in their ability to reason beyond simple pattern matching. To explore this gap, we systematically evaluate these models across different noise levels and temperature settings. Our results reveal that the introduction of noise consistently impairs model performance, regardless of architecture. This decline highlights a shared vulnerability: current LLMs, despite showing signs of abstract reasoning, remain highly sensitive to input perturbations. Such fragility raises concerns about their real-world applicability, where noise and uncertainty are common. By comparing how different model architectures respond to these challenges, we offer insights into the structural weaknesses of modern LLMs in reasoning tasks. This work underscores the need for developing more robust and adaptable AI systems capable of handling the ambiguity and variability inherent in real-world scenarios. Our findings aim to guide future research toward enhancing model generalization, robustness, and alignment with human-like cognitive flexibility.
Figures
Figures from the paper (14 more)
Forward citations
Cited by 1 Pith paper
-
LLM Performance for Code Generation on Noisy Tasks
LLMs solve heavily obfuscated benchmark tasks, and performance decay under obfuscation differs sharply between old and new datasets, which the authors interpret as a signature of training-data contamination.
Reference graph
Works this paper leans on
-
[2]
M. Johnson, T. Hofmann, and J. B. Tenenbaum, ”Symbolic AI for ARC: A Rule-Based Approach,” in Proceedings of the NeurIPS 2020 Workshop on Abstraction and Reasoning , 2020. Retrieved from https://neurips.cc/ Conferences/2020/Schedule?showEvent=17938
work page 2020
-
[3]
R. Tervo and A. Patel, ”Pretrained Language Models for Abstract Reasoning: Suc- cesses and Failures on the ARC Challenge,” in Proceedings of the ACL 2022 ,
work page 2022
-
[4]
Das, R. E., & Das, R. (2024). Iterative Theory of Mind Assay of Multi- modal AI Models . In ICML 2024 Workshop on LLMs and Cognition. https: //openreview.net/forum?id=PsGVVQJZGk
work page 2024
-
[5]
Messeri, L., & Crockett, M. J. (2024). Artificial intelligence and illusions of understanding in scientific research. Nature, 627, 2024. https://doi.org/10. 1038/s41586-024-07146-0
work page 2024
-
[6]
M. Mitchell, ”Artificial intelligence still can’t form concepts,” Communica- tions of the ACM , August 2023. Retrieved from https://cacm.acm.org/news/ artificial-intelligence-still-cant-form-concepts/
work page 2023
-
[7]
Mitchell, ”AI models of conceptual abstraction and analogy- making,” May 12, 2023
M. Mitchell, ”AI models of conceptual abstraction and analogy- making,” May 12, 2023. Retrieved from https://melaniemitchell.me/ PostdocProjectDescription.pdf
work page 2023
-
[8]
A. Moskvichev, V. V. Odouard, and M. Mitchell, ”The ConceptARC benchmark: Evaluating understanding and generalization in the ARC domain,” arXiv preprint arXiv:2305.07141, 2023. Retrieved from https://arxiv.org/abs/2305.07141
arXiv 2023
-
[2019]
Retrieved from https://arxiv.org/abs/1911.01547
arXiv 1911
Show all 10 references
-
[2022]
Retrieved from https://aclanthology.org/2022.acl-long.120.pdf
2022
-
[2023]
Retrieved from https://arxiv.org/abs/2311.09247 29 8 Appendix Pictorial Representation Prompt With Orignal Prompt Prompt 1 Find t h e common r u l e t h a t maps an i n p u t g r i d t o an o u t p u t g r i d , g i v e n t h e e x a m p l e s below . Example1 I n p u t : 0 0 ...
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.