{"id":"3dbd5ebb-bbdb-4451-9e96-9ba2b68f678e","arxiv_id":"2504.15903","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Adding tiny structured noise to ARC example grids sharply reduces GPT-4o's exact-match solve rate, but the paper's zero-noise success headline is built into its own task-selection rule.","lead":"This paper added small amounts of random noise to the example grids of ARC reasoning puzzles and measured how well large language models still solved them. It reports that noise sharply drops GPT-4o's accuracy, but the seven puzzles tested were chosen because GPT-4o could already solve them.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported collapse at 0.05% noise is not supported by the paper's own noise-injection formula: for every grid used, floor(n × total cells) = 0 altered cells at that level.","rationale":"The reader's verdict of REJECT is well supported. The floor-formula contradiction is visible in the text and directly controls the central quantitative claim; it is not a matter of interpretation. At 0.05%, every grid in the paper has fewer than one altered cell under floor, so the condition should equal the zero-noise baseline. If the authors instead used a different rounding or scaled the stated percentage differently, they need to say so and report actual modified-cell counts; the current text does neither. The new-prompt experiment in Section 4.5 also supports the concern that unlabeled noisy examples make the mapping ambiguous, and its accuracy recovery is evidence that the drop is not pure fragility; this reinforces rejection but is secondary to the arithmetic. I recommend REJECT, matching the reader's verdict: the paper should be rejected or returned for major revision with full per-task numeric results and a corrected noise-injection description. Independent strengths—task IDs, prompt text, and a linked repository—mean the core experiment is re-runnable, which is why a precise computational check is feasible.","tokens_in":85742,"tokens_out":5566,"duration_ms":53371,"concrete_test":"Run the linked repository's noise-injection routine on task 272f95fa at n=0.0005 and n=0.00125, and print the number of altered cells per grid. If the count is 0 for both (as the floor formula implies), the Section 4.1 'minimal noise' collapse is not explained by the stated noise manipulation. As a second check, inspect the saved prompt text for at least one 0.05%-noise trial and diff it against the clean prompt; zero differing cells would confirm the inconsistency.","verdict_should_be":"REJECT","load_bearing_attack":"Section 3.2 defines the number of altered elements as T = ⌊n × T⌋, where T is the total element count, and the paper invokes this to explain the 'minimal noise level (0.05%)' collapse in Section 4.1. The largest grid in the study is 19×18 = 342 cells; 0.0005 × 342 = 0.171, so floor gives 0. The same is true for every grid in the seven-task set (e.g., 6×6, 8×8, 5×5), and even the 0.125% level shown in Figures 2–3 yields floor(0.00125 × 342) = 0. If the implementation follows the stated formula, the 0.05% condition is identical to the zero-noise condition, so the reported drop to near-zero cannot be caused by injected noise. The paper does not provide per-task numeric tables or readable figure traces, and the figure blocks in the manuscript are corrupted text, so there is no independent way to verify the actual cell counts used. This is not a stylistic issue: the paper's headline fragility conclusion rests on an effect at a noise level that, by its own arithmetic, may not exist. The selection rule in Section 3.1 (only tasks GPT-4o solves more than half the time at zero noise) and the abstract's 'all ARC tasks' phrasing versus the seven tasks in Section 5 further weaken the generalizability, but the noise-level arithmetic is the most load-bearing internal inconsistency.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript reports an empirical study of GPT-4o, DeepSeek R1, and LLaMA 3.2 on a subset of seven Abstraction and Reasoning Corpus (ARC) tasks under injected grid noise and varying sampling temperature. The authors select tasks by requiring that GPT-4o solve them in more than half of 30 trials at zero noise, then inject noise into either the input or output grids of the few-shot examples, run 30 trials per condition, and report accuracy under correct- and partial-match criteria. The paper's central claims, stated in the abstract and Section 4.1, are that GPT-4o solves all ARC tasks at zero noise, that a minimal stated noise level of 0.05% causes accuracy to collapse to near zero, that higher temperature degrades performance, and that the introduction of noise consistently impairs performance across architectures.","tokens_in":85891,"tokens_out":8378,"duration_ms":74362,"significance":"If established, the reported collapse at a 0.05% noise level would be a striking finding about the fragility of LLM reasoning on grid-abstraction tasks, and the study's structure has some meritorious components: the separation of input-grid versus output-grid noise, the use of a modified-prompt control condition (Section 4.5), the temperature sweep, and the provision of a public code repository link. However, the significance is not realized in the manuscript as written: the headline effect is arithmetically inconsistent with the paper's own noise-injection formula, the zero-noise result is guaranteed by the task-selection rule rather than discovered, the abstract's 'all ARC tasks' claim exceeds the seven-task scope, the promised cross-model comparison is not presented (Sections 4-5 report only GPT-4o results), and the quantitative results are not verifiable because no numeric tables are given and the results figures render as unreadable font-encoded text. The paper identifies a worthwhile question but does not currently provide credible evidence bearing on it.","major_comments":[{"comment":"The central quantitative claim is internally inconsistent with the stated noise-injection protocol. Section 3.2 defines the number of altered elements as T = floor(n x T), where T is the total element count, and the noise levels are described throughout as percentages (0.05% to 0.3%). Under that formula, 0.05% (n = 0.0005) yields floor(0.0005 x 342) = 0 altered cells for the largest grid in the study (19 x 18) and zero for every other grid in the seven-task set (6 x 6, 8 x 8, 7 x 7, 5 x 5, 15 x 17, 14 x 12); even the 0.125% level depicted in Figures 2-3 and the top level of 0.3% alter at most one cell on the largest grid and zero cells on most others. The 0.05% condition is therefore identical to the zero-noise condition under the stated formula, so the sharp accuracy decline attributed to that level in Section 4.1 cannot be caused by the described injection. Conversely, the noisy prompts reproduced in the Appendix contain dozens of altered cells, which is irreconcilable with any of the stated percentages; the executed protocol and the documented protocol cannot both be correct as written. The paper provides no per-task numeric tables or readable figures with which to determine which protocol was actually used, so this is not a presentation issue but a load-bearing inconsistency in the paper's headline result.","section":"3.2, 4.1"},{"comment":"The zero-noise result reported in Section 5 is an artifact of the task-selection rule in Section 3.1. Tasks were admitted to the study only if GPT-4o solved them in more than 50% of 30 trials at noise level 0, and Section 5 then reports that GPT-4o solves exactly those seven tasks under noise-free conditions as though it were a finding. The abstract compounds this by stating that GPT-4o solves 'all ARC tasks under zero-noise conditions,' which contradicts the seven-task scope acknowledged in Section 5. In addition, the abstract and title claim that noise impairs performance 'regardless of architecture' and 'across all models,' but no experimental results for DeepSeek R1 or LLaMA 3.2 appear anywhere in Sections 4-5; the only information about these models is the zero-noise screening failure stated in the selection criteria. The claims as framed in the abstract are therefore unsupported by the presented experimental design and evidence.","section":"3.1, 5, Abstract"},{"comment":"The quantitative evidence for the headline results is not verifiable. The claims of a 'perfect score of 30/30,' a decline 'often reducing it to near-zero' at 0.05% noise, and 'significantly more incorrect predictions' at temperature 1.0 are asserted in prose without any numeric table of per-task, per-condition accuracy. The results figures (Figures 5, 7, 9, 11, 13, 15, 17) render in the manuscript as corrupted font-encoded text rather than readable plots, and even the caption of Figure 5 does not consistently describe the stated axes (the right y-axis is said to show both counts and percentages). Because the paper's central claims are quantitative collapses at specific noise levels, the absence of any readable data point, mean, or standard deviation is a substantive gap that prevents the results from being checked.","section":"4.1, 4.3, 5, Figures 5-17"},{"comment":"The interpretation that the accuracy drop reflects 'pattern memorization rather than generalizable rule learning' is not established, because the noisy prompts are underdetermined. The model is never told which cells are noise, so a demonstration pair with altered cells is consistent with many alternative transformation rules; a robust reasoner could not be expected to infer the original rule from it. The paper's own new-prompt experiment (Section 4.5) partially restores accuracy when the model is told that different input grids may map to the same output, which is direct evidence that the original noisy prompts were ambiguous rather than merely perturbed. The fragility conclusion should therefore be framed as an ambiguity effect unless a condition is reported in which an oracle or human reasoner can solve the same corrupted prompts.","section":"4.5, 4.1"}],"minor_comments":[{"comment":"The notation in the noise-injection formula is confusing: T denotes both the total element count and the number of altered elements in 'T = floor(n x T)'; distinct symbols should be used for the two quantities.","section":"3.2"},{"comment":"Section 3.3 describes experimental configurations as (1 x k)-, (3 x k)-, and (9 x k)-shot, while Section 3.1 states that the seven selected tasks consist of three 2-shot and four 3-shot examples; the relationship between the k-shot scheme and the task set is never explained.","section":"3.3, 3.1"},{"comment":"Sections 4.6 and 4.7 contain identical text, both claiming to present 'Figure 3,' and Figures 2 and 3 appear to be duplicates; the cross-referencing and figure numbering need to be corrected.","section":"4.6, 4.7, Figures 2-3"},{"comment":"Several bibliography entries lack sufficient detail to be verified (e.g., references [2] and [3] give no archive or DOI identifiers, and [4] cites only a workshop URL); the Mitchell entries are also inconsistently formatted.","section":"References"},{"comment":"The text contains many typographical and stylistic errors that should be corrected in any revision, including 'Franois Chollet' (missing cedilla), 'introdusing,' 'Orignal' in the Appendix headings, 'T ask ID' in figure headings, and the sentence fragment in Section 5 beginning 'We demonstrating how noise is introduced.'","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The manuscript reads as a preliminary report rather than a completed study. The most serious issue for the editor is that the paper's quantitative headline cannot be reconstructed from its own stated method: under the Section 3.2 formula, the reported 0.05% collapse condition is identical to the zero-noise condition for every grid used, while the Appendix examples show alteration counts inconsistent with any stated percentage. Combined with the selection-induced circularity of the zero-noise result, the overbroad 'all ARC tasks' and cross-architecture claims in the abstract, and the absence of any readable quantitative evidence, this goes beyond what a revision could repair; correcting it would require a new experimental design and re-run. I would encourage the authors to redo the study with a correct, explicitly documented noise-injection protocol, an unselected or differently selected task set, results for all three models, and per-task numeric tables, and to resubmit as a new manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this one is not ready. The central claim—that 0.05% noise collapses GPT-4o's accuracy—is contradicted by the paper's own formula. Section 3.2 says the number of altered cells is floor(n × T); the largest grid in the set is 19×18 = 342 cells, and 0.0005 × 342 = 0.171, which floors to 0. At the 0.125% level used in Figures 2–3, the count is still 0 for every grid shown. If the implementation follows the stated formula, the 0.05% condition is identical to zero noise, so the reported collapse to near-zero cannot be caused by injected noise. This is a load-bearing internal inconsistency, not a typo.\n\nThe other headline number is also built in. Section 3.1 selects tasks only when GPT-4o solves them in more than half of 30 trials at noise level 0; Section 5 then reports that GPT-4o solves these 7 tasks at zero noise as a finding, and the abstract overclaims 'all ARC tasks.' That is circular.\n\nWhat is genuinely useful: the authors name the seven task IDs, provide prompt text, and link a repository. The experimental idea is simple enough to re-run, and the new-prompt experiment in Section 4.5 is a reasonable check. But that check partially restores accuracy when the model is told noise may be present, which undercuts the fragility interpretation—it suggests the original noisy prompts were ambiguous, not merely perturbed versions of a rule the model could still follow.\n\nThe presentation blocks any independent verification: the figure content is corrupted text, there are no per-task numeric tables, and references [2] and [3] look unverifiable. The temperature finding is plausible but could easily be sampling variance, especially with only 30 trials per condition.\n\nBottom line: the paper's conclusions are not supported by its own procedures. It could be salvaged with corrected noise arithmetic, real noise levels, per-task results, and a claim limited to the seven selected tasks. As written, it is not worth referee time; I would desk-reject it, and I would not bring it to a reading group.","headline":"The paper's headline fragility result is contradicted by its own noise-injection arithmetic, and its zero-noise success is an artifact of task selection.","tokens_in":86618,"tokens_out":2370,"would_cite":false,"duration_ms":22457,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that perturbing as little as 0.05% of the cells in ARC demonstration grids drives LLM exact-match accuracy toward zero, even on tasks the same model solves perfectly when clean.","keywords":["Abstraction and Reasoning Corpus","LLM robustness","input perturbation","few-shot learning","model temperature","grid reasoning","GPT-4o","exact-match accuracy"],"falsifier":"Give the same noisy example grids to a reasoner that already knows the official ARC rule for the task but is not told which cells were altered; if that reasoner also cannot solve the 0.05% noise prompt, the corrupted demonstrations are ambiguous and the accuracy collapse is not evidence of LLM-specific fragility.","tokens_in":85345,"feed_emoji":"🧩","tokens_out":5541,"duration_ms":54285,"temperature":0.7,"pith_summary":"On the Abstraction and Reasoning Corpus (ARC), where a model infers a grid transformation from a few input-output pairs, this paper tries to establish that current LLMs are brittle: perturbing a tiny fraction of cells in the demonstration grids collapses exact-match accuracy to near zero even though the same tasks are solved cleanly. The supporting evidence is GPT-4o, which solves seven selected ARC tasks at zero noise and temperature 0, and then loses most or all correct outputs when 0.05% of cells in the examples are replaced with other colors from the same grid. The paper also claims that no other tested model solves any of these tasks at zero noise, and that raising temperature from 0 to 1 hurts even noise-free performance. A modified prompt that tells the model the examples contain noise partially restores accuracy, which the authors read as evidence that explicit awareness helps but does not cure the fragility. A sympathetic reader would take the paper as a caution: LLM success on clean ARC-style reasoning does not transfer to even lightly corrupted inputs.","feed_headline":"0.05% noise collapses LLM accuracy on ARC reasoning tasks","feed_subtitle":"GPT-4o solves clean ARC puzzles but drops to near zero when 0.05% of example cells are altered; temperature 1 hurts too.","key_machinery":"The operative object is the corrupted few-shot prompt: one or more ARC input-output pairs in which a fraction n of cells in either the input or output grid is randomly changed to another value from the same grid's palette while the paired grid stays fixed. This object carries the argument because it isolates whether the model can recover a rule when the demonstrations themselves are unreliable. The supporting machinery is exact-match accuracy computed over 30 trials per condition, model temperature as a second knob, and a modified prompt that explicitly warns the model that noise has been added to the examples.","core_discovery":"The central claim is that noise in the demonstrations, not in the test input, is enough to break LLM abstraction on ARC. The paper defines noise as replacing chosen cells of an example grid by other values already present in the grids, at stated fractions such as 0.05%, while leaving the paired grid untouched. Under exact-match evaluation over 30 trials, GPT-4o's 30/30 zero-noise performance dropped sharply, often to near zero, at the lowest noise level, for both input-noise and output-noise variants; adding more k-shot examples did not reliably help, and temperature 1 reduced accuracy even with clean prompts. The authors conclude that the drop is a shared vulnerability of current LLMs to input perturbation, not a peculiarity of one model.","pith_inferences":["A direct extension would be to mark the actual noise locations in the prompt rather than merely warning that noise exists; the paper's partial-recovery result predicts that locating the altered cells would restore much more of the accuracy if the failure is attention-based.","The 0.05% collapse may be amplified by exact-match scoring, since a model that finds the rule but misplaces one cell scores zero; the paper's partial-match statistics are the right check, and a future study could compare partial-match trends to separate rule loss from output transcription error.","The same protocol could be run on an oracle or human baseline that knows the true transformation rule; without that floor, the 'fragility' label is relative to an unspecified robust reasoner.","The paper's evidence under noise is concentrated on GPT-4o, so the claim that the vulnerability is architecture-independent is an extrapolation rather than a directly measured comparison across all tested models."],"forward_implications":["If a few changed cells in demonstrations cause near-total collapse, then success on clean ARC tasks cannot be taken as evidence of robust rule induction.","Deploying current LLMs on visual or spatial reasoning with noisy input data will require noise-awareness in the prompt or training, since adding more examples alone does not restore accuracy.","Deterministic decoding at temperature 0 is the appropriate setting for grid-reasoning tasks, while temperature 1 consistently sacrifices accuracy.","Explicitly telling the model that examples contain noise is a partial mitigation, since the recovered accuracy remains modest at higher noise levels."],"supporting_citations":[{"why":"Supplies the ARC benchmark and the few-shot grid-to-grid task format that the experiments perturb.","marker":"[1]"},{"why":"Prior evidence that pretrained language models fail when reasoning requires extrapolation beyond their training distribution.","marker":"[3]"},{"why":"Prior finding that adding noise to inputs causes performance drops in multimodal models, the pattern this paper extends to ARC.","marker":"[4]"},{"why":"ConceptARC benchmark evidence that models struggle with understanding and generalization in the ARC domain.","marker":"[8]"},{"why":"Human and GPT-4 comparison framing ARC as a domain where human cognition is more robust than current models.","marker":"[9]"}],"fun_headline_variants":["Tiny noise wrecks LLM abstraction on ARC puzzles","0.05% noise slashes GPT-4o ARC accuracy to near zero","LLMs collapse under 0.05% example noise on ARC","Even clean prompts fail at temperature 1 on ARC","ARC reasoning fragile: minimal noise flips LLM success"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusions assume that a few corrupted example grids still reveal the original transformation rule well enough that a capable reasoner could ignore the altered cells and solve the task, even though the model is never told which cells are noise.","fun_headline_variants_meta":{"raw":{"variants":["Tiny noise wrecks LLM abstraction on ARC puzzles","0.05% noise slashes GPT-4o ARC accuracy to near zero","LLMs collapse under 0.05% example noise on ARC","Even clean prompts fail at temperature 1 on ARC","ARC reasoning fragile: minimal noise flips LLM success"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000681,"raw_usage":{"total_tokens":3094,"prompt_tokens":947,"completion_tokens":2147,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":2060}},"tokens_in":563,"tokens_out":2147,"duration_ms":13625,"temperature":1.0,"reasoning_tokens":2060,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:16:24.491625+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give the same noisy example grids to a reasoner that already knows the official ARC rule for the task but is not told which cells were altered; if that reasoner also cannot solve the 0.05% noise prompt, the corrupted demonstrations are ambiguous and the accuracy collapse is not evidence of LLM-specific fragility.","supporting_citations":[{"cited_title":"Tervo and A","cited_arxiv_id":null,"evidence_quote":"Prior evidence that pretrained language models fail when reasoning requires extrapolation beyond their training distribution."},{"cited_title":"E., & Das, R","cited_arxiv_id":null,"evidence_quote":"Prior finding that adding noise to inputs causes performance drops in multimodal models, the pattern this paper extends to ARC."}],"review_version":1}