{"id":"40b9025f-365f-4dda-ae50-c9244cb3962d","arxiv_id":"2603.06642","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The reported SR-TTT retrieval gains are evaluation artifacts; under corrected evaluation exact match is 0%, with separate storage and addressing bottlenecks.","lead":"This paper retracts an earlier claim that a surprise-gated memory helps test-time-training language models remember exact strings. With the evaluation bug fixed, the model scores 0% exact recall in 2,250 trials, and the failure separates into a storage problem and an addressing problem.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Oracle-storage experiment may leak needle content, so the addressing-bottleneck decomposition is not yet supported.","rationale":"The reader correctly identified the oracle-storage experiment as the weakest link in the central two-bottleneck claim. I agree because the abstract's decomposition hinges on a clean isolation of addressing from storage, and the supplied v1 text demonstrates that the pipeline has previously had subtle causal and metric leaks. The concrete poisoned-value control directly tests whether the oracle leaks answer content through values, which is the most plausible confound. Since the reader's verdict is already UNVERDICTED due to missing corrected details, this concern does not shift the verdict; it reinforces the need for the corrected code and a controlled oracle experiment before the mechanistic post-mortem can be accepted.","tokens_in":4787,"tokens_out":4132,"duration_ms":42144,"concrete_test":"Run the corrected oracle-storage experiment with a poisoned-value control: store the needle keys keyed by ground-truth position, but replace every stored value vector with a random vector drawn from the same distribution (so values contain no needle content). If token accuracy rises above the reported 0% or extracted nats exceed ~0.06, the original oracle leaked answer content through values; if results are statistically unchanged, the addressing bottleneck is confirmed. Also verify with a causal-mask self-test that cache attention never attends to tokens at or after the query position.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SR-TTT's failure decomposes into two independent bottlenecks rests on the oracle-storage experiment, whose details are absent from the supplied text. For \"storage solved by an oracle\" to isolate addressing, the oracle must give the read-time attention access to perfectly stored needles without leaking the answer through the output path. The v1 body shows two prior leakage modes: the metric read logits one position early, and the cache attended non-causally over future tokens including the answer. Neither is described as controlled in the corrected oracle experiment. If the oracle stores ground-truth value vectors and the read-time fusion (α·CacheAttention) is added at the current token's hidden state, the model could copy the value directly, inflating the measured \"addressing mass\" toward 2.5x while token accuracy remains unchanged only because the supervision signal is misaligned. Conversely, if any non-causal path remains, the 0% EM could be artificially depressed rather than reflecting a genuine content-addressing failure. The abstract's numbers (0.06→0.15 mass, ~0.06 of 2.30 nats) cannot be audited without the corrected code; the v1 body's contrary \"massive improvements\" make this gap the load-bearing point.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript is a correction paper for an earlier preprint. The corrected abstract claims that the previously reported SR-TTT gains on Needle-in-a-Haystack were evaluation artifacts: the loss and metric read logits at the answer positions rather than one position earlier, and the Residual Cache attended non-causally over future tokens including the answer. Under corrected evaluation, the abstract reports that exact match remains 0% in all 2,250 paired trials and that the failure decomposes into a storage bottleneck (surprisal gating stores early needles at 0–1% containment) and an addressing bottleneck (content addressing extracts only about 0.06 of 2.30 nats). The paper also releases a corrected implementation with startup causality self-tests and retracts the v1 claims. The body supplied in the manuscript, however, is the original v1 text, which reports large retrieval improvements and does not contain the corrected experimental protocols or results described in the abstract.","tokens_in":5076,"tokens_out":5131,"duration_ms":54922,"significance":"If the corrected negative result is substantiated, the paper would provide a valuable cautionary data point for surprise-gated memory architectures: routing high-surprisal tokens to an exact-attention cache does not, by itself, fix exact recall in TTT models. The artifact diagnosis—reading answer-position logits and non-causal cache attention—is plausible and important, and the proposed two-bottleneck decomposition, if cleanly measured, would be a useful mechanistic contribution. The paper also has strengths in releasing code, providing diagnostic self-tests, and testing against an external benchmark rather than against fitted parameters. However, the supplied manuscript body does not currently contain the corrected experiments, trial counts, containment measurements, or oracle-storage protocol; all load-bearing numbers appear only in the abstract. The significance of the paper therefore cannot be assessed from the submitted text as it stands.","major_comments":[{"comment":"The central claim — \"Exact match remains 0% in all 2,250 paired trials across all corrected conditions\" — appears only in the abstract. The supplied body is the v1 manuscript: §5 reports \"massive retrieval improvements\" of +23% and +20% exact match, and Figure 4 is said to show gates opening to about 10%. There is no corrected methods section, no definition of containment, no trial protocol, no table of exact-match counts, and no description of how the two bottlenecks were measured. The abstract's numbers (0.06 → 0.15 addressing mass, ~0.06 of 2.30 nats, 0–1% containment at depth 0.1) cannot be audited. The manuscript must either include the full corrected experimental section or explicitly mark the v1 body as retracted and append the corrected results; otherwise the principal claim is unsupported by the submitted text.","section":"Abstract vs. §5"},{"comment":"The two-bottleneck decomposition depends on the oracle-storage experiment, but no implementation is reported. To isolate addressing, the oracle must give the read-time attention access to perfectly stored needles without leaking the answer through the fused output path in Eq. (1). The v1 body documents two leakage modes: the metric reads logits at answer positions, and the cache attends non-causally over future tokens including the answer itself. The corrected abstract does not state how these are controlled in the oracle experiment. The manuscript should report the causal-masking scheme, what exactly the oracle stores (e.g., keys/values versus raw token embeddings), and how α·CacheAttention is computed. A useful control would be to include a condition with oracle storage but scrambled or empty queries to estimate any direct-copy contribution. Without these details, the \"addressing mass\"","section":"Abstract / Eq. (1), §3.2, §5"},{"comment":"The body's conclusions directly contradict the corrected abstract. Section 5 claims that SR-TTT substantially mitigates exact-recall failures, and §5.1 attributes the 4096-token collapse entirely to RoPE extrapolation. Both statements are incompatible with the corrected abstract's conclusion that the earlier gains were artifacts of the evaluation and that the method does not learn retrieval. The manuscript should state explicitly which figures, tables, and numerical claims come from v1, which have been superseded, and which are newly corrected. As written, readers cannot distinguish the retracted evidence from the evidence meant to replace it.","section":"§5 and §5.1"}],"minor_comments":[{"comment":"The routing thresholds are not reported numerically. τ_EMA, the 0.8·τ_EMA chunk threshold, α_max, and the Stage 1/Stage 2 step split (7,000/3,000 steps) are free parameters; give their values and sensitivity, otherwise the surprising-token criterion is not reproducible.","section":"§3.1"},{"comment":"Figures 1–4 are referenced but not present in the supplied text. At minimum, the corrected paper must include the actual figures or link to an accessible appendix; the current text's reliance on missing figures prevents verification of the v1 claims and any corrected versions.","section":"Figures"},{"comment":"The notation is unclear: α is called a \"learned gate vector\" and also a clamped scalar parameter. Specify its dimension, whether it is per-head or per-layer, and whether it depends on the current query hidden state.","section":"Eq. (1)"},{"comment":"The v1 exact-match percentages are reported without trial counts, confidence intervals, or paired-trial details. The corrected abstract mentions 2,250 paired trials, but the body never ties those trials to the percentages.","section":"§5"},{"comment":"Two references are listed as \"Anonymous Authors\" ([5] and [14]). If the companion papers have been revealed, update the citations; otherwise note that they are anonymized preprints.","section":"References"},{"comment":"The typeset title contains a spacing artifact: \"AwareResidualTest-Time\"; please fix. A few other spacing/grammar issues in the opening pages should also be cleaned up.","section":"Title/Abstract"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be an unusual correction preprint: the abstract contains the new negative result, but the body is the original v1 paper with its original, contradictory claims. The corrected experiments, including the 2,250 trials, containment rates, nats extracted, and the oracle-storage control, are not in the submitted text. If these materials are available in the linked repository, they need to be brought into the paper with full protocol details. I see no reason to doubt the good faith of the retraction, but the paper as it stands is not evaluable as a scientific contribution because the evidence for its central claim is absent. A major revision that replaces/annotates the v1 body and reports the corrected experiments in full would make the paper potentially valuable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This submission is in a strange state: the abstract says the v1 claims were artifact-driven and retracts them, but the attached full text is the uncorrected v1 paper, complete with the “massive improvements” the abstract now says are fake. You cannot tell a current claim from a retracted one, which alone should stop any normal review. Still, the abstract contains a substantive negative result that deserves serious attention.\n\nThe artifact diagnosis is the most credible piece. The retrieval-impossible control reaching 100% under the flawed metric is a clean smoking gun, and the non-causal cache attention is a plausible second bug. Those two findings, if backed by code, are genuinely useful to the TTT/hybrid-memory community. The decomposition of the failure into a storage bottleneck (surprisal gating is position-biased, early needles stored at 0–1%) and an addressing bottleneck (content addressing extracts ~0.06 of 2.30 nats) is a concrete, falsifiable hypothesis, and the 0% exact match across 2,250 paired trials is a sharp negative result.\n\nThe soft spots are mostly about evidence presentation. The corrected experiments appear only in the abstract: no methods, no tables, no code, no ablation. The body contradicts the abstract, and the reader is left to guess which sections are supposed to be read as the artifact under post-mortem. The stress-test note about the oracle-storage experiment is fair: if the oracle stores ground-truth value vectors and the read-time fusion copies them through the output path, the measured 2.5x “addressing mass” increase does not cleanly isolate addressing. Conversely, any residual non-causal path could depress the 0% EM artificially. The abstract’s numbers cannot be audited without the corrected code.\n\nAll of this does not mean the underlying correction is wrong; the artifact explanation is plausible, and the author is honestly retracting their own prior work. But this manuscript, as submitted, is not a coherent paper—it is a promise of a post-mortem with the post-mortem missing. I would bring the abstract to a reading group as a cautionary tale about evaluation metrics, but I would not cite the 0% result until the corrected code and experiments are actually available.\n\nRecommendation: don't desk reject, but don't send the current text to referees as-is. Ask the authors to resubmit with the corrected experiments in the body, the v1 sections clearly marked as the artifact being analyzed, and the oracle-storage experiment specified in enough detail to rule out content leakage. The artifact diagnosis and the negative result are important enough to warrant a serious referee once the paper is actually complete.","headline":"The abstract retracts the very paper in the body; the new negative result is real work but not verifiable from this submission, and the oracle-storage experiment needs scrutiny.","tokens_in":5540,"tokens_out":3068,"would_cite":false,"duration_ms":32200,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"SR-TTT does not learn retrieval: its reported gains were evaluation artifacts, and corrected evaluation yields 0% exact match across 2,250 trials.","keywords":["Test-Time Training","Surprisal gating","Hybrid memory","Needle-in-a-Haystack","Evaluation artifact","Causal attention","Exact recall","Negative result"],"falsifier":"Run the oracle-storage condition with position information removed from stored needles (shuffle or drop position encodings) while keeping the same read-time attention; if exact match rises above 0% or token accuracy changes materially, the addressing bottleneck is not independent of position. Conversely, store all early needles with a position-unbiased oracle; if exact match remains 0%, storage alone is not the barrier.","tokens_in":4664,"feed_emoji":"⚠️","tokens_out":3601,"duration_ms":33424,"temperature":0.7,"pith_summary":"This paper retracts the retrieval claims of an earlier version of SR-TTT, a hybrid test-time-training model that routes surprising tokens to an exact-attention cache. The original gains came from two evaluation errors: the loss and metric read logits at answer positions rather than one position earlier, and the cache attended non-causally over future tokens, including the answer itself. Under a corrected protocol, the model achieves 0% exact match in all 2,250 paired trials. The failure decomposes into a storage bottleneck (surprisal gating is position-biased and stores early needles at 0-1% containment) and an addressing bottleneck (content addressing extracts only about 0.06 nats of a 2.30-nat needle, and even oracle-perfect storage does not produce retrieval). The paper's value is a cautionary negative result and a reusable diagnostic protocol for surprise-gated memory architectures.","feed_headline":"Corrected evaluation: SR-TTT retrieves nothing, 0% in 2,250 trials","feed_subtitle":"A post-mortem pins the failure on two bottlenecks: surprise gating misses early needles, and content addressing cannot pick the answer out.","key_machinery":"The paper's method for isolating the failure is the two-bottleneck decomposition. Storage is measured by containment: the fraction of a needle's tokens physically present in the residual cache. Addressing is measured by an oracle-storage experiment: feed perfectly stored needles to read-time attention and probe how much of the needle's information (in nats) is extracted, plus per-slot attention supervision. The original artifact is traced to two concrete mechanisms: logits read at answer positions instead of one position earlier, and non-causal cache attention that could attend to the answer token itself. The diagnostic protocol includes startup causality self-tests to catch both errors.","core_discovery":"The central claim is negative: SR-TTT does not learn retrieval, and the earlier positive results were measurement artifacts. With a corrected evaluation that reads logits one position earlier and enforces causality, exact match is 0% in all 2,250 paired trials across every condition. The paper decomposes the failure into two independent, separately measured bottlenecks: storage, where the surprisal filter's EMA threshold requires burn-in so early-context needles are stored at near-zero rates; and addressing, where even an oracle that perfectly stores needles supports content-based attention that extracts only ~0.06 of 2.30 nats, leaving token accuracy statistically unchanged after 2.5x more","pith_inferences":["The two-bottleneck story suggests a deeper design principle: routing by reconstruction loss captures tokens that are hard to compress after burn-in, but recall-critical tokens in early context are not yet distinguishable; a position-aware or curriculum-based routing signal could be a testable fix.","The addressing bottleneck may generalize beyond TTT: any content-addressable memory faced with near-interchangeable, order-dependent items needs explicit slot indices or positional decoding; this is a hypothesis worth testing in linear-attention and state-space hybrids.","One can test the storage bottleneck directly by replacing the surprisal filter with an oracle that stores every token and checking whether exact match rises above 0%; if it does, the storage side is sufficient and addressing is the remaining wall.","The artifact decomposition (logit misalignment plus non-causal attention) may explain other reported 'massive gains' in small-scale memory-augmented models; scan prior work for the same two signatures."],"forward_implications":["If the central claim is right, surprisal-gated memory routing as implemented does not restore exact recall in TTT models; at best it leaves recall at zero.","Storage's position bias means the routing signal fails exactly where long-context memory is most needed: early-context needles require burn-in before they become relatively surprising.","Addressing over near-interchangeable ordered slots appears not to be solvable by content-based attention alone, even with perfectly stored keys and values.","The corrected evaluation protocol (causal self-tests, one-position-earlier logit reading) should be applied to any hybrid memory claim before reporting retrieval gains.","The released negative results provide a baseline for future attempts at surprise-gated memory: any proposed fix must clear the 0% bar in this protocol."],"fun_headline_variants":["SR-TTT learns zero retrieval: 0% in 2,250 corrected trials","Retracted: SR-TTT's gains were artifacts, exact match 0%","SR-TTT post-mortem: no retrieval, two bottlenecks exposed","Corrected SR-TTT: 0% exact match, surprises miss early needles","SR-TTT retrieval is zero: artifact exposed, storage and addressing fail"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The two-bottleneck decomposition assumes that the oracle-storage experiment cleanly isolates the addressing step; if the oracle leaks position information or interacts with the frozen TTT backbone, the measured addressing-mass increase would not cleanly prove that addressing is an independent failure.","fun_headline_variants_meta":{"raw":{"variants":["SR-TTT learns zero retrieval: 0% in 2,250 corrected trials","Retracted: SR-TTT's gains were artifacts, exact match 0%","SR-TTT post-mortem: no retrieval, two bottlenecks exposed","Corrected SR-TTT: 0% exact match, surprises miss early needles","SR-TTT retrieval is zero: artifact exposed, storage and addressing fail"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000379,"raw_usage":{"total_tokens":1935,"prompt_tokens":915,"completion_tokens":1020,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":659,"completion_tokens_details":{"reasoning_tokens":918}},"tokens_in":659,"tokens_out":1020,"duration_ms":8218,"temperature":1.0,"reasoning_tokens":918,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T20:40:19.455151+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the oracle-storage condition with position information removed from stored needles (shuffle or drop position encodings) while keeping the same read-time attention; if exact match rises above 0% or token accuracy changes materially, the addressing bottleneck is not independent of position. Conversely, store all early needles with a position-unbiased oracle; if exact match remains 0%, storage alone is not the barrier.","supporting_citations":[],"review_version":1}