{"id":"9b81c49c-a729-4293-ac9c-f96595bfaec1","arxiv_id":"2505.12392","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SLOT adapts an LLM to each prompt by optimizing a lightweight final-layer vector to minimize prompt loss, boosting benchmark reasoning accuracy by a few points.","lead":"SLOT is a test-time method that tunes a per-prompt vector in the last layer of a frozen language model, using only the prompt text as the training signal. It reports accuracy gains of up to 8.6 points on math benchmarks, but the evaluation lacks error bars and the code is not released.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central causal claim—prompt-only cross-entropy optimization improves answer accuracy—is unvalidated and likely confounded by format-token overfitting; the paper's own negative entries and missing headline result leave it unsubstantiated.","rationale":"The reader identified prompt-only CE as the weakest assumption; I agree and sharpen it. The concern is load-bearing because all reported gains are attributed to δ learned from Eq. (4), and the paper provides no link from that loss to answer accuracy. The circularity of the LMV analysis, the negative entries in Table 2, the suboptimal default hyperparameters, and the mismatch between the abstract's headline number and Table 1 further weaken the evidence. This does not prove the method is ineffective; it means the central claim is not currently substantiated. The proposed control experiment directly tests whether the objective or merely the format template drives the improvement, and whether the effects survive format-independent scoring. If the control supports the claim, the paper would need only modest revision; if not, the main contribution collapses. Thus the conditional verdict is appropriate.","tokens_in":18247,"tokens_out":8427,"duration_ms":85311,"concrete_test":"Run a controlled GSM8K experiment with Qwen2.5-7B under identical T=3, eta=0.01 settings: (a) the exact SLOT prompt-CE objective; (b) the same optimization on a fixed format-only template containing the same tags but no question; (c) a random δ with the same norm. Report answer accuracy both with the paper's parser (which likely requires <answer> tags) and with answer extraction that ignores format. If condition (b) or (c) reproduces most of SLOT's gain, or if SLOT's gain disappears when format is not required, the proxy is confounded and the central claim fails. In addition, compute the per-sample correlation between prompt-CE reduction and correctness on a held-out split; a non-positive correlation would directly disprove the causal story.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The method's central claim is that minimizing the prompt-only loss Eq. (4) yields a δ that improves accuracy on the held-out answer. This requires prompt CE to be a valid proxy for answer correctness, but no mechanism or correlation is established. The proposed mechanism is suspect because δ is optimized to increase likelihood of the exact prompt tokens; on the GSM8K template this directly rewards format/reasoning tokens (think, reasoning) and suppresses end-of-text, so the LMV analysis in Fig. 3 is a consequence of the training objective, not independent evidence of deeper reasoning. The empirical support is weak: Table 2 includes several negative deltas (e.g., Qwen2.5-Math-1.5B GPQA -0.51, Qwen2.5-14B Math500 -0.20, Llama-3.1-8B Math500 -0.60, DeepSeek-R1-Distill-Qwen-32B Math500 -0.60 and GPQA -1.01) with no error bars, and the default T=3, eta=0.01 is suboptimal in the paper's own ablation (Table 4: 26.67% on AIME-24, equal to baseline, vs. 40.00% at T=4, eta=0.05). Moreover, the abstract's headline Qwen2.5-7B GSM8K gain (57.54 to 66.19) does not appear in Table 1, which reports Qwen-7B 51.2 to 54.2; the strongest claimed result is not reproducible from the tables. Thus the causal link between prompt CE and accuracy is the load-bearing assumption, and it is currently unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SLOT (Sample-specific Language Model Optimization at Test-time), a method that, for each input prompt, performs a few gradient steps to optimize a small per-sample vector delta that is added to the final hidden layer before the language-model head. The optimization minimizes the cross-entropy loss on the prompt tokens only, after which the optimized delta is reused during generation. The authors argue that this test-time adaptation better aligns the model with each individual instruction. They evaluate SLOT across several model families (Qwen, Llama, DeepSeek-R1-Distill) on benchmarks including GSM8K, AIME24, Math500, GPQA Diamond, HumanEval, and C-Eval, and report accuracy gains in many cases, with the largest claimed gains being an 8.6% improvement on GSM8K for Qwen2.5-7B and a 68.69% accuracy on GPQA Diamond for DeepSeek-R1-Distill-Llama-70B. They also provide an ablation over the number of optimization steps and learning rate, an inference-time cost analysis, a logit-modulation analysis, and a code snippet.","tokens_in":18618,"tokens_out":4554,"duration_ms":46048,"significance":"If the claimed effects are real, SLOT would be an appealingly simple and parameter-efficient test-time adaptation technique: it adds only a single vector of dimension d per sample, avoids backpropagation through the full model by caching last-layer features, and is reported to improve accuracy across diverse benchmarks and model sizes. The paper also has strengths in breadth (many model-benchmark pairs are reported) and in shipping a concrete algorithm description with a core implementation snippet. However, the significance is substantially constrained by the current evidence: the headline result does not appear in the main results table, no error bars or significance tests are provided, default hyperparameters are inconsistent with the paper's own ablation, and the core causal mechanism (prompt-loss minimization improves answer accuracy) is not validated beyond anecdotal examples. As presented, the paper is best viewed as an initial exploration rather than a fully supported claim of a new state-of-the-art test-time adaptation method.","major_comments":[{"comment":"The headline claim in the abstract and introduction of an 8.6% GSM8K gain for Qwen2.5-7B (57.54% to 66.19%) does not appear in Table 1, which reports Qwen-7B improving from 51.2% to 54.2% on GSM8K. The paper must either reconcile this discrepancy (e.g., by clarifying that Qwen-7B is a different model) or remove the abstract claim, because the central advertised result is not reproducible from the presented tables.","section":"Section 1 and Table 1"},{"comment":"The main experiments use T=3 and eta=0.01, yet the hyperparameter ablation on AIME-24 shows that this configuration achieves only 26.67% accuracy, identical to the baseline, while T=4, eta=0.05 achieves 40.00%. The authors need to justify why they selected the default hyperparameters in Section 4.1 rather than the ablation-optimal ones, and to report results with the optimal settings; otherwise the reported gains may reflect an under-tuned configuration.","section":"Section 4.6, Table 4"},{"comment":"The central assumption that minimizing the prompt-only cross-entropy loss improves accuracy on the held-out answer is not validated. The analysis in Figure 3, which shows that reasoning-related tokens are increased and end-of-text is suppressed, is a direct consequence of optimizing the prompt loss on a template that contains the tokens 'think', 'answer', and '<|endoftext|>'; it does not provide independent evidence of deeper reasoning. A controlled test, such as correlating per-sample prompt-loss decrease with answer correctness, or ablating the objective to exclude format tokens, is needed to support the causal claim.","section":"Section 3.2, Eq. (4) and Section 4.3"},{"comment":"The paper reports numerous small positive and negative deltas (e.g., DeepSeek-R1-Distill-Qwen-32B Math500 -0.60 and GPQA -1.01, Llama-3.1-8B Math500 -0.60, Qwen2.5-Math-1.5B GPQA -0.51) without error bars, confidence intervals, or statistical significance tests, and the authors explicitly state in the checklist that such reporting is not provided. On small benchmarks like AIME24, where one question corresponds to several percentage points, these differences may be within sampling noise. The authors should report multiple runs with variance estimates and, where possible, significance tests so that the reader can assess which improvements are reliable.","section":"Tables 2 and 3; Checklist item 7"},{"comment":"The experiments do not include any same-compute test-time baselines, such as continuous prompt tuning with a comparable number of gradient steps, LoRA-style adaptation at test time, or even simple logit-adjustment heuristics. Without such comparisons, the claim that SLOT is a competitive test-time technique relative to existing efficient adaptation approaches is not established; the current comparison is only against the unmodified base model.","section":"Section 4.3"}],"minor_comments":[{"comment":"The pseudocode in Algorithm 1 recomputes the hidden features H inside each optimization iteration, which contradicts the text in Section 3.3 and the appendix that state the features are cached and reused; the pseudocode should be updated to reflect the caching implementation.","section":"Algorithm 1"},{"comment":"There is a typo in the table header: 'A VERAGE' should be 'AVERAGE'.","section":"Table 1"},{"comment":"The claim that 68.69% on GPQA Diamond is 'SOTA' for 70B-level open-source models is not accompanied by a citation or a leaderboard reference; please provide the source of this comparison.","section":"Abstract and Section 4.3"},{"comment":"The implementation in Listing 1 uses an os.environ variable as a global flag to switch between prompt-stage and generation-stage behavior; this approach is fragile and not thread-safe, and the released code should use a cleaner mechanism if the authors intend the code to be widely used.","section":"Appendix A"},{"comment":"The checklist states that the method has no obvious limitations, yet Table 2 contains several negative results and Table 4 shows sensitivity to hyperparameters; the authors should provide a limitations discussion that addresses these observations.","section":"NeurIPS Checklist, item 2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising core idea, but the empirical support is substantially weaker than the abstract suggests. The main issues are (1) the headline GSM8K result is missing from the results tables, (2) the default hyperparameters differ from the ablation-optimal ones, (3) the causal link between prompt-loss minimization and answer accuracy is not demonstrated, and (4) the lack of error bars and same-compute baselines makes it difficult to judge the strength of the effect. These issues are addressable within the scope of a revision, but they require additional experiments and careful rewriting of the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest take: the core idea is simple and cheap. Learn a per-sample additive vector on the final hidden states by minimizing the prompt's cross-entropy, then reuse it during generation. That is a reasonable new application of known components—prompt tuning plus test-time adaptation. The efficiency argument is sound: cache the last-layer features, update only the input to the LM head, and the overhead is genuinely small. I'd buy that.\n\nWhat the paper does well: it states the method clearly, runs a wide sweep of models and benchmarks, and includes negative results in Table 2 instead of hiding them. The LMV analysis in Figure 3 is an honest attempt at interpretation, even if the conclusion mostly says the optimizer learns to favor 'think' and suppress end-of-text.\n\nThe soft spots are real, and one is serious. The central assumption—that prompt-only cross-entropy optimization improves held-out answer accuracy—is never validated. It may simply improve format compliance on template prompts, which is a confound, not evidence of deeper reasoning. Table 2 shows several negative deltas (e.g., -1.01 on GPQA for DeepSeek-R1-Distill-Qwen-32B) with no error bars and no explanation. There is no statistical significance anywhere. The ablation in Table 4 recommends T=4, eta=0.05, yet the main results use T=3, eta=0.01—suboptimal by the paper's own numbers. Worse, the abstract's headline Qwen2.5-7B GSM8K gain (57.54 to 66.19) does not appear in Table 1, which reports Qwen-7B going from 51.2 to 54.2. That is a glaring inconsistency for the paper's showcase claim. The code is not released despite the abstract saying it is; the NeurIPS checklist admits as much.\n\nNone of this kills the idea. The method is plausible, inexpensive, and could be genuinely useful for deployment if the gains survive proper evaluation. But as it stands, the central causal claim is unsubstantiated. The authors also answer 'NA' to limitations, which is unsupported given the negative results they themselves report.\n\nWho this is for: researchers working on test-time adaptation or efficient inference for LLMs. A serious referee should engage—the idea deserves scrutiny and a careful redo of the evaluation. My recommendation: send to peer review, but expect heavy revision. The authors need to fix the abstract/table mismatch, report variance and significance, compare against same-compute baselines like test-time prompt tuning, and either address the format-token confound or soften the mechanism claim.","headline":"A cheap test-time adaptation idea with a plausible mechanism, but the evidence is under-powered and the abstract's headline number doesn't match the tables.","tokens_in":767,"tokens_out":722,"would_cite":false,"duration_ms":28795,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Optimizing a tiny additive vector on the prompt's own text at test time improves LLM instruction following and reasoning accuracy, with reported gains of up to 8.6 points on GSM8K.","keywords":["test-time adaptation","language model optimization","sample-specific parameters","prompt cross-entropy loss","logit modulation","reasoning benchmarks","parameter-efficient inference"],"falsifier":"Run SLOT on prompts whose words are randomly shuffled but whose answers are unchanged; if accuracy still rises, the effect is not specific to understanding the instruction, and if prompt loss decreases while answer accuracy falls, the proxy is broken.","tokens_in":18066,"feed_emoji":"🧠","tokens_out":5067,"duration_ms":47832,"temperature":0.7,"pith_summary":"SLOT claims that a language model can be made more accurate on a single question by spending a few gradient steps at inference time on the question's own text, without retraining or fine-tuning. The adaptation is a small per-sample vector $\\delta$ added to the final hidden layer before the output head, and the only supervision is the standard next-token cross-entropy loss on the prompt itself. The paper reports consistent gains across several open models and reasoning benchmarks, including an 8.6-point improvement for Qwen2.5-7B on GSM8K and a state-of-the-art 68.69% for DeepSeek-R1-Distill-Llama-70B on GPQA Diamond. If the link between prompt likelihood and answer correctness holds, this gives a cheap, universal way to sharpen instruction following.","feed_headline":"A tiny per-prompt vector lifts LLM accuracy up to 8.6%","feed_subtitle":"SLOT optimizes a single vector on the prompt's own text, boosting GSM8K and GPQA scores with minimal extra compute.","key_machinery":"The load-bearing object is the sample-specific additive parameter $\\delta\\in\\mathbb{R}^{1\\times d}$, broadcast over the sequence and added to the cached last-layer hidden features before the LM head. Its effect is equivalent to an additive shift $W_{\\mathrm{LM}}\\delta$ on the vocabulary logits, named the Logit Modulation Vector. The two-stage scheme—optimize $\\delta$ on the prompt for $T$ steps, then reuse the fixed $\\delta$ during autoregressive generation—keeps the extra cost small, and zero initialization preserves the original model at step zero. The gradient path is short because only the final linear head participates in the update.","core_discovery":"The central claim is that optimizing a zero-initialized additive vector on the final hidden features, $H' = H + \\delta$, with a few AdamW steps against the prompt's own cross-entropy loss, biases subsequent generation toward the intended answer. Because $\\delta$ touches only the layer just before the language-model head, the per-sample optimization reuses cached features and costs only the head computation, so the method is nearly free at generation time. The paper interprets the resulting logit shift, $W_{\\mathrm{LM}}\\delta$, as a Logit Modulation Vector and observes that it raises tokens like 'think' and 'reasoning' while suppressing numbers and the end-of-text token, which the authors read as encouraging deeper reasoning before answering. Empirically, SLOT improves both answer accuracy and format compliance on GSM8K, AIME24, Math500, GPQA Diamond, C-Eval, and HumanEval across Qwen, Llama, and DeepSeek-R1 models.","pith_inferences":["The prompt-loss proxy is likely to be least reliable when the prompt is already highly predictable or when the correct answer requires information the prompt does not contain; testing SLOT on trivia or open-ended generation would separate format alignment from factual gains.","The Logit Modulation Vector analysis suggests SLOT is effectively a learned decoding bias; a cheaper or more controllable variant might compute the bias analytically or tune it per task rather than per sample.","Negative results in Table 2 hint that the method can overfit to prompt surface form; comparing SLOT against prompt paraphrases or shuffled token orders would reveal whether it captures instruction semantics or just local token statistics."],"forward_implications":["Any transformer with an exposed pre-softmax layer can carry SLOT as a drop-in inference wrapper; no training data, labels, or model access beyond the final hidden states is needed.","The overhead is bounded by the head dimension and prompt length, not by model size: the reported wall-clock increase is about 7.9% with five optimization steps on GSM8K.","Instructions with strict output formats should become easier to satisfy, since format accuracy rises alongside answer accuracy as $T$ increases.","The reported negative entries (e.g., Qwen2.5-14B on Math500 and DeepSeek-R1-Distill-Qwen-32B on GPQA Diamond) are part of the paper's own data, so the benefit is not guaranteed across all models and datasets."],"supporting_citations":[{"why":"Test-time training with self-supervision; supplies the TTA paradigm SLOT extends.","marker":"[32]"},{"why":"TENT entropy minimization; the standard TTA baseline SLOT contrasts with.","marker":"[33]"},{"why":"Test-time reinforcement learning; the most recent per-instance adaptation approach SLOT compares against.","marker":"[42]"},{"why":"In-context learning; frames the alternative no-gradient adaptation that SLOT distinguishes itself from.","marker":"[23]"},{"why":"LoRA; the parameter-efficient fine-tuning baseline, showing task-level versus sample-level adaptation.","marker":"[12]"},{"why":"AdamW; the optimizer used for the per-sample delta updates.","marker":"[22]"},{"why":"GSM8K; one of the main benchmarks where SLOT reports accuracy gains.","marker":"[4]"},{"why":"GPQA Diamond; the benchmark where SLOT reports state-of-the-art 70B-level accuracy.","marker":"[28]"}],"fun_headline_variants":["Test-time vector tweak lifts LLM accuracy by 8.6%","Per-prompt parameter vector boosts LLM scores efficiently","SLOT: tiny per-sample vector improves LLM reasoning","Optimize a small vector at test time, gain 8.6% accuracy","Last-layer vector update sharpens LLM output per prompt"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that lowering the language-modeling loss on the prompt text itself makes the model more likely to produce the correct held-out answer, and this proxy link is empirical rather than derived.","fun_headline_variants_meta":{"raw":{"variants":["Test-time vector tweak lifts LLM accuracy by 8.6%","Per-prompt parameter vector boosts LLM scores efficiently","SLOT: tiny per-sample vector improves LLM reasoning","Optimize a small vector at test time, gain 8.6% accuracy","Last-layer vector update sharpens LLM output per prompt"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000355,"raw_usage":{"total_tokens":1947,"prompt_tokens":983,"completion_tokens":964,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":875}},"tokens_in":599,"tokens_out":964,"duration_ms":8022,"temperature":1.0,"reasoning_tokens":875,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:34:07.404530+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SLOT on prompts whose words are randomly shuffled but whose answers are unchanged; if accuracy still rises, the effect is not specific to understanding the instruction, and if prompt loss decreases while answer accuracy falls, the proxy is broken.","supporting_citations":[{"cited_title":"Test- time training with self-supervision for generalization under distribution shifts","cited_arxiv_id":null,"evidence_quote":"Test-time training with self-supervision; supplies the TTA paradigm SLOT extends."}],"review_version":1}