{"id":"2a9e8618-b939-4cc2-8b01-45fac9743773","arxiv_id":"2412.12157","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"The paper derives a sufficient condition for when a demonstration should reduce an LLM's math reasoning loss in a linear attention model, and builds an example selection and rejection method, LMS3, from a heuristic version of that condition.","lead":"This paper asks why adding a few solved examples to a math prompt helps some large language models and hurts others, and proposes a way to pick or reject examples automatically. The authors derive a condition from a simplified model of attention and turn it into a scoring method called LMS3, which they test on three math benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's proof identifies the test-dependent attention matrix W0 (Eq. 4) with the pretraining optimum W_hat (Eq. 6); without this premise the influence-function expansion (Eq. 8) does not apply and the central bound is unproved.","rationale":"The reader's verdict is REJECT, and the stated weakest assumption is that the theorem is proven only for a single-layer linear attention surrogate under unverified conditions. The concern identified here is more specific and more internal: even within the linear surrogate, the proof conflates the test-dependent rank-one matrix W0 (Eq. 4) with the pretraining loss minimizer W_hat (Eq. 6). Standard influence functions require expanding around a stationary point of the training objective; the paper supplies no argument that W0 is such a point. This makes the derivation of Eq. (7) unsupported as written. Because the abstract's central claim—that reasoning efficacy is bounded by semantic similarity and inference stability—rests directly on Theorem 1, this internal gap undermines the theoretical contribution. The empirical results may still support LMS3 as a heuristic prompt-selection method, but they do not fix the proof. This does not change the reader's REJECT verdict; it reinforces it with a sharper technical defect. I set agreement_with_reader to 'partial' because the reader correctly identified the linear-surrogate/LLM gap but did not isolate the W_hat = W0 conflation inside the theorem.","tokens_in":19586,"tokens_out":11548,"duration_ms":119320,"concrete_test":"Settle analytically with a scalar linear model: let d=d'=1, F(z)=Wz, and use squared loss L(F(z))=(Wz - y)^2. Choose Dpre with z_i=1, y_i=2, so the pretraining minimizer is W_hat = Σ z_i y_i / Σ z_i^2 = 2. Set htest=1, WV=WK=1, so Eq. (4) gives W0 = 1, which differs from W_hat to 2. The Hessian is 2Σ z_i^2 > 0 and Lipschitz continuity holds, so the paper's other assumptions can be met. Compute the gradient of the pretraining loss at W0: it is nonzero, so the stationarity premise required for Eq. (8) fails. If this configuration is admitted by the paper's assumptions, then Theorem 1's proof as written is invalid; if the authors intended Dpre to force W_hat = W0, they must provide such a construction explicitly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is in Section 3, where the paper defines the 'initialized parameters' as W0 = (WV/√d) htest (WK htest)^T (Eq. 4) and then states that the pretrained parameters W_hat from Eq. (6) are exactly W0. The influence-function formula (Eq. 8) and the Taylor expansion in Eq. (9) are only valid when W_hat is a stationary point of the pretraining objective. But W0 is a rank-one matrix constructed from the test representation htest; there is no reason it minimizes any fixed pretraining loss over Dpre, and generically it does not. Unless Dpre and the loss are engineered so that W0 is exactly the minimizer, the expansion in Eq. (9) is not around the true optimum, so the sign of the loss change claimed in Theorem 1 is not established even for the linear surrogate. The LMS3 score (Sim(X)*Stab(X)) inherits this gap because it is justified by Eq. (7). This is not only an unverified empirical assumption; it is a missing premise inside the proof. A secondary internal mismatch exists between Eq. (16), which has an extra ||htest|| factor, and Eq. (7), but the W_hat = W0 conflation is the more fundamental defect.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies when in-context demonstrations improve LLM performance on mathematical reasoning. It models the one-shot ICL forward pass as linear attention, interprets the demonstration as one gradient step of a linear function, and proves (Theorems 1 and 2) a sufficient condition under which adding one or k demonstrations reduces the prediction loss. The condition involves a LLM-oriented semantic similarity and a quantity called inference stability. Based on this condition, the paper proposes a demonstration selection method, LMS3, with a rejection mechanism, and reports experiments on MAWPS, GSM8K, and MATH with Llama2-13B, Llama3-8B, ChatGPT, and GPT-4.","tokens_in":19836,"tokens_out":6932,"duration_ms":78290,"significance":"The question addressed is important: demonstrations can hurt, and a quantitative account of when they help would be a valuable theoretical contribution. The empirical study is broad, comparing ten baselines across three datasets and several LLMs, and LMS3 is competitive in the reported tables. However, the central theoretical claim is not established: the proof of Theorem 1 requires identifying the test-dependent initialization W0 with the pretrained optimum \\hat W, which is not justified and is generally false. Because the proposed scoring function is derived from Eq. (7), this flaw undermines the claimed theoretical grounding of LMS3. The paper also contains an internal inconsistency between Eq. (16) and Eq. (7). The empirical method may still be useful as an unsupervised heuristic, but the manuscript's main claimed contribution is a proof, and that proof is not sound.","major_comments":[{"comment":"The proof of Theorem 1 relies on the statement, made just after Eq. (6), that the pretrained parameters \\hat W are exactly the initialized parameters W0 defined in Eq. (4). This is not a harmless simplification. W0 = (WV/\\sqrt{d}) htest (WK htest)^T is a rank-one matrix constructed from the test representation htest, while \\hat W is defined as the minimizer of the pretraining loss over Dpre and is a fixed parameter of the model. There is no reason for these matrices to be equal, and generically they are not; equality would require Dpre and the loss to be engineered so that a test-dependent rank-one matrix minimizes the pretraining objective for every test point. The influence-function expansion in Eqs. (8)-(9) is valid only when the expansion point is a stationary point of the objective being upweighted. Since W0 is not shown to be such a stationary point, Eq. (9) is not justified, and the inequality in Theorem 1 is not proved even for the linear surrogate. Because the LMS3 score is justified by Eq. (7), this defect is load-bearing for the paper's central claim.","section":"Section 3, Eqs. (4), (6)-(9)"},{"comment":"There is a mathematical inconsistency in the proof. Substituting Eq. (13) into Eq. (12) gives the term mu * ||WV/\\sqrt{d} htest|| * ||WK htest|| * ||htest - z0|| * ||htest||, which equals mu * C1 * ||htest - z0|| because C1 already contains the factor ||htest||. Eq. (16), however, writes this term as mu * C1 * ||htest - z0|| * ||htest||, introducing an extra ||htest|| factor. The sufficient condition in Eq. (7) does not contain this extra factor, so Eq. (16) and Eq. (7) are inconsistent. This needs to be corrected before the derivation of the score can be regarded as internally coherent.","section":"Section 3, Eqs. (12)-(16) versus Eq. (7)"},{"comment":"The passage from the theorem to the LMS3 score is not a derivation but an approximation that drops several quantities appearing in the theorem. The sufficient condition in Eq. (7) involves the eigenvalue ratio lambda_dd'/lambda_1, the gradient norm ||\\nabla_W L(htest, \\hat W)||, and the additive term mu*C1*||htest - z0||. Minimizing Score(X) = Sim(X) * Stab(X) is not equivalent to satisfying Eq. (7), because the additive mu*C1 term is discarded and the unknown eigenvalue ratio and gradient norm are ignored. The paper should state clearly that Eq. (24) is a heuristic inspired by the theorem rather than a consequence of it.","section":"Section 4, Eqs. (21)-(24)"}],"minor_comments":[{"comment":"Please state explicitly how the rejection threshold lambda is chosen for the results in Tables 2 and 3. If lambda is selected by inspecting test-set curves such as those in Figure 5, the reported gains may be optimistic; if a validation set is used, that should be stated and the validation split described.","section":"Section 5.4, Figure 5 and Table 2"},{"comment":"Stab(X) is described as the length of the gradient of the loss on X, but Eq. (22) defines it as the norm of \\nabla_F L, the gradient with respect to the function output, not the parameter gradient. Please make this distinction explicit.","section":"Section 4, Eq. (22)"},{"comment":"The redefinition of WK as WK^T WQ is introduced informally. The notation in Eq. (21) and in the sentence 'recall z0 = WKh and WK is indeed WK^T * WQ' is easy to misread; define a new symbol or otherwise make the redefinition unambiguous.","section":"Section 3, notation around Eq. (3)"},{"comment":"The paper claims in the checklist that all code is included, but no code appendix is visible in the manuscript. Please ensure the code and hyperparameter details are either included or removed from the checklist claims.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The empirical comparison is useful and the negative-result observation about one-shot demonstrations is interesting, but the paper's headline contribution is a theoretical proof, and that proof has a load-bearing gap that cannot be repaired by local edits: the equality W0 = \\hat W is essential to the influence-function argument and is generally false. A resubmission could either supply a corrected theorem with a properly stated surrogate model or reposition the paper as a purely empirical study of the LMS3 heuristic. As written, the central claim in the abstract is not supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is best read as an empirical prompt-selection method with a theoretical wrapper that doesn't quite close. The wrapper is the problem. Theorem 1 hinges on identifying W0—a rank-one matrix built from the test representation htest—with the pretraining optimum Ŵ. The influence-function expansion in Eq. (8) and the Taylor argument in Eq. (9) are only valid around a stationary point of the pretraining objective. Nothing in the construction of Eq. (4) guarantees that W0 minimizes any loss over Dpre; generically, it does not. So the sign of the loss change in Theorem 1 is not established, and the LMS3 score inherits that gap. The stress-test note is right: this is a missing premise inside the proof, not just an unverified empirical assumption. There is also a secondary mismatch between Eq. (16) and Eq. (7) involving an extra ||htest|| factor, but that is fixable; the Ŵ = W0 step is not.\n\nWhat the paper does well is the empirical package. LMS3 balances LLM-oriented semantic similarity and an inference-stability term, adds a rejection mechanism that abstains from demonstrations when the best candidate looks too dissimilar, and the experiments on Llama2-13B and Llama3-8B across MAWPS, GSM8K, and MATH show consistent gains over the baselines in one-shot and few-shot settings. The rejection mechanism is a genuinely useful idea, and the case study in Figure 6 is illustrative. I also appreciate that the authors report statistical significance and show the effect of the rejection threshold λ. That is careful empirical work.\n\nThe soft spots beyond the theorem: the scoring function in Eq. (24) ignores the unknown constants from the bound, and the rejection threshold λ is tuned on the validation set; so the 'theoretical advantage' claim overstates what the theory actually delivers. The abstract says 'consistent improvements on all datasets,' but in Table 3 LMS3 on GPT-4 MATH (0.752) is below zero-shot (0.758), and the paper attributes that to all methods being negative. That is fine as an observation, but it does contradict the unqualified abstract sentence. The O(M+N) complexity claim also assumes the per-input representations can be precomputed, which is fair but worth stating more carefully.\n\nWho gets value from this? Someone building a demonstration-selection heuristic for math reasoning would find LMS3 useful and the comparison against Similar-ICL and influence-based methods informative. A theory reader will find the proof gap quickly and may stop there. I would send it to a serious referee because the empirical method deserves scrutiny and the theoretical flaw is precisely what a good referee should catch; the paper should not be desk-rejected, but it needs major revision before it can be accepted. My own verdict on the central claim is skeptical: the bound is not proved as stated.","headline":"The empirical demonstration-selection method is decent and the experiments are honest, but the paper's central theorem has a load-bearing gap: the proof conflates the test-dependent initialization W0 with the pretrained optimum, so the influence-function expansion and the bound do not follow.","tokens_in":20375,"tokens_out":1514,"would_cite":false,"duration_ms":17298,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proves that in-context demonstrations help mathematical reasoning exactly when the model finds them semantically similar and answers them stably, and that a selector built on this condition improves accuracy consistently across…","keywords":["in-context learning","mathematical reasoning","demonstration selection","influence functions","linear attention","semantic similarity","inference stability","few-shot learning"],"falsifier":"On an open model such as Llama-3-8B, take a set of test problems and candidate demonstrations, compute $\\mathrm{Sim}(X)$, $\\mathrm{Stab}(X)$, and the inequality (7) threshold, then measure the actual change in answer accuracy when each demonstration is prepended; if demonstrations that satisfy the theorem's condition frequently fail to reduce the loss, or demonstrations that violate it frequently improve it, the linear-attention surrogate does not transfer to the real model.","tokens_in":19362,"feed_emoji":"🧮","tokens_out":9298,"duration_ms":90073,"temperature":0.7,"pith_summary":"This paper sets out to answer when adding a worked example to a math prompt helps a large language model reason better, and when it hurts. It claims to prove that the prediction loss of a test problem after adding a demonstration is bounded by two quantities that depend on the inference model itself: how similar the demonstration looks to the test problem in the model's representation space, and how stably the model can reason out the demonstration's answer. If both are good enough, one-shot reasoning beats zero-shot; otherwise the example can actively mislead the model. The same condition extends to few-shot prompting through an additive rule, so demonstrations can be ranked almost independently. Based on this, the paper proposes LMS3, a demonstration selector that scores candidates by the product of the two quantities and rejects the shot entirely when the best candidate is still too dissimilar.","feed_headline":"One-shot math reasoning can backfire — theory explains when it helps","feed_subtitle":"Proof ties few-shot math gains to the model's own similarity and stability scores, then turns them into a selector.","key_machinery":"The machinery is an influence-function analysis applied to a linearized attention layer. The paper follows the interpretation of in-context learning as implicit gradient descent by dropping the softmax from the attention block and writing $F_{\\rm ICL}(h_{\\rm test})$ as the output of a linear function $F(z)=W\\cdot z$ updated by one gradient step on the demonstration representation $z_0=W_K h$. Upweighting this demonstration by $\\epsilon=1/|D_{\\rm pre}|$ changes the test loss through the classic influence function $-\\nabla_W L(h_{\\rm test},\\hat W)^T H_{\\hat W}^{-1}\\nabla_W L(z_0,\\hat W)$; Theorem 1 bounds this change below using the largest and smallest eigenvalues of the Hessian and the Lipschitz constant of the loss gradient, producing the sufficient condition that yields $\\mathrm{Sim}(X)$ and $\\mathrm{Stab}(X)$. Theorem 2 repeats the argument for $k$ samples and shows the condition is additive over demonstrations. The practical selection method LMS3 converts this into a score $\\mathrm{Sim}(X)\\cdot\\mathrm{Stab}(X)$ with $O(M+N)$ LLM calls, plus a rank-based rejection threshold.","core_discovery":"The central claim is that the effect of an in-context demonstration on mathematical reasoning performance is quantifiable: under a linear-attention approximation of the Transformer, adding a demonstration $X$ reduces the prediction loss on a test problem $X_{\\rm test}$ exactly when the sufficient condition in inequality (7) holds, namely when the demonstration is close to the test problem in the model's own key-query space and small in the gradient norm of its reasoning loss. The paper writes this bound as a combination of an LLM-oriented semantic similarity, $\\mathrm{Sim}(X)=\\|h_{\\rm test}-W_K^TW_Qh\\|$, and an inference stability of the demonstration, $\\mathrm{Stab}(X)=\\|W_V h/\\sqrt{d}\\|$. Theorem 2 extends the result to $k$ demonstrations with an additive condition, implying that the selection of each demonstration can be treated approximately independently. On this basis the paper argues that the sufficient condition is what makes one-shot and few-shot prompting work, and that a selector which balances these two quantities, together with a rejection rule based on the similarity ranking, will improve reasoning accuracy beyond zero-shot levels.","pith_inferences":["If the linear-attention surrogate holds beyond this setting, the same two-term score should transfer to non-mathematical generation tasks, since the derivation does not use anything specific to math; that is a testable extension the paper leaves open.","The rejection rule suggests a broader design principle for retrieval-augmented prompting: when no candidate example clears a similarity threshold, the safer default is to omit demonstrations altogether, and a relative threshold like the paper's could be adapted to other domains.","The paper computes representations by feeding each problem into the LLM individually; a practical extension would be to approximate $\\mathrm{Sim}(X)$ with hidden states or embeddings from a smaller model, which would let closed models benefit from LMS3 without internal parameter access.","Theorem 2's additivity implies that ordering effects in few-shot prompts might be explainable by how each demonstration's stability term interacts with the accumulated loss, which could connect to existing work on demonstration ordering."],"forward_implications":["Demonstration selection for mathematical reasoning can be reduced to ranking candidates by a product of two model-dependent quantities, both computable in $O(M+N)$ LLM calls.","The additive form of Theorem 2 justifies choosing few-shot demonstrations one at a time rather than jointly optimizing the whole set.","When the best available demonstration is still too dissimilar in the model's representation space, the paper's rejection mechanism says to fall back to zero-shot prompting rather than force an example.","Because the two scores are defined for the inference model itself, the same selection pipeline can be adapted to different backbones and can reuse demonstrations picked with one model when testing on a stronger model."],"supporting_citations":[{"why":"It supplies the linear-attention and meta-gradient interpretation that turns in-context learning into one gradient step on a linear function.","marker":"Dai et al. 2023"},{"why":"It provides the influence-function formula the proof uses to compute how upweighting a demonstration changes the test loss.","marker":"Koh and Liang 2017"},{"why":"It is cited as the classical source for the residual and influence analysis used in the derivation.","marker":"Ling 1984"},{"why":"It is the prior influence-based ICL demonstration-selection work whose positive-definite Hessian assumption and setup this paper adapts.","marker":"Van, Wu et al. 2024"},{"why":"It defines the Similar-ICL baseline that the paper generalizes by making semantic similarity depend on the inference LLM.","marker":"Liu et al. 2022"},{"why":"It provides the MATH benchmark used to test LMS3 against zero-shot and selection baselines.","marker":"Hendrycks et al. 2021"},{"why":"It provides the GSM8K benchmark used in the one-shot, few-shot, and generalization experiments.","marker":"Cobbe et al. 2021"},{"why":"It provides the MAWPS dataset used as the third benchmark in the experiments.","marker":"Koncel-Kedziorski et al. 2016"}],"fun_headline_variants":["Math few-shot backfires? Theory says check similarity","Why math demos fail: a bound on in-context gains","New proof picks math demos that actually help LLMs","LLM math reasoning: stability plus similarity predicts wins","When few-shot math hurts: theory and a fix in LMS3"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole argument rests on treating a real large language model's in-context behavior as a single linear attention layer in which the demonstration acts as one very small gradient step, and on assuming the pretrained parameters sit at a loss minimum with a positive-definite Hessian; the theorems are proven for that surrogate model, not for the full LLM.","fun_headline_variants_meta":{"raw":{"variants":["Math few-shot backfires? Theory says check similarity","Why math demos fail: a bound on in-context gains","New proof picks math demos that actually help LLMs","LLM math reasoning: stability plus similarity predicts wins","When few-shot math hurts: theory and a fix in LMS3"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000139,"raw_usage":{"total_tokens":1171,"prompt_tokens":976,"completion_tokens":195,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":112}},"tokens_in":592,"tokens_out":195,"duration_ms":3227,"temperature":1.0,"reasoning_tokens":112,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:57:59.850127+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On an open model such as Llama-3-8B, take a set of test problems and candidate demonstrations, compute $\\mathrm{Sim}(X)$, $\\mathrm{Stab}(X)$, and the inequality (7) threshold, then measure the actual change in answer accuracy when each demonstration is prepended; if demonstrations that satisfy the theorem's condition frequently fail to reduce the loss, or demonstrations that violate it frequently improve it, the linear-attention surrogate does not transfer to the real model.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the linear-attention and meta-gradient interpretation that turns in-context learning into one gradient step on a linear function."},{"cited_title":"W.; and Liang, P","cited_arxiv_id":null,"evidence_quote":"It provides the influence-function formula the proof uses to compute how upweighting a demonstration changes the test loss."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It is cited as the classical source for the residual and influence analysis used in the derivation."},{"cited_title":"B.; Carin, L.; and Chen, W","cited_arxiv_id":null,"evidence_quote":"It defines the Similar-ICL baseline that the paper generalizes by making semantic similarity depend on the inference LLM."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the MAWPS dataset used as the third benchmark in the experiments."}],"review_version":1}