What Makes In-context Learning Effective for Mathematical Reasoning: A Theoretical Analysis

Chaokun Wang; ChengXiang Zhai; Enhong Chen; Jiayu Liu; Xunpeng Huang; Zhenya Huang

REVIEW 3 major objections 4 minor 1 cited by

What Makes In-context Learning Effective for Mathematical Reasoning: A Theoretical Analysis

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proves that in-context demonstrations help mathematical reasoning exactly when the model finds them semantically similar and answers them stably, and that a selector built on this condition improves accuracy consistently across…

desk verdict The empirical demonstration-selection method is decent and the experiments are honest, but the paper's central theorem has a load-bearing gap: the proof conflates the test-dependent initialization W0 with the pretrained optimum, so the influence-function expansion and the bound do not follow. read the letter →

arxiv 2412.12157 v1 pith:KOJWUW4J submitted 2024-12-11 cs.CL cs.AI

Jiayu Liu , Zhenya Huang , Chaokun Wang , Xunpeng Huang , Chengxiang Zhai , Enhong Chen This is my paper · ORCID

classification cs.CLcs.AI

keywords in-contextlearningmathematicalreasoningdemonstrationselectioninfluencefunctionslinearattentionsemanticsimilarityinferencestabilityfew-shot

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to answer when adding a worked example to a math prompt helps a large language model reason better, and when it hurts. It claims to prove that the prediction loss of a test problem after adding a demonstration is bounded by two quantities that depend on the inference model itself: how similar the demonstration looks to the test problem in the model's representation space, and how stably the model can reason out the demonstration's answer. If both are good enough, one-shot reasoning beats zero-shot; otherwise the example can actively mislead the model. The same condition extends to few-shot prompting through an additive rule, so demonstrations can be ranked almost independently. Based on this, the paper proposes LMS3, a demonstration selector that scores candidates by the product of the two quantities and rejects the shot entirely when the best candidate is still too dissimilar.

What carries the argument

The machinery is an influence-function analysis applied to a linearized attention layer. The paper follows the interpretation of in-context learning as implicit gradient descent by dropping the softmax from the attention block and writing $F_{\rm ICL}(h_{\rm test})$ as the output of a linear function $F(z)=W\cdot z$ updated by one gradient step on the demonstration representation $z_0=W_K h$. Upweighting this demonstration by $\epsilon=1/|D_{\rm pre}|$ changes the test loss through the classic influence function $-\nabla_W L(h_{\rm test},\hat W)^T H_{\hat W}^{-1}\nabla_W L(z_0,\hat W)$; Theorem 1 bounds this change below using the largest and smallest eigenvalues of the Hessian and the Lipschitz constant of the loss gradient, producing the sufficient condition that yields $\mathrm{Sim}(X)$ and $\mathrm{Stab}(X)$. Theorem 2 repeats the argument for $k$ samples and shows the condition is additive over demonstrations. The practical selection method LMS3 converts this into a score $\mathrm{Sim}(X)\cdot\mathrm{Stab}(X)$ with $O(M+N)$ LLM calls, plus a rank-based rejection threshold.

What would settle it

On an open model such as Llama-3-8B, take a set of test problems and candidate demonstrations, compute $\mathrm{Sim}(X)$, $\mathrm{Stab}(X)$, and the inequality (7) threshold, then measure the actual change in answer accuracy when each demonstration is prepended; if demonstrations that satisfy the theorem's condition frequently fail to reduce the loss, or demonstrations that violate it frequently improve it, the linear-attention surrogate does not transfer to the real model.

Watch

Extended reading notes

Core claim

The central claim is that the effect of an in-context demonstration on mathematical reasoning performance is quantifiable: under a linear-attention approximation of the Transformer, adding a demonstration $X$ reduces the prediction loss on a test problem $X_{\rm test}$ exactly when the sufficient condition in inequality (7) holds, namely when the demonstration is close to the test problem in the model's own key-query space and small in the gradient norm of its reasoning loss. The paper writes this bound as a combination of an LLM-oriented semantic similarity, $\mathrm{Sim}(X)=\|h_{\rm test}-W_K^TW_Qh\|$, and an inference stability of the demonstration, $\mathrm{Stab}(X)=\|W_V h/\sqrt{d}\|$. Theorem 2 extends the result to $k$ demonstrations with an additive condition, implying that the selection of each demonstration can be treated approximately independently. On this basis the paper argues that the sufficient condition is what makes one-shot and few-shot prompting work, and that a selector which balances these two quantities, together with a rejection rule based on the similarity ranking, will improve reasoning accuracy beyond zero-shot levels.

Load-bearing premise

The whole argument rests on treating a real large language model's in-context behavior as a single linear attention layer in which the demonstration acts as one very small gradient step, and on assuming the pretrained parameters sit at a loss minimum with a positive-definite Hessian; the theorems are proven for that surrogate model, not for the full LLM.

Editorial extensions

If this is right

Demonstration selection for mathematical reasoning can be reduced to ranking candidates by a product of two model-dependent quantities, both computable in $O(M+N)$ LLM calls.
The additive form of Theorem 2 justifies choosing few-shot demonstrations one at a time rather than jointly optimizing the whole set.
When the best available demonstration is still too dissimilar in the model's representation space, the paper's rejection mechanism says to fall back to zero-shot prompting rather than force an example.
Because the two scores are defined for the inference model itself, the same selection pipeline can be adapted to different backbones and can reuse demonstrations picked with one model when testing on a stronger model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the linear-attention surrogate holds beyond this setting, the same two-term score should transfer to non-mathematical generation tasks, since the derivation does not use anything specific to math; that is a testable extension the paper leaves open.
The rejection rule suggests a broader design principle for retrieval-augmented prompting: when no candidate example clears a similarity threshold, the safer default is to omit demonstrations altogether, and a relative threshold like the paper's could be adapted to other domains.
The paper computes representations by feeding each problem into the LLM individually; a practical extension would be to approximate $\mathrm{Sim}(X)$ with hidden states or embeddings from a smaller model, which would let closed models benefit from LMS3 without internal parameter access.
Theorem 2's additivity implies that ordering effects in few-shot prompts might be explainable by how each demonstration's stability term interacts with the accumulated loss, which could connect to existing work on demonstration ordering.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is best read as an empirical prompt-selection method with a theoretical wrapper that doesn't quite close. The wrapper is the problem. Theorem 1 hinges on identifying W0—a rank-one matrix built from the test representation htest—with the pretraining optimum Ŵ. The influence-function expansion in Eq. (8) and the Taylor argument in Eq. (9) are only valid around a stationary point of the pretraining objective. Nothing in the construction of Eq. (4) guarantees that W0 minimizes any loss over Dpre; generically, it does not. So the sign of the loss change in Theorem 1 is not established, and the LMS3 score inherits that gap. The stress-test note is right: this is a missing premise inside the proof, not just an unverified empirical assumption. There is also a secondary mismatch between Eq. (16) and Eq. (7) involving an extra ||htest|| factor, but that is fixable; the Ŵ = W0 step is not.

What the paper does well is the empirical package. LMS3 balances LLM-oriented semantic similarity and an inference-stability term, adds a rejection mechanism that abstains from demonstrations when the best candidate looks too dissimilar, and the experiments on Llama2-13B and Llama3-8B across MAWPS, GSM8K, and MATH show consistent gains over the baselines in one-shot and few-shot settings. The rejection mechanism is a genuinely useful idea, and the case study in Figure 6 is illustrative. I also appreciate that the authors report statistical significance and show the effect of the rejection threshold λ. That is careful empirical work.

The soft spots beyond the theorem: the scoring function in Eq. (24) ignores the unknown constants from the bound, and the rejection threshold λ is tuned on the validation set; so the 'theoretical advantage' claim overstates what the theory actually delivers. The abstract says 'consistent improvements on all datasets,' but in Table 3 LMS3 on GPT-4 MATH (0.752) is below zero-shot (0.758), and the paper attributes that to all methods being negative. That is fine as an observation, but it does contradict the unqualified abstract sentence. The O(M+N) complexity claim also assumes the per-input representations can be precomputed, which is fair but worth stating more carefully.

Who gets value from this? Someone building a demonstration-selection heuristic for math reasoning would find LMS3 useful and the comparison against Similar-ICL and influence-based methods informative. A theory reader will find the proof gap quickly and may stop there. I would send it to a serious referee because the empirical method deserves scrutiny and the theoretical flaw is precisely what a good referee should catch; the paper should not be desk-rejected, but it needs major revision before it can be accepted. My own verdict on the central claim is skeptical: the bound is not proved as stated.

Referee Report

3 major / 4 minor

Summary. The paper studies when in-context demonstrations improve LLM performance on mathematical reasoning. It models the one-shot ICL forward pass as linear attention, interprets the demonstration as one gradient step of a linear function, and proves (Theorems 1 and 2) a sufficient condition under which adding one or k demonstrations reduces the prediction loss. The condition involves a LLM-oriented semantic similarity and a quantity called inference stability. Based on this condition, the paper proposes a demonstration selection method, LMS3, with a rejection mechanism, and reports experiments on MAWPS, GSM8K, and MATH with Llama2-13B, Llama3-8B, ChatGPT, and GPT-4.

Significance. The question addressed is important: demonstrations can hurt, and a quantitative account of when they help would be a valuable theoretical contribution. The empirical study is broad, comparing ten baselines across three datasets and several LLMs, and LMS3 is competitive in the reported tables. However, the central theoretical claim is not established: the proof of Theorem 1 requires identifying the test-dependent initialization W0 with the pretrained optimum \hat W, which is not justified and is generally false. Because the proposed scoring function is derived from Eq. (7), this flaw undermines the claimed theoretical grounding of LMS3. The paper also contains an internal inconsistency between Eq. (16) and Eq. (7). The empirical method may still be useful as an unsupervised heuristic, but the manuscript's main claimed contribution is a proof, and that proof is not sound.

major comments (3)

[Section 3, Eqs. (4), (6)-(9)] The proof of Theorem 1 relies on the statement, made just after Eq. (6), that the pretrained parameters \hat W are exactly the initialized parameters W0 defined in Eq. (4). This is not a harmless simplification. W0 = (WV/\sqrt{d}) htest (WK htest)^T is a rank-one matrix constructed from the test representation htest, while \hat W is defined as the minimizer of the pretraining loss over Dpre and is a fixed parameter of the model. There is no reason for these matrices to be equal, and generically they are not; equality would require Dpre and the loss to be engineered so that a test-dependent rank-one matrix minimizes the pretraining objective for every test point. The influence-function expansion in Eqs. (8)-(9) is valid only when the expansion point is a stationary point of the objective being upweighted. Since W0 is not shown to be such a stationary point, Eq. (9) is not justified, and the inequality in Theorem 1 is not proved even for the linear surrogate. Because the LMS3 score is justified by Eq. (7), this defect is load-bearing for the paper's central claim.
[Section 3, Eqs. (12)-(16) versus Eq. (7)] There is a mathematical inconsistency in the proof. Substituting Eq. (13) into Eq. (12) gives the term mu * ||WV/\sqrt{d} htest|| * ||WK htest|| * ||htest - z0|| * ||htest||, which equals mu * C1 * ||htest - z0|| because C1 already contains the factor ||htest||. Eq. (16), however, writes this term as mu * C1 * ||htest - z0|| * ||htest||, introducing an extra ||htest|| factor. The sufficient condition in Eq. (7) does not contain this extra factor, so Eq. (16) and Eq. (7) are inconsistent. This needs to be corrected before the derivation of the score can be regarded as internally coherent.
[Section 4, Eqs. (21)-(24)] The passage from the theorem to the LMS3 score is not a derivation but an approximation that drops several quantities appearing in the theorem. The sufficient condition in Eq. (7) involves the eigenvalue ratio lambda_dd'/lambda_1, the gradient norm ||\nabla_W L(htest, \hat W)||, and the additive term mu*C1*||htest - z0||. Minimizing Score(X) = Sim(X) * Stab(X) is not equivalent to satisfying Eq. (7), because the additive mu*C1 term is discarded and the unknown eigenvalue ratio and gradient norm are ignored. The paper should state clearly that Eq. (24) is a heuristic inspired by the theorem rather than a consequence of it.

minor comments (4)

[Section 5.4, Figure 5 and Table 2] Please state explicitly how the rejection threshold lambda is chosen for the results in Tables 2 and 3. If lambda is selected by inspecting test-set curves such as those in Figure 5, the reported gains may be optimistic; if a validation set is used, that should be stated and the validation split described.
[Section 4, Eq. (22)] Stab(X) is described as the length of the gradient of the loss on X, but Eq. (22) defines it as the norm of \nabla_F L, the gradient with respect to the function output, not the parameter gradient. Please make this distinction explicit.
[Section 3, notation around Eq. (3)] The redefinition of WK as WK^T WQ is introduced informally. The notation in Eq. (21) and in the sentence 'recall z0 = WKh and WK is indeed WK^T * WQ' is easy to misread; define a new symbol or otherwise make the redefinition unambiguous.
[Throughout] The paper claims in the checklist that all code is included, but no code appendix is visible in the manuscript. Please ensure the code and hyperparameter details are either included or removed from the checklist claims.

Circularity Check

1 steps flagged · score 6.0 of 10

The theoretical bound is partially circular: the pretrained optimum W_hat is identified with a test-dependent rank-one matrix W0, so the theorem's condition and the Sim/Stab scores are properties of that definitional identification rather than derived facts about a fixed pretrained model.

self definitional [Section 3 (Theoretical Analysis), Eq. (4) through Eq. (6), sentence before Theorem 1]
"Based on this idea, Eq. (3) can be interpreted as: 1) We have a linear function F(z) with initialized parameters W0 = WV√d htest · (WK htest)T. ... It is worth noting that the pretrained parameters ˆW are actually the initialized parameters in our above setting, i.e., ˆW = W0."

W_hat is defined in Eq. (6) as the argmin over pretraining data Dpre, independent of the test sample, while W0 is defined in Eq. (4) as the rank-one outer product WV htest (WK htest)^T / √d, which depends on the test representation htest. The proof of Theorem 1 applies the influence-function expansion (Eq. 8) and first-order Taylor expansion (Eq. 9); both require W_hat to be a stationary point of the pretraining objective. Asserting W_hat = W0 makes that stationarity true only by identifying the pretrained optimum with a matrix constructed from the very test input whose loss change is being predicted. The sufficient condition (7) and the resulting Sim(X)/Stab(X) scores (Eqs.

full rationale

The empirical core of the paper is self-contained and independently informative: LMS3 is tested against ten baselines on MAWPS, GSM8K, and MATH with Llama2-13B, Llama3-8B, ChatGPT, and GPT-4, and the reported improvements are real experiments rather than consequences of the proof. There is no load-bearing self-citation or imported uniqueness theorem; citations to Dai et al. and Koh & Liang are standard external tools. The circular/definitional step is confined to the theoretical derivation in Section 3. There, W0 is built from the test representation htest and is then asserted to equal the pretraining optimum W_hat, even though W_hat was defined independently as a minimizer over Dpre. The influence-function and Taylor expansions used in the proof of Theorem 1 are valid only at a stationary point of the pretraining loss, so the equality makes the theorem true by construction for a test-dependent matrix rather than by a first-principles argument about a fixed pretrained model. The later step from Eq. (7) to the multiplicative score Sim(X) * Stab(X) is a further heuristic simplification, and the rejection threshold lambda is scanned empirically rather than derived; these weaken the claim but are not themselves circular. Overall, the theoretical claim is partially circular because its central bound depends on a definitional identification of the pretrained parameters with the test-dependent initialization, while the empirical method retains independent support.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central derivation relies on a series of approximations and unverified assumptions: linear attention, the gradient-descent analogy, the influence function expansion, and Lipschitz/Hessian conditions. These are standard tools in the ICL theory literature but are not established for the tested models. The only fitted parameter is the rejection threshold λ.

free parameters (1)

Rejection threshold λ = selected from {1%, 5%, 10%, 20%, 40%, 60%, 80%, 100%}, optimal within 10% for both backbones
The rejection mechanism's threshold is chosen by validation performance (Section 5.4, Figure 5), making it a data-dependent hyperparameter.

assumptions (4)

domain assumption The Transformer attention can be approximated by linear attention (softmax removed)
Eq (3) removes softmax to obtain a linear form; this approximation is unverified for the models tested.
ad hoc to paper The ICL forward pass is equivalent to one gradient step on a linear function with a training sample derived from the demonstration
Section 3, 'Analogy to Linear Optimization', interprets Eq (3) as W0 plus a gradient term; this analogy underlies the influence function analysis.
domain assumption The pretrained model is at a loss minimum with positive definite Hessian, and 1/|Dpre| is small enough for a Taylor expansion
Needed for the influence function formula (Eq 8-9); not verified for real LLMs.
domain assumption The loss gradient ∇F L is Lipschitz continuous with constant µ
Assumed in Theorems 1 and 2; µ is unknown and not estimated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of What Makes In-context Learning Effective for Mathematical Reasoning: A Theoretical Analysis." pith.science (2026). https://pith.science/paper/KOJWUW4J

@misc{pith2026241212157,
  author       = {Pith},
  title        = {Pith review of: What Makes In-context Learning Effective for Mathematical Reasoning: A Theoretical Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KOJWUW4J}},
  note         = {Machine review of arXiv:2412.12157}
}

read the original abstract

Owing to the capability of in-context learning, large language models (LLMs) have shown impressive performance across diverse mathematical reasoning benchmarks. However, we find that few-shot demonstrations can sometimes bring negative performance and their effectiveness on LLMs' reasoning abilities remains unreliable. To this end, in this paper, we aim to theoretically analyze the impact of in-context demonstrations on LLMs' reasoning performance. We prove that the reasoning efficacy (measured by empirical prediction loss) can be bounded by a LLM-oriented semantic similarity and an inference stability of demonstrations, which is general for both one-shot and few-shot scenarios. Based on this finding, we propose a straightforward, generalizable, and low-complexity demonstration selection method named LMS3. It can adaptively facilitate to select the most pertinent samples for different LLMs and includes a novel demonstration rejection mechanism to automatically filter out samples that are unsuitable for few-shot learning. Through experiments on three representative benchmarks, two LLM backbones, and multiple few-shot settings, we verify that our LMS3 has superiority and achieves consistent improvements on all datasets, which existing methods have been unable to accomplish.

Figures

Figures reproduced from arXiv: 2412.12157 by the authors.

**Figure 2.** Illustration of our proposed LMS3 method. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Firstly, it indicates that our method remains superior across different settings, which is consistent with our +$' $( & ' $( $)&' $( )&, +$' $( & ' $( $)&' $( )&, +$' $( & ' $( $)&' $( )&, # $" [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 6.** Figure 6: Case Study (Case 1). timal results, demonstrating the excellent generalization and representativeness of our selected demonstrations. Besides, we observe that all methods negatively impact the reasoning performance on MATH dataset when applied to GPT-4. We attribute t…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

BoostStep: Boosting mathematical capability of Large Language Models via improved single-step reasoning
cs.CL 2025-01 conditional novelty 7.0 of 10

Step-aligned in-context learning with a first-try retrieval strategy improves LLM mathematical reasoning over problem-level few-shot prompting on multiple benchmarks.

Reference graph

Works this paper leans on

53 extracted references · 27 canonical work pages · cited by 1 Pith paper

[1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023
[4]

An, S.; Lin, Z.; Fu, Q.; Chen, B.; Zheng, N.; Lou, J.-G.; and Zhang, D. 2023. How Do In-Context Examples Affect Compositional Generalization? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 11027--11052

work page 2023
[5]

Asai, A.; Wu, Z.; Wang, Y.; Sil, A.; and Hajishirzi, H. 2024. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. In The Twelfth International Conference on Learning Representations

work page 2024
[6]

Besta, M.; Blach, N.; Kubicek, A.; et al. 2024. Graph of thoughts: Solving elaborate problems with large language models. In AAAI, volume 38, 17682--17690

work page 2024
[7]

D.; et al

Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877--1901

work page 2020
[8]

Chang, T.-Y.; and Jia, R. 2023. Data Curation Alone Can Stabilize In-context Learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 8123--8144

work page 2023

Show all 53 references

[9]

Chen, J.; Xiao, S.; Zhang, P.; et al. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216

2024 arXiv
[10]

Chen, W.; Ma, X.; Wang, X.; and Cohen, W. W. 2023. Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks. Transactions on Machine Learning Research

2023
[11]

Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

2021 arXiv
[12]

Dai, D.; Sun, Y.; Dong, L.; Hao, Y.; Ma, S.; Sui, Z.; and Wei, F. 2023. Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers. In Findings of the Association for Computational Linguistics: ACL 2023, 4005--4019

2023
[13]

Dong, Q.; Li, L.; Dai, D.; Zheng, C.; Wu, Z.; Chang, B.; Sun, X.; Xu, J.; and Sui, Z. 2022. A survey on in-context learning. arXiv preprint arXiv:2301.00234

2022 arXiv
[14]

A.; Feldman, J.; et al

Feigenbaum, E. A.; Feldman, J.; et al. 1963. Computers and thought. New York McGraw-Hill

1963
[15]

Fletcher, C. R. 1985. Understanding and solving arithmetic word problems: A computer simulation. Behavior Research Methods, Instruments, & Computers, 17(5): 565--571

1985
[16]

Fu, Y.; Peng, H.; Sabharwal, A.; Clark, P.; and Khot, T. 2022. Complexity-based prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations

2022
[17]

Gao, L.; Madaan, A.; Zhou, S.; Alon, U.; Liu, P.; Yang, Y.; Callan, J.; and Neubig, G. 2023. Pal: Program-aided language models. In International Conference on Machine Learning, 10764--10799. PMLR

2023
[18]

Gao, X.; and Das, K. 2024. Customizing Language Model Responses with Contrastive In-Context Learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 18039--18046

2024
[19]

A.; and Zettlemoyer, L

Gonen, H.; Iyer, S.; Blevins, T.; Smith, N. A.; and Zettlemoyer, L. 2023. Demystifying Prompts in Language Models via Perplexity Estimation. In Findings of the Association for Computational Linguistics: EMNLP 2023, 10136--10148

2023
[20]

Han, C.; Wang, Z.; Zhao, H.; and Ji, H. 2023. Explaining emergent in-context learning as kernel regression. arXiv preprint arXiv:2305.12766

2023 arXiv
[21]

Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021. Measuring Mathematical Problem Solving With the MATH Dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)

2021
[22]

W.; and Liang, P

Koh, P. W.; and Liang, P. 2017. Understanding black-box predictions via influence functions. In International conference on machine learning, 1885--1894. PMLR

2017
[23]

S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y

Kojima, T.; Gu, S. S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199--22213

2022
[24]

Koncel-Kedziorski, R.; Roy, S.; Amini, A.; et al. 2016. MAWPS: A math word problem repository. In NAACL-HLT, 1152--1157

2016
[25]

Lewkowycz, A.; Andreassen, A.; Dohan, D.; et al. 2022. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35: 3843--3857

2022
[26]

Lightman, H.; Kosaraju, V.; Burda, Y.; et al. 2023. Let's Verify Step by Step. In The Twelfth International Conference on Learning Representations

2023
[27]

Ling, R. F. 1984. Residuals and influence in regression

1984
[28]

Liu, J.; Huang, Z.; Zhai, C.; and Liu, Q. 2023. Learning by applying: A general framework for mathematical reasoning via enhancing explicit knowledge learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 4497--4506

2023
[29]

B.; Carin, L.; and Chen, W

Liu, J.; Shen, D.; Zhang, Y.; Dolan, W. B.; Carin, L.; and Chen, W. 2022. What Makes Good In-Context Examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, 100--114

2022
[30]

Luo, H.; Sun, Q.; Xu, C.; Zhao, P.; Lou, J.; Tao, C.; Geng, X.; Lin, Q.; Chen, S.; and Zhang, D. 2023 a . Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583

2023 arXiv
[31]

Luo, M.; Xu, X.; Dai, Z.; Pasupat, P.; Kazemi, M.; Baral, C.; Imbrasaite, V.; and Zhao, V. Y. 2023 b . Dr. icl: Demonstration-retrieved in-context learning. arXiv preprint arXiv:2305.14128

2023 arXiv
[32]

Meta, A. 2024. Introducing meta llama 3: The most capable openly available llm to date. Meta AI

2024
[33]

Min, S.; Lyu, X.; Holtzman, A.; Artetxe, M.; Lewis, M.; Hajishirzi, H.; and Zettlemoyer, L. 2022. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 11048--11064

2022
[34]

M.; Tworek, J.; Yuan, Q.; Tezak, N.; Kim, J

Neelakantan, A.; Xu, T.; Puri, R.; Radford, A.; Han, J. M.; Tworek, J.; Yuan, Q.; Tezak, N.; Kim, J. W.; Hallacy, C.; et al. 2022. Text and code embeddings by contrastive pre-training. arXiv preprint arXiv:2201.10005

2022 arXiv
[35]

Nguyen, T.; and Wong, E. 2023. In-context example selection with influences. arXiv preprint arXiv:2302.11042

2023 arXiv
[36]

Pan, J.; Gao, T.; Chen, H.; and Chen, D. 2023. What In-Context Learning" Learns" In-Context: Disentangling Task Recognition and Task Learning. In The 61st Annual Meeting Of The Association For Computational Linguistics

2023
[37]

Peng, K.; Ding, L.; Yuan, Y.; Liu, X.; Zhang, M.; Ouyang, Y.; and Tao, D. 2024. Revisiting demonstration selection strategies in in-context learning. arXiv preprint arXiv:2401.12087

2024 arXiv
[38]

Qin, C.; Zhang, A.; Dagar, A.; and Ye, W. 2023. In-context learning with iterative demonstration selection. arXiv preprint arXiv:2310.09881

2023 arXiv
[39]

Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 1--67

2020
[40]

Robertson, S.; Zaragoza, H.; et al. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval , 3(4): 333--389

2009
[41]

Sorensen, T.; Robinson, J.; Rytting, C.; Shaw, A.; Rogers, K.; Delorey, A.; Khalil, M.; Fulda, N.; and Wingate, D. 2022. An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels. In Proceedings of the 60th Annual Meeting of the Association for Comput...

2022
[42]

Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

2023 arXiv
[43]

H.; Wu, Y.; Le, Q

Trinh, T. H.; Wu, Y.; Le, Q. V.; He, H.; and Luong, T. 2024. Solving olympiad geometry without human demonstrations. Nature, 625(7995): 476--482

2024
[44]

Van, M.-H.; Wu, X.; et al. 2024. In-Context Learning Demonstration Selection via Influence Analysis. arXiv preprint arXiv:2402.11750

2024 arXiv
[45]

V.; Zhou, D.; et al

Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V.; Zhou, D.; et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 24824--24837

2022
[46]

Wu, Z.; Wang, Y.; Ye, J.; and Kong, L. 2023. Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Ordering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape...

2023
[47]

Yan, J.; Xu, J.; Song, C.; Wu, C.; Li, Y.; and Zhang, Y. 2024. Understanding In-Context Learning from Repetitions. In The Twelfth International Conference on Learning Representations

2024
[48]

Yao, S.; Yu, D.; Zhao, J.; et al. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36

2024
[49]

Ye, J.; Wu, Z.; Feng, J.; Yu, T.; and Kong, L. 2023. Compositional exemplars for in-context learning. In International Conference on Machine Learning, 39818--39833. PMLR

2023
[50]

Yu, L.; Jiang, W.; Shi, H.; Jincheng, Y.; Liu, Z.; Zhang, Y.; Kwok, J.; Li, Z.; Weller, A.; and Liu, W. 2024. MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models. In The Twelfth International Conference on Learning Representations

2024
[51]

Yue, X.; Qu, X.; Zhang, G.; Fu, Y.; Huang, W.; Sun, H.; Su, Y.; and Chen, W. 2024. MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning. In The Twelfth International Conference on Learning Representations

2024
[52]

Zhang, D.; Wang, L.; et al. 2020. The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(9): 2287--2305

2020
[53]

Zhang, P.; Xiao, S.; Liu, Z.; Dou, Z.; and Nie, J.-Y. 2023. Retrieve anything to augment large language models. arXiv preprint arXiv:2310.07554

2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023

[4] [4]

An, S.; Lin, Z.; Fu, Q.; Chen, B.; Zheng, N.; Lou, J.-G.; and Zhang, D. 2023. How Do In-Context Examples Affect Compositional Generalization? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 11027--11052

work page 2023

[5] [5]

Asai, A.; Wu, Z.; Wang, Y.; Sil, A.; and Hajishirzi, H. 2024. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. In The Twelfth International Conference on Learning Representations

work page 2024

[6] [6]

Besta, M.; Blach, N.; Kubicek, A.; et al. 2024. Graph of thoughts: Solving elaborate problems with large language models. In AAAI, volume 38, 17682--17690

work page 2024

[7] [7]

D.; et al

Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877--1901

work page 2020

[8] [8]

Chang, T.-Y.; and Jia, R. 2023. Data Curation Alone Can Stabilize In-context Learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 8123--8144

work page 2023