Pith. sign in

REVIEW 3 major objections 6 minor 47 references

Selecting Demonstrations for Many-Shot In-Context Learning via Gradient Matching

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Demonstration sets that match the full training set's gradient beat random selection in many-shot ICL.

desk verdict Solid empirical paper on many-shot ICL demonstration selection via gradient matching; the central claim holds on average, but missing variance for the stochastic selection run tempers 'consistently outperforms'. read the letter →

arxiv 2506.04579 v1 pith:WNW5KH3B submitted 2025-06-05 cs.CL

Jianfei Zhang , Bei Li , Jun Bai , Rumei Li , Yanmeng Wang , Chenghua Lin , Wenge Rong This is my paper
classification cs.CL
keywords many-shotin-contextlearningdemonstrationselectiongradientmatchinglatentconceptcurriculumtask-leveldatasetcondensationlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the right way to choose demonstrations for many-shot in-context learning is to treat the problem as fine-tuning data selection: pick the n examples whose average per-example gradient, measured on a learned latent task embedding in a small language model, best matches the average gradient of the whole training set. On 9 NLP datasets and from 4 to 128 shots, demonstrations chosen this way consistently beat random selection on larger models of the same family and on several closed-source models, with roughly 4% average gain on the largest open models. The reason to care is that many-shot ICL is currently stuck with random selection, and a better fixed demonstration set improves performance with no extra inference cost because the same prefix can be cached across queries.

What carries the argument

The central object is the Curriculum Latent Gradient (CLG): for each training example $(x,y)$, the concatenation of gradients $\nabla_{z_\theta} \mathcal{L}(z_\theta; x, y)$ of the negative log-likelihood loss with respect to the trainable latent task-context embedding $z_\theta \in \mathbb{R}^{k \times h}$, evaluated at the initial embedding and at the end of each of $E$ training epochs. These per-example trajectory gradients are averaged over the full training set $D_N$ and over a candidate subset $D_n$, and selection minimizes $\lVert \mathbb{E}_{D_N}[G_{0:E}] - \mathbb{E}_{D_n}[G_{0:E}] \rVert_2$ via greedy forward addition followed by up to 32 swap-based local optimization steps. This object carries the argument because it turns 'which demonstrations teach the task' into a vector-matching problem over the learning trajectory, rather than a static similarity or likelihood score.

What would settle it

Compute the Spearman correlation between fine-tuning loss and ICL accuracy per dataset across the paper's 128-shot runs; any dataset with near-zero or negative correlation that still shows a CLG gain would contradict the claimed mechanism. A sharper test: pick a dataset where the small proxy model cannot learn from the full training set (validation loss does not decrease); if CLG-selected demonstrations nonetheless beat random on a large model, the gradient signal is not what drives the improvement.

Watch

Extended reading notes

Core claim

The central discovery is that a fixed demonstration set $D_n$ selected by minimizing the L2 distance between the average curriculum latent gradient of the selected set and that of the full training set, $\lVert \mathbb{E}_{(x,y)\in D_N} [G_{0:E}(x,y)] - \mathbb{E}_{(x,y)\in D_n} [G_{0:E}(x,y)] \rVert_2$, reproduces at inference time more of the learning effect that fine-tuning on the whole set would produce. The authors realize this by training continuous latent task embeddings on a small model, saving the embeddings at the end of each epoch, computing for every training example the gradient of its loss with respect to those embeddings at each saved checkpoint, and then greedily assembling the n-shot set whose average gradient is closest to the full-set average. The selected sets outperform random selection from 4-shot to 128-shot on larger open models and on five closed-source models, and reversing the objective to maximize gradient mismatch makes performance drop below random. The paper also shows that the selected sets naturally match the test label distribution better than random sets, suggesting that learning-dynamics matching subsumes coverage.

Load-bearing premise

The method's usefulness rests on the assumption that the examples which teach a small model during fine-tuning are also the examples that teach a large model during in-context learning; if that analogy breaks, the selected demonstrations have no reason to be better than random.

Editorial extensions

If this is right

  • Fixed demonstration sets chosen this way improve many-shot ICL on larger models of the same family and on closed-source models, so the gain costs nothing at inference time beyond one-time selection.
  • The advantage holds from 4-shot to 128-shot and remains positive at 1024-shot, with a smaller margin once the demonstration set is large enough to cover the task.
  • Combining the selected task-level set with a few instance-level retrieved demonstrations yields further gains on most datasets, so the two selection paradigms are complementary.
  • Demonstration sets chosen by gradient matching also align much better with the test label distribution than random selection, indicating that coverage emerges from the learning-dynamics objective.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the fine-tuning/in-context-learning analogy is causal rather than correlational, CLG should extend to other inference-time adaptation surfaces, such as retrieval-augmented demonstration pools or multi-turn tool-use contexts, wherever gradients on a small proxy model can be computed.
  • The shrinking advantage at 1024 shots suggests a practical saturation point: selection matters most in the 16-128 shot regime, so users with very long contexts may not need CLG.
  • A direct stress test would select demonstrations with a model much smaller than the 3B-8B proxies used here, or from a different family, and measure how far the transfer holds; the paper's closed-source results already hint that within-family similarity is not required, but the boundary is unmapped.
  • The label-coverage result makes a testable prediction: on tasks with skewed label distributions, CLG should resist the collapse that similarity-based retrieval often exhibits, which could be checked by measuring per-class recall of the selected set.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Curriculum Latent Gradient (CLG), a task-level demonstration selection method for many-shot in-context learning. CLG first trains a small set of continuous latent concept tokens on a small language model, computes per-example gradients of the negative log-likelihood with respect to those tokens at initialization and after each of E epochs, and then selects n demonstrations whose average gradient is closest in L2 distance to the average gradient of the full training set (Eq. 7). Selection is performed by greedy construction followed by local search (Algorithm 1). The authors evaluate the selected sets on 9 datasets from 5 tasks, on open-source Llama3-70B and Qwen2.5 models from 3B to 72B, on five closed-source models, and at scales from 4-shot to 1024-shot. The central claim is that CLG consistently outperforms random demonstration selection, with average gains of about 4% on the two large open models and about 2% on the closed models.

Significance. If the results hold, this is a useful contribution: it is one of the first proposals specifically aimed at demonstration selection for many-shot ICL rather than few-shot retrieval, it produces a fixed demonstration set that is compatible with prefix caching, and it shows cross-family transfer to closed-source APIs. The paper's strengths include a clean and falsifiable formulation of the selection objective, an ablation that reverses the matching direction (Sec. 4.7), full per-dataset appendices (C and D), a scaling analysis to 1024 shots (Sec. 4.6), and a public code repository. However, the statistical support for the word 'consistently' is incomplete: every headline comparison is based on a single CLG run, whereas the random baseline is averaged over five seeds, and the CLG selection pipeline is itself stochastic. The FT-ICL analogy that motivates the method is also only weakly validated. These issues are fixable within the scope of the paper, so I am not recommending rejection.

major comments (3)
  1. [§4.4, Tables 3-4, Algorithm 1] The main claim that CLG 'consistently outperforms' random selection is not statistically supported because the paper reports a single selection run for CLG. The pipeline is stochastic: Sec. 3.2 randomly initializes the latent concept tokens and optimizes them with SGD over shuffled data, and Algorithm 1's greedy trajectory depends on that initialization. Tables 3-5 and 11-13 compare this single CLG result to a random baseline averaged over 5 seeds with standard deviations. The magnitude of the issue is visible in the paper's own numbers: in Table 4, CLG's mean gain over random on GLM-4-flash is +2.33%, while the random baseline has a standard deviation of ±2.24%; in Table 3, CLG is below random on CMSQA for Llama3-70B (84.52 vs. 84.73) and essentially tied on NL2Bash (29.59 vs. 29.72). The reported 4% and 2% averages could therefore depend on a favorable draw of the stochastic selection procedure. Please report at least the mean and standard deviation (or a confidence interval) over multiple CLG runs using the same seeds as the random baseline, and ideally a paired bootstrap test of the average gains. At minimum, the phrase 'consistently outperforms' should be qualified if only one selection outcome is available.
  2. [§4.9 / Fig. 6] The paper's motivating hypothesis is that data requirements for ICL and fine-tuning are analogous, but the support provided in Fig. 6 is only a qualitative correlation plot for Qwen models. It does not test the specific claim that minimizing the L2 gradient distance in Eq. 7 is what transfers across model families, and it does not include the Llama or closed-source model families used in the main experiments. The cross-family results in Tables 4 and 13 are the only direct evidence for transfer, and no mechanism is offered for why latent gradients from a 3B model should improve ICL on a 70B model of a different series. I do not consider this a reason to reject, because the empirical evidence is substantial, but the paper should either add a clear caveat that the FT-ICL analogy is a heuristic whose validation is the correlation plot, or include one additional experiment that varies the selection model family while fixing the target model.
  3. [§4.10, Table 6] The paper does not compare CLG against a label-balanced random baseline. Table 6 shows that CLG's selected sets have much lower KL divergence to the test label distribution than random selection on all four classification datasets. Since label balance is known to affect ICL accuracy, and since the random baseline draws from a distribution that naturally produces imbalanced label proportions, part of CLG's gains over random may be attributable to label diversification rather than to the gradient-matching mechanism itself. Please add a random baseline that is sampled without replacement under a label-balanced or test-matching constraint, or otherwise demonstrate that CLG's advantage persists after controlling for label distribution.
minor comments (6)
  1. [Tables 3, 11] The abbreviation 'HeSwag.' should be 'HellaSwag.' for consistency with the dataset name used elsewhere.
  2. [Fig. 6] The text says the figure 'shows a positive relationship' between FT and ICL performance, but no correlation coefficient is reported. Please report Pearson or Spearman values and, if possible, confidence intervals.
  3. [§4.9] The fine-tuning setup for the FT-ICL correlation uses 1000 epochs of prefix-tuning; please clarify whether this was early-stopped, and if so, what validation criterion was used, since 1000 epochs is unusual even for prefix-tuning on 128 demonstrations.
  4. [§3.2-3.3] The number of latent concept tokens k, the number of epochs E, the learning rate, and the local search iterations l are free parameters, but no sensitivity analysis is reported. A short analysis over k or E would help assess the robustness of the method.
  5. [§3.3 / Eq. 7] Eq. 7 is described as an NP-complete subset sum problem, but the objective is a cardinality-constrained minimization of the L2 norm of a difference of average vector-valued gradients; this is not literally subset sum. Please either state the reduction or cite a more appropriate reference for vector subset selection.
  6. [§4.5] The sentence that CLG achieves gains 'far more than the standard deviation' refers to the average over datasets in Fig. 3, but per-dataset standard deviations are not shown there; consider reporting per-dataset error bars or a table of per-dataset gains to make the claim verifiable.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the CLG selection objective is self-contained and evaluated on held-out validation sets; the only mild concern is a non-independent FT-ICL correlation check, which is not load-bearing.

full rationale

The paper's central derivation is not circular. Eq. 7 defines the selected demonstration set as the subset minimizing the L2 distance between the average curriculum latent gradient of the subset and that of the full training set, where gradients are computed on a small LM (Qwen2.5-3B or Llama3-8B). This objective is computed entirely from the small model and the task training set; no target-model ICL output, validation score, or headline number is used as a fitting signal. The claim that these demonstrations transfer to larger and closed-source LLMs is an empirical hypothesis test, not an equivalence by construction. Hyperparameters (E=10, learning rate, batch size, number of latent tokens) are fixed before evaluation, so no fitted parameter is renamed as a prediction. The paper contains no load-bearing self-citation: its cited latent-concept learning and dataset-condensation foundations are external prior work, and no 'uniqueness theorem' is imported. The only mild concern is Section 4.9, where the FT-ICL correlation that 'validate[s] our hypothesis' is computed on the same demonstration sets CLG was designed to select; this makes the correlation a consistency check rather than an independent test of the FT-ICL analogy. That weakness does not reduce the main derivation, because the ICL gains in Tables 3-5 and 11-13 are measured on held-out validation sets against an averaged random baseline. Separately, the lack of a CLG variance estimate is a statistical robustness concern, but it is not a circularity concern.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method introduces a new selection criterion (curriculum latent gradients) but relies on several unproven domain assumptions, especially the FT-ICL analogy and the transferability of small-model gradients to larger models. Hyperparameters (E, l, learning rate) are chosen without sensitivity analysis.

free parameters (5)
  • number of latent concept tokens k
    Dimension of the latent task concept z_theta; not reported in the paper, chosen by hand.
  • number of training epochs E = 10
    Chosen for latent concept learning; no sensitivity analysis.
  • learning rate = 1e-3
    For latent concept learning; chosen without sensitivity analysis.
  • batch size = 64
    For latent concept learning.
  • local optimization iterations l = 32
    Maximum swaps in Algorithm 1; no sensitivity analysis.
assumptions (5)
  • domain assumption Data requirements for in-context learning and fine-tuning are analogous
    Sec 1 and Sec 4.9; the entire method is motivated by this, but only a correlation plot is provided.
  • domain assumption Latent concept embeddings z_theta trained on a small LM are a faithful proxy for in-context demonstrations
    Borrowed from Wang et al. (2023); used in Eq. 2-3.
  • domain assumption Minimizing L2 distance between average latent gradients approximates matching learning dynamics
    Core of Eq. 7; assumed, not proven.
  • domain assumption Gradients computed on a small model (Qwen2.5-3B, Llama3-8B) transfer to larger models and other families
    The method selects on small models and evaluates on larger/closed-source models; only empirical support.
  • ad hoc to paper The greedy + local search heuristic finds a near-optimal solution to the subset sum problem
    Algorithm 1; the optimal solution is NP-complete, so the heuristic is assumed adequate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Selecting Demonstrations for Many-Shot In-Context Learning via Gradient Matching." pith.science (2026). https://pith.science/paper/WNW5KH3B

@misc{pith2026250604579,
  author       = {Pith},
  title        = {Pith review of: Selecting Demonstrations for Many-Shot In-Context Learning via Gradient Matching},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WNW5KH3B}},
  note         = {Machine review of arXiv:2506.04579}
}
read the original abstract

In-Context Learning (ICL) empowers Large Language Models (LLMs) for rapid task adaptation without Fine-Tuning (FT), but its reliance on demonstration selection remains a critical challenge. While many-shot ICL shows promising performance through scaled demonstrations, the selection method for many-shot demonstrations remains limited to random selection in existing work. Since the conventional instance-level retrieval is not suitable for many-shot scenarios, we hypothesize that the data requirements for in-context learning and fine-tuning are analogous. To this end, we introduce a novel gradient matching approach that selects demonstrations by aligning fine-tuning gradients between the entire training set of the target task and the selected examples, so as to approach the learning effect on the entire training set within the selected examples. Through gradient matching on relatively small models, e.g., Qwen2.5-3B or Llama3-8B, our method consistently outperforms random selection on larger LLMs from 4-shot to 128-shot scenarios across 9 diverse datasets. For instance, it surpasses random selection by 4% on Qwen2.5-72B and Llama3-70B, and by around 2% on 5 closed-source LLMs. This work unlocks more reliable and effective many-shot ICL, paving the way for its broader application.

Figures

Figures reproduced from arXiv: 2506.04579 by the authors.

Figure 1
Figure 1. Instance-level retrieved demonstrations are not [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overview of our proposed method for many-shot demonstration selection. We perform latent concept [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The scaling performances with respect to (w.r.t.) the number of shots and the model size. For random [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: ICL with hybrid demonstrations: task-level [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 4
Figure 4. Figure 4: Ablation study on the impact of gradient mis [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 7
Figure 7. Figure 7: The scaling trends of random selection with respect to the number of shots on all models. We illustrate the [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: The performance of different models on MNLI. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 29 canonical work pages

  1. [1]

    Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D

    Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co - Reyes, Eric Chu, Feryal M. P. Behbahani, Aleksandra Faust, and Hugo Larochelle. 2024. https://doi.org/10.48550/ARXIV.2404.11018 Many-shot in-context learning . CoRR, abs/2404.11018

    arXiv doi:10.48550/arxiv.2404.11018 2024

  2. [2]

    Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman, Percy Liang, Christopher H. Lin, Ilya Lintsbakh, Andy McGovern, Aleksandr N...

    2020

  3. [3]

    Cem Anil, Esin Durmus, Nina Rimsky, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Meg Tong, Jesse Mu, Daniel J Ford, et al. 2024. Many-shot jailbreaking. In Proceedings of the 38th Annual Conference on Neural Information Processing Systems

    work page 2024

  4. [4]

    Gormley, and Graham Neubig

    Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R. Gormley, and Graham Neubig. 2024. https://doi.org/10.48550/ARXIV.2405.00200 In-context learning with long-context models: An in-depth exploration . CoRR, abs/2405.00200

    arXiv doi:10.48550/arxiv.2405.00200 2024

  5. [5]

    L \' e on Bottou. 2010. Large-scale machine learning with stochastic gradient descent. In Proceedings of the 19th International Conference on Computational Statistics, pages 177--186

    work page 2010

  6. [6]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert - Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litw...

    work page 2020

  7. [7]

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. https://doi.org/10.48550/ARXIV.2402.03216 BGE m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation . CoRR, abs/2402.03216

    arXiv doi:10.48550/arxiv.2402.03216 2024

  8. [8]

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. https://doi.org/10.48550/ARXIV.2306.15595 Extending context window of large language models via positional interpolation . CoRR, abs/2306.15595

    arXiv doi:10.48550/arxiv.2306.15595 2023

Show all 47 references
  1. [9]

    Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT : Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu...

    2019

  2. [10]

    Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. LongRoPE : Extending LLM context window beyond 2 million tokens. In Proceedings of the 41st International Conference on Machine Learning, pages 11091--11104

    2024

  3. [11]

    Andrew Drozdov, Honglei Zhuang, Zhuyun Dai, Zhen Qin, Razieh Rahimi, Xuanhui Wang, Dana Alon, Mohit Iyyer, Andrew McCallum, Donald Metzler, and Kai Hui. 2023. https://doi.org/10.48550/ARXIV.2310.14408 Parade: Passage ranking using demonstrations with large language models . Co...

    2023 doi arXiv

  4. [12]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al - Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Z...

    2024 doi arXiv

  5. [13]

    Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2023. Complexity-based prompting for multi-step reasoning. In Proceedings of the 11th International Conference on Learning Representations

    2023

  6. [14]

    Smith, and Luke Zettlemoyer

    Hila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith, and Luke Zettlemoyer. 2023. Demystifying prompts in language models via perplexity estimation. In Findings of the Association for Computational Linguistics: EMNLP , pages 10136--10148

    2023

  7. [15]

    Matan Hasson and Jonathan Berant. 2021. Question decomposition with dependency graphs. In Proceedings of the 3rd Conference on Automated Knowledge Base Construction

    2021

  8. [16]

    Chen, and Andrew Y

    Yixing Jiang, Jeremy Irvin, Ji Hun Wang, Muhammad Ahmed Chaudhry, Jonathan H. Chen, and Andrew Y. Ng. 2024. https://doi.org/10.48550/ARXIV.2405.09798 Many-shot in-context learning in multimodal foundation models . CoRR, abs/2405.09798

    2024 doi arXiv

  9. [17]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Princip...

    2023

  10. [18]

    J. C. Lagarias and Andrew M. Odlyzko. 1985. Solving low-density subset sum problems. Journal of The ACM , 32(1):229--246

    1985

  11. [19]

    Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. 2021. MTOP: A comprehensive multilingual task-oriented semantic parsing benchmark. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisti...

    2021

  12. [20]

    Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4582--4597

    2021

  13. [21]

    Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. 2023 a . Unified demonstration retriever for in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pages 4644--4668

    2023

  14. [22]

    Xiaonan Li and Xipeng Qiu. 2023. https://doi.org/10.48550/ARXIV.2302.13539 Finding supporting examples for in-context learning . CoRR, abs/2302.13539

    2023 doi arXiv

  15. [23]

    Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. 2023 b . Compressing context to enhance inference efficiency of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6342--6353

    2023

  16. [24]

    Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D. Ernst. 2018. NL2Bash : A corpus and semantic parser for natural language interface to the linux operating system. In Proceedings of the 11th International Conference on Language Resources and Evaluation

    2018

  17. [25]

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3 ? In Proceedings of of the 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100--114

    2022

  18. [26]

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pa...

    2022

  19. [27]

    Man Luo, Xin Xu, Zhuyun Dai, Panupong Pasupat, Seyed Mehran Kazemi, Chitta Baral, Vaiva Imbrasaite, and Vincent Y. Zhao. 2023. https://doi.org/10.48550/ARXIV.2305.14128 Dr.ICL : Demonstration-retrieved in-context learning . CoRR, abs/2305.14128

    2023 doi arXiv

  20. [28]

    OpenAI. 2023. https://doi.org/10.48550/ARXIV.2303.08774 GPT-4 technical report . CoRR, abs/2303.08774

    2023 doi arXiv

  21. [29]

    Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models. In Proceedings of the 34th Annual Conference on Neural Information Processing Systems, pages 11054--11070

    2021

  22. [30]

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2023. Efficiently scaling transformer inference. In Proceedings of the 6th Conference on Machine Learning and Systems

    2023

  23. [31]

    Robertson and Hugo Zaragoza

    Stephen E. Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333--389

    2009

  24. [32]

    Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655--2671

    2022

  25. [33]

    Peter Shaw, Ming - Wei Chang, Panupong Pasupat, and Kristina Toutanova. 2021. Compositional generalization and natural language variation: Can a semantic parsing approach handle both? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics an...

    2021

  26. [34]

    Manning, Andrew Y

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Langua...

    2013

  27. [35]

    Mingyang Song, Mao Zheng, and Xuan Luo. 2025. Can many-shot in-context learning help llms as evaluators? A preliminary empirical study. In Proceedings of the 31st International Conference on Computational Linguistics, pages 8232--8241

    2025

  28. [36]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human ...

    2019

  29. [37]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st Annual Conference on Neural Information Processing Systems, pages 5998--6008

    2017

  30. [38]

    Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/3255a7554605a88800f4e120b3a929e1-Abstract-Conference.html Large language models are latent variable models: Explaining and finding good demons...

    2023

  31. [39]

    Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn...

    2018

  32. [40]

    Tomer Wolfson, Mor Geva, Ankit Gupta, Yoav Goldberg, Matt Gardner, Daniel Deutch, and Jonathan Berant. 2020. https://doi.org/10.1162/TACL\_A\_00309 Break it down: A question understanding benchmark . Transactions of the Association for Computational Linguistics, 8:183--198

    2020 doi

  33. [41]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 doi arXiv

  34. [42]

    Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. Compositional exemplars for in-context learning. In Proceedings of the 40th International Conference on Machine Learning, pages 39818--39833

    2023

  35. [43]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag : Can a machine really finish your sentence? In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 4791--4800

    2019

  36. [44]

    Yiming Zhang, Shi Feng, and Chenhao Tan. 2022. Active example selection for in-context learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9134--9148

    2022

  37. [45]

    Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. 2021. Dataset condensation with gradient matching. In Proceedings of the 9th International Conference on Learning Representations

    2021

  38. [46]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  39. [47]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.