Pith. sign in

REVIEW 4 major objections 5 minor 32 references

The Power of Adaptation: Boosting In-Context Learning through Adaptive Prompting

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Pith's one-line read The paper claims that selecting in-context learning exemplars adaptively — one at a time, based on the model's uncertainty given the examples already chosen — outperforms selecting all exemplars at once on reasoning tasks.

desk verdict An incremental but genuinely new exemplar-selection mechanism is undermined by an off-by-one in the algorithm and effect sizes within API noise. read the letter →

arxiv 2412.17891 v1 pith:XXJ2UYJL submitted 2024-12-23 cs.CL cs.AI

classification cs.CLcs.AI
keywords in-contextlearningchain-of-thoughtpromptingadaptiveexemplarselectionactiveuncertaintyestimationlargelanguagemodelsreasoningtasksannotationbudget
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the exemplars used for in-context learning should be selected sequentially, not in one batch. Its method, Adaptive-Prompt, repeatedly asks a language model to answer each remaining unlabeled question using the exemplars already chosen, scores the question by how much the model's responses disagree, and annotates the most uncertain one. Across arithmetic, commonsense, and symbolic reasoning datasets, this iterative procedure gets higher average accuracy than selecting the most uncertain questions once and annotating them together. The reason to care is practical: if the claim is right, a fixed budget of human annotation yields more reasoning accuracy when the selection itself is model-aware and sequential.

What carries the argument

The load-bearing object is the conditional uncertainty score $u(q \mid E)$, defined as the disagreement ratio $t/l$ (unique responses over total repeated queries) or the entropy of the answer distribution when the model is shown the current exemplar set $E$ together with question $q$. The selection rule $q_j = \arg\max_{q \in Q} u(q \mid E)$ turns this score into a greedy sequential algorithm: after each annotation, $E$ grows and every remaining candidate is rescored, which is exactly the step that distinguishes Adaptive-Prompt from one-shot Active-Prompt.

What would settle it

On a single dataset, record for each candidate question both its uncertainty score $u(q \mid E)$ and the actual change in held-out accuracy caused by annotating and adding it; if the two are not positively correlated, the selection rule's justification fails.

Watch

Extended reading notes

Core claim

In its own terms, the paper's central claim is that conditional uncertainty is a better guide for exemplar selection than unconditional uncertainty. Starting with an empty exemplar set, the algorithm evaluates every candidate question under the prompt formed by the current exemplars plus that question, computes a disagreement ratio or an entropy score over repeated responses, and picks the highest-uncertainty question for human annotation. The chosen question is added to the exemplar set and removed from the candidate pool, and the scores are recomputed, so each selection is conditioned on all previous ones. On GPT-3.5 Turbo the entropy variant averages 76.0 percent accuracy versus 75.3 for the one-shot Active-Prompt baseline; on GPT-4o mini the corresponding numbers are 86.9 and 86.7 percent, with the adaptive method ahead on most individual datasets.

Load-bearing premise

The method assumes that a question the model currently answers with high response divergence is the question whose annotation most improves test accuracy, and the paper does not measure that link directly.

Editorial extensions

If this is right

  • For a fixed annotation budget $k$, adaptively choosing exemplars gives higher or equal average accuracy than selecting the $k$ most uncertain questions at once.
  • The gains appear on arithmetic and commonsense reasoning and on both tested GPT models, while shrinking on the stronger model, so adaptive selection behaves as a complementary enhancement of the model's existing capability.
  • On datasets where zero-shot chain-of-thought is already strong, such as GPT-4o mini on letter concatenation, adaptive selection narrows the gap between few-shot and zero-shot performance rather than closing it.
  • At very small exemplar counts the method behaves like one-shot selection because there are not enough iterations to exploit feedback; at very large counts all methods converge, so the adaptive advantage sits at intermediate budgets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not test is to replace the uncertainty proxy with a direct estimate of marginal information gain. If the proxy is doing the work, both should rank candidates similarly.
  • The redundancy argument implies a measurable side effect: adaptive exemplar sets should contain less pairwise redundancy than one-shot top-k sets. Measuring embedding similarity or answer-type overlap of the chosen sets would test that mechanism directly.
  • The same greedy conditional-selection idea should carry over to other budgeted-choice settings for LLMs, such as selecting documents for retrieval-augmented generation or examples for few-shot classification, wherever redundant selections are known to hurt.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Adaptive-Prompt, an iterative exemplar-selection method for in-context learning. Starting from an empty exemplar set, the method repeatedly computes an uncertainty score u(q | E) for each remaining unlabeled question given the current exemplar set, selects the most uncertain question, annotates it, and adds it to the exemplar set. The authors compare this adaptive procedure with non-adaptive baselines, especially Active-Prompt, on six reasoning datasets using GPT-3.5 Turbo and GPT-4o mini, and report accuracy gains on most datasets. They also report experiments on annotation variability, exemplar-set size, and a weaker model (LLaMA3-8B).

Significance. If the adaptive-selection mechanism is validated with a correctly enforced annotation budget, the paper would provide a plausible and practical improvement over one-shot uncertainty-based exemplar selection: greedy feedback-driven selection can reduce redundancy among exemplars and use a fixed annotation budget more effectively. The manuscript includes clear pseudocode, multiple public benchmarks, human-annotation consistency checks, and additional analyses of k and annotator effects, which are useful contributions. However, the current evidence is not yet persuasive because the reported gains are small, no statistical significance is established, the algorithm pseudocode appears to permit one extra exemplar, and the load-bearing uncertainty proxy is not directly validated.

major comments (4)
  1. [Section 4, Algorithm 1] The stopping condition 'while |E| ≤ k' in Algorithm 1 causes the loop to execute one additional time after |E| becomes k, yielding an exemplar set of size k+1. This contradicts Section 4.1, which states that the process 'continues until the exemplar set reaches the desired size k.' If the implementation follows the pseudocode, Adaptive-Prompt uses a strictly larger prompt than all baselines, which are budgeted at k exemplars. Because the reported gains over Active-Prompt are only 0.1–0.8 percentage points on GPT-3.5 and 0.2–0.6 on GPT-4o mini, the extra exemplar alone could account for the advantage. Please correct the condition to 'while |E| < k' or explicitly confirm that the implementation enforces a size-k budget, and report whether the results in Tables 1 and 2 change under the corrected budget.
  2. [Section 6, Tables 1 and 2] The tables report only mean accuracies, with no error bars, confidence intervals, or significance tests, even though Section 5 states that the entire selection process was repeated three times. The abstract's claim that Adaptive-Prompt 'significantly enhances' performance is therefore unsupported by the reported evidence. The differences are often within a few tenths of a percentage point; for example, on GPT-4o mini, Adaptive-Prompt (D) reaches an average of 86.9 versus 86.7 for Active-Prompt (E), and it is worse than Active-Prompt on AQuA, SVAMP, and Letter Concat. Please provide per-run values, standard deviations, and a paired significance test or effect-size measure, and temper the significance language accordingly.
  3. [Sections 4.1 and 4.2] The central mechanism rests on the assumption that a high uncertainty score u(q | E) identifies a question whose addition to E most improves downstream test accuracy. The paper provides no direct evidence for this link: there is no correlation analysis between u(q | E) and the actual marginal accuracy contribution of q, and no ablation comparing the greedy uncertainty rule with alternative adaptive rules (e.g., random adaptive selection, diversity-aware adaptive selection, or re-computed top-k selection). Adding such an analysis, or at least a sensitivity check with a different uncertainty metric, would greatly strengthen the claim that the reported gains come from adaptive feedback rather than from the specific uncertainty estimator.
  4. [Section 6, Evaluation with Weaker Models] The paper states that on LLaMA3-8B, 'in most cases, Zero-Shot CoT outperforms both baselines and our method.' This is a direct counterexample to the general claim that Adaptive-Prompt enhances LLM performance, and it should be treated as a scope condition on the method rather than a side observation. Please discuss the implications for the central claim and qualify the abstract and conclusions accordingly, or provide an explanation supported by additional experiments.
minor comments (5)
  1. [Throughout] There are several typos and formatting issues: 'T able 1' in the table caption, 'SV AMP' should be 'SVAMP', 'examplars' should be 'exemplars', and 'Random-Prompt' and 'Random-CoT' are used inconsistently in the text and figure captions.
  2. [Section 5] The paper says the selection process was repeated three times, but it does not report the random seeds, API sampling temperature, or other details needed to reproduce the three runs. Please add these details or clarify which sources of randomness were controlled.
  3. [Algorithm 1] Algorithm 1 does not specify how ties in the arg max over uncertainty scores are broken. A deterministic tie-breaking rule (e.g., lowest index or random tie-break) should be stated for reproducibility.
  4. [Section 4.2] Reference [6] is cited for the uncertainty score, but the cited paper appears to be about pool-based batch active learning; please verify that this is the correct citation for the disagreement and entropy metrics.
  5. [Section 6] Figures 2 and 3 report results from 'an experiment run' without error bars or multiple runs, which limits the interpretability of the k-sensitivity analysis; please clarify or add variability information.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: Adaptive-Prompt selects exemplars from unlabeled training questions using model uncertainty and is evaluated on a held-out test set; neither test labels nor fitted parameters enter the construction, so the claimed accuracy gains are independent empirical outcomes.

full rationale

I walked the paper's claimed derivation chain. The chain is: define uncertainty u(q | E) as disagreement or entropy over l independent LLM responses (Section 4.2); select the question with the highest u(q | E), annotate it, append it to the exemplar set (Algorithm 1, Section 4.1); repeat until the exemplar set reaches the desired size; finally evaluate accuracy on the test set P with Self-Consistency (Section 5). The selection signal is computed entirely from the unlabeled training pool and the LLM's own responses conditioned on the current exemplar set; it never uses test labels or test-set accuracy, and no parameter is fitted to the evaluation metric. Therefore the reported gains are not equivalent to the inputs by construction. The uncertainty metric is explicitly borrowed from the external baseline Active-Prompt [3], not from prior work by the present authors, so there is no self-citation chain carrying the argument. The heuristic that high-uncertainty questions are the best next exemplars is an empirical premise, not a definition of the outcome; the paper itself reports datasets where Adaptive-Prompt does not beat baselines (AQuA and SVAMP on GPT-4o mini), which shows the result is not forced. I also noted an internal inconsistency between Algorithm 1's 'while |E| ≤ k' and the text's 'continues until the exemplar set reaches the desired size k': as written, the loop would add a (k+1)-th exemplar. That is a potential experimental confound or correctness bug, but it is not a circularity, because even a k+1-shot comparison is still an out-of-sample empirical measurement rather than an identity between the derivation and its inputs. Accordingly, no circular step can be exhibited, and the circularity score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim rests on three unproven assumptions: the uncertainty proxy measures coverage, greedy selection is near-optimal, and the random candidate pool represents the full training distribution. The experimental hyperparameters l, s, k, and the self-consistency count are chosen by hand, but none is fitted to the test set. The paper introduces no new entities.

free parameters (4)
  • number of repeated queries l = 10
    Set in Section 5 to match prior work [3]; uncertainty scores are computed over 10 independent model responses.
  • candidate pool size s = 50 x k (200 to 400)
    Section 5; large training sets are randomly subsampled to this size before selection, influencing which questions can be chosen.
  • self-consistency voting samples = 6
    Section 5; each test question is answered six times and the majority answer is used, which affects the reported accuracy.
  • per-dataset exemplar budget k = 4 to 8
    Adopted from Wei et al. [27]; it sets the stopping condition and varies per dataset.
assumptions (3)
  • domain assumption The disagreement or entropy of l independent model responses is a valid measure of whether the current exemplar set covers a question's required knowledge.
    Invoked in Section 4.2 and used for selection in Algorithm 1; no validation is provided that u(q | E) correlates with downstream test accuracy.
  • ad hoc to paper Greedy selection of the currently most uncertain question leads to an exemplar set that is near-optimal for test accuracy.
    Stated in Section 4.1 step 3 as an intuition ('adding q_j to E is expected to best expand the LLM's knowledge base'); no submodularity or optimality guarantee is shown.
  • domain assumption The random candidate subset of size s = 50 x k is representative of the full training distribution.
    Section 5 Settings; large training sets are subsampled, which can bias selection and results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Power of Adaptation: Boosting In-Context Learning through Adaptive Prompting." pith.science (2026). https://pith.science/paper/XXJ2UYJL

@misc{pith2026241217891,
  author       = {Pith},
  title        = {Pith review of: The Power of Adaptation: Boosting In-Context Learning through Adaptive Prompting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XXJ2UYJL}},
  note         = {Machine review of arXiv:2412.17891}
}
read the original abstract

Large Language Models (LLMs) have demonstrated exceptional abilities across a broad range of language-related tasks, including generating solutions to complex reasoning problems. An effective technique to enhance LLM performance is in-context learning, which encourages a step-by-step reasoning process by including explanatory examples to guide the model's responses. However, selecting appropriate exemplars for the model poses a challenge, as each dataset demands a distinct set of exemplars to enable the LLM to learn effectively and perform well on the test set. Current studies often rely on uncertainty- or diversity-based selection strategies to select exemplars for annotation and to improve model learning. However, these studies typically employ a non-adaptive approach, selecting a set of exemplars all at once. We argue that this non-adaptive strategy may result in a set of exemplars with high redundancy in terms of the knowledge covered, ultimately reducing their overall informativeness. To address this limitation, we propose \textsc{Adaptive-Prompt}, a novel method that adaptively selects exemplars by leveraging model feedback from previously chosen exemplars. Experimental results show that \textsc{Adaptive-Prompt} significantly enhances LLM performance across a variety of reasoning tasks.

Figures

Figures reproduced from arXiv: 2412.17891 by the authors.

Figure 1
Figure 1. Illustration of Adaptive-Prompt. 4 Design of Adaptive-Prompt In this paper, we introduce a novel in-context learning framework, Adaptive￾Prompt. An illustration of Adaptive-Prompt is presented in [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. GSM8K [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 13 canonical work pages

  1. [1]

    arXiv preprint arXiv:2303.08774 (2023)

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

  2. [2]

    arXiv preprint arXiv:2110.14168 (2021)

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al.: Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 (2021)

  3. [3]

    arXiv preprint arXiv:2302.12246 (2023)

    Diao, S., Wang, P., Lin, Y., Zhang, T.: Active prompting with chain-of-thought for large language models. arXiv preprint arXiv:2302.12246 (2023)

  4. [4]

    Association for Computational Linguistics (2019)

    Erdmann, A., Wrisley, D.J., Brown, C., Cohen-Bod´ en` es, S., Elsner, M., Feng, Y., Joseph, B., Joyeux-Prunel, B., de Marneffe, M.C.: Practical, efficient, and cus- tomizable active learning for named entity recognition in the digital humanities. Association for Computational Linguistics (2019)

  5. [5]

    In: The Eleventh International Conference on Learning Representations (2022)

    Fu, Y., Peng, H., Sabharwal, A., Clark, P., Khot, T.: Complexity-based prompting for multi-step reasoning. In: The Eleventh International Conference on Learning Representations (2022)

  6. [6]

    Journal of Machine Learning Research 25(262), 1–42 (2024)

    Gentile, C., Wang, Z., Zhang, T.: Fast rates in pool-based batch active learning. Journal of Machine Learning Research 25(262), 1–42 (2024)

  7. [7]

    Transactions of the Association for Computational Linguistics 9, 346–361 (2021)

    Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., Berant, J.: Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics 9, 346–361 (2021)

  8. [8]

    Advances in neural information processing systems 35, 22199–22213 (2022)

    Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large language models are zero-shot reasoners. Advances in neural information processing systems 35, 22199–22213 (2022)

Show all 32 references
  1. [9]

    arXiv preprint arXiv:2211.08358 (2022)

    K¨ oksal, A., Schick, T., Sch¨ utze, H.: Meal: stable and active learning for few-shot prompting. arXiv preprint arXiv:2211.08358 (2022)

  2. [10]

    arXiv preprint arXiv:1705.04146 (2017)

    Ling, W., Yogatama, D., Dyer, C., Blunsom, P.: Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146 (2017)

  3. [11]

    Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., Chen, W.: What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804 (2021)

  4. [12]

    Transactions of the Association for Computational Linguistics 12, 157–173 (2024)

    Liu, N.F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., Liang, P.: Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics 12, 157–173 (2024)

  5. [13]

    Advances in Neural Information Processing Systems 36, 43136–43155 (2023)

    Ma, H., Zhang, C., Bian, Y., Liu, L., Zhang, Z., Zhao, P., Zhang, S., Fu, H., Hu, Q., Wu, B.: Fairness-guided few-shot prompting for large language models. Advances in Neural Information Processing Systems 36, 43136–43155 (2023)

  6. [14]

    arXiv preprint arXiv:2109.03764 (2021) 12 S

    Margatina, K., Vernikos, G., Barrault, L., Aletras, N.: Active learning by acquiring contrastive examples. arXiv preprint arXiv:2109.03764 (2021) 12 S. Cai et al

  7. [15]

    arXiv preprint arXiv:2310.20046 (2023)

    Mavromatis, C., Srinivasan, B., Shen, Z., Zhang, J., Rangwala, H., Faloutsos, C., Karypis, G.: Which examples to annotate for in-context learning? towards effective and efficient selection. arXiv preprint arXiv:2310.20046 (2023)

  8. [16]

    OpenAI: Openai models documentation (2024), https://platform.openai.com/ docs/models

  9. [17]

    Patel, A., Bhattamishra, S., Goyal, N.: Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191 (2021)

  10. [18]

    arXiv preprint arXiv:2008.07267 (2020)

    Schr¨ oder, C., Niekler, A.: A survey of active learning for text classification using deep neural networks. arXiv preprint arXiv:2008.07267 (2020)

  11. [19]

    arXiv preprint arXiv:2107.05687 (2021)

    Schr¨ oder, C., Niekler, A., Potthast, M.: Revisiting uncertainty-based query strate- gies for active learning with transformers. arXiv preprint arXiv:2107.05687 (2021)

  12. [20]

    Settles, B.: Active learning literature survey (2009)

  13. [21]

    In: International Conference on Machine Learning

    Shao, Z., Gong, Y., Shen, Y., Huang, M., Duan, N., Chen, W.: Synthetic prompt- ing: Generating chain-of-thought demonstrations for large language models. In: International Conference on Machine Learning. pp. 30706–30775. PMLR (2023)

  14. [22]

    arXiv preprint arXiv:2209.01975 (2022)

    Su, H., Kasai, J., Wu, C.H., Shi, W., Wang, T., Xin, J., Zhang, R., Ostendorf, M., Zettlemoyer, L., Smith, N.A., et al.: Selective annotation makes language models better few-shot learners. arXiv preprint arXiv:2209.01975 (2022)

  15. [23]

    In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)

    Talmor, A., Herzig, J., Lourie, N., Berant, J.: Commonsenseqa: A question an- swering challenge targeting commonsense knowledge. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ...

  16. [24]

    arXiv preprint arXiv:2307.09288 (2023)

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bash- lykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

  17. [25]

    arXiv preprint arXiv:2203.11171 (2022)

    Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., Zhou, D.: Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022)

  18. [26]

    Transactions on Machine Learning Research (2022)

    Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al.: Emergent abilities of large language models. Transactions on Machine Learning Research (2022)

  19. [27]

    Advances in neural information processing systems 35, 24824–24837 (2022)

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35, 24824–24837 (2022)

  20. [28]

    arXiv preprint arXiv:2210.03493 (2022)

    Zhang, Z., Zhang, A., Li, M., Smola, A.: Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493 (2022)

  21. [29]

    arXiv preprint arXiv:2303.18223 (2023)

    Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al.: A survey of large language models. arXiv preprint arXiv:2303.18223 (2023)

  22. [30]

    In: Findings of the Association for Computational Linguistics: EMNLP 2020

    Zhao, Y., Zhang, R.H., Zhou, S., Zhang, Z.: Active learning approaches to enhanc- ing neural machine translation. In: Findings of the Association for Computational Linguistics: EMNLP 2020. pp. 1796–1806 (2020)

  23. [31]

    In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

    Zhou, C., You, W., Li, J., Ye, J., Chen, K., Zhang, M.: Inform: Information entropy based multi-step reasoning for large language models. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pp. 3565–3576 (2023)

  24. [32]

    In: 22nd International Conference on Computational Linguistics, Coling 2008

    Zhu, J., Wang, H., Yao, T., Tsou, B.K.: Active learning with sampling by uncer- tainty and density for word sense disambiguation and text classification. In: 22nd International Conference on Computational Linguistics, Coling 2008. pp. 1137– 1144 (2008)

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.