Pith. sign in

REVIEW 4 major objections 5 minor 47 references

Surprise Calibration for Better In-Context Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that the surprise a language model registers just before a label is revealed — the negative log probability of that label under the current context — is a reliable signal for how much and in which direction its class…

desk verdict A practical ICL calibration method that trains a GRU on signed surprise values to adjust class priors per query; the empirical gains are real, but the paper's theoretical framing outstrips the evidence. read the letter →

arxiv 2506.12796 v2 pith:HTGF7DRL submitted 2025-06-15 cs.CL

Zhihang Tan , Jingrui Hou , Ping Wang , Qibiao Hu , Peng Zhu This is my paper · ORCID
classification cs.CL
keywords in-contextlearningbiascalibrationclasspriorshiftsurprisesignalsequentialBayesianinferencelargelanguagemodelsdynamicdemonstrationselectionrecurrent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that surprise — the negative log probability a language model assigns to a demonstration's true label just before that label is revealed — is a usable signal for detecting class-prior shifts during in-context learning, and that a calibration method built on it beats fixed-prior calibration with fewer inference passes. The authors view in-context learning as implicit sequential Bayesian inference, in which each demonstration updates the model's belief about an unobserved task concept, with surprise amplifying the update. From this they build Surprise Calibration (SC), which converts each demonstration into a signed surprise vector and feeds the sequence through a recurrent network whose output adjusts the model's log-probabilities for each query. This matters because in dynamic in-context learning every query can have its own demonstration context, so a single fixed prior is the wrong tool, while per-query prior estimation is expensive; SC is designed to give the benefit of per-query priors without that cost.

What carries the argument

The central object is the surprise vector, defined from the model's own pre-label predictive probability: for a demonstration $(e_j, y_j)$, entry $c$ equals $(1 - 2\delta_{c,y}) \log p(y=c \mid e_j, D_{j-1})$, where the sign encodes the expected direction of prior adjustment for that class and the magnitude encodes how surprising the revealed label was. Collecting these vectors over the demonstration sequence gives a surprise sequence, and a gated recurrent unit processes that sequence to produce a cumulative per-class prior adjustment $a$, applied as $-\log p_{\mathrm{calib}}(y' \mid e, z) = -\log p_{\mathrm{orig}}(y' \mid e, z) - a_{y'}$. The machinery carries the argument because it converts a theoretical claim about Bayesian belief updates — surprise amplifies the covariance-driven shift in priors — into a trainable, context-dependent correction that requires no extra per-query sampling.

What would settle it

Repeat the paper's Figure 2 protocol on datasets where the input likelihood $p(e_j \mid D_{j-1})$ varies widely across demonstrations; if the label-conditioned Spearman correlation between surprise and measured prior shift is not consistently positive and statistically significant, the surprise signal is not reliable and SC's input features would be noise.

Watch

Extended reading notes

Core claim

The central claim is that the probability the model assigns to a label before seeing it, $p(y_j \mid e_j, D_{j-1})$, tracks how the class prior should move: a low probability (high surprise) for the label that actually appears signals a larger shift, with the direction set by whether the label matches the class in question. In the paper's Bayesian reading, the update to the prior for a class $y'$ splits into a baseline expectation plus a covariance term between $p(y' \mid z)$ and $p(e_j, y_j \mid z)$, and a lower average joint likelihood amplifies that term. SC operationalizes this by forming, for every demonstration, a surprise vector whose $c$-th entry is $(1 - 2\delta_{c,y}) \log p(y=c \mid e_j, D_{j-1})$, feeding the resulting sequence through a gated recurrent unit, and decoding the final hidden state into a per-class adjustment that is subtracted from the original log-probabilities. Empirically, the paper reports that SC improves on vanilla in-context learning by an average of +4.59 points on a 3-billion-parameter model and +3.54 points on a 7-billion-parameter model across eight datasets and six NLP tasks, matches or exceeds the best fixed-prior baseline in most settings, and needs only a single inference pass per query plus a small labeled training set rather than repeated per-query sampling. The paper also shows its calibrated probability ratios track batch-calibrated priors with $R^2 = 0.838$ and $R^2 = 0.977$ on two datasets.

Load-bearing premise

The load-bearing premise is that high surprise — the true label looking unlikely before it is revealed — reliably signals a larger class-prior shift, a link the paper explicitly concedes is not formally guaranteed because the predictive probability is normalized by $p(e_j \mid D_{j-1})$.

Editorial extensions

If this is right

  • Per-query class-prior calibration no longer requires repeated sampling: once trained, SC costs one forward pass per query plus a small labeled training set, instead of the multiplicative inference count of dynamic baselines.
  • Across eight datasets and two model sizes, SC outperforms vanilla in-context learning by an average of +4.59 and +3.54 accuracy points and matches or exceeds existing calibration baselines in most settings.
  • In 2-shot settings, SC can match or beat other calibration methods in 3-shot settings and vanilla ICL in 5-shot settings, meaning shorter demonstration contexts can be used at the same accuracy.
  • Surprise magnitude carries information: ablating it to sign-only reduces accuracy on most datasets, so the strength of the surprise, not just its direction, is needed for prior estimation.
  • SC's improvements persist across demonstration selection strategies, ordering strategies, verbalizer choices, and contexts with 1 to 15 demonstrations, indicating the adjustment is not tied to one prompt format.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because surprise is read from the model's own next-token probabilities before labels are revealed, the same signal is available for any autoregressive language model that exposes per-token logits, so the method should transfer beyond the two model sizes tested here.
  • The paper's observation that anti-recency bias shifts mean priors while preserving the surprise-prior relationship suggests surprise could be used to separate recency effects from genuine prior shifts, something fixed-prior methods cannot do.
  • A testable extension is to use the accumulated adjustment not only to correct final predictions but also to re-rank retrieved demonstrations, since demonstrations that provoke high surprise may be the ones that most strongly move the model's priors.
  • The reported saturation of gains beyond 10 demonstrations implies the surprise sequence could be truncated once the GRU's adjustment stabilizes, giving an even cheaper variant that stops reading new demonstrations once priors stop moving.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper interprets in-context learning (ICL) as implicit sequential Bayesian inference, defines a per-demonstration surprise signal as the negative log probability of the observed label before it is revealed, and hypothesizes that surprise indicates a shift in the class prior. On this basis it proposes Surprise Calibration (SC), which feeds a sequence of surprise vectors into a GRU, decodes a prior-adjustment vector, and uses it to recalibrate the model's log-probabilities. The paper reports experiments on eight datasets with Qwen2.5-3B and Qwen2.5-7B, comparing SC against ICL, BC, LinC, CC+, BC+, and LinC+ across demonstration selection and ordering strategies, and claims that SC outperforms state-of-the-art ICL bias calibration baselines while requiring fewer inference iterations.

Significance. If the central interpretive claim held, the paper would make a useful conceptual and practical contribution: it connects a measurable quantity (surprise) to class-prior adjustment in ICL and offers a calibration method that avoids per-query repeated sampling. The Bayesian identities in Eqs. (5)–(7) are correctly derived, and the empirical evaluation is broad, covering two model sizes, eight datasets, multiple demonstration selection and ordering strategies, and several shot counts. The paper explicitly acknowledges limitations, including the heuristic nature of the surprise-to-prior link and the need for labeled training data. However, the load-bearing premise is not formally guaranteed and the direct empirical evidence is only partially supportive, while the headline superiority claim is not uniformly supported by the reported tables. The method appears competitive and often strong, but the paper's framing overstates what is established.

major comments (4)
  1. [Section 3.1, Eq. (7)] The central theoretical link is explicitly heuristic: the paper states that low p(y_j | e_j, D_{j-1}) 'may suggest' lower E_z[p(e_j,y_j|z)] but that this 'is not formally guaranteed due to the normalization by p(e_j | D_{j-1})'. Since Eq. (8) constructs the surprise vector entirely from this unproven proxy, the derivation does not establish that surprise 'systematically covaries with prior adjustments.' The authors should either provide a formal condition under which the proxy is valid (for instance, bounded or slowly varying p(e_j|z)), or clearly reframe the contribution as an empirically motivated heuristic rather than a theoretically grounded signal.
  2. [Section 3.2, Figure 2] The empirical validation reported in Figure 2 is conditioned on the inserted label's class, and the text admits that 'aggregate correlations occasionally showed instability' due to an anti-recency bias. The conclusion that the results 'robustly support' the theoretical assertion is therefore too strong: the scatter plots demonstrate label-conditioned monotonicity, not a general, unconditional relationship between surprise and prior shift. The authors should report the aggregate correlations with appropriate statistical tests across both insertion classes, or explicitly qualify the claim as label-conditioned evidence for a heuristic.
  3. [Section 5.2, Table 2] The abstract's claim that SC outperforms state-of-the-art ICL bias calibration baselines is not uniformly supported by Table 2. On Qwen2.5-3B, BC+ outperforms SC on SST-2 (90.33 vs 89.68), MNLI (79.07 vs 76.33), and QNLI (79.53 vs 78.01); on Qwen2.5-7B, LinC+ outperforms SC on QNLI (81.01 vs 80.48) and WiC (63.92 vs 62.93). The paper should report win rates over all dataset-model combinations and use paired significance tests or confidence intervals before claiming general superiority over the best baseline.
  4. [Section 5.4, Table 3 and Figure 8] The ablation and effectiveness analyses do not isolate surprise as the operative mechanism. The ablation in Table 3 only removes surprise magnitude, retaining the sign and the full GRU sequence, so it does not test whether the surprise values themselves are necessary; a control with random features or label-only features is needed to show that the GRU is not simply exploiting label order and frequencies. In addition, removing magnitude improves accuracy on MNLI (77.02 vs 76.33 with magnitude), contradicting the claim that surprise magnitude plays a 'crucial role' in all cases. The high R^2 in Figure 8 between SC and BC calibrated ratios may merely reflect that both methods are derived from the same labeled data and approximate the same empirical class-conditional distribution, rather than demonstrating that surprise drives the adjustment.
minor comments (5)
  1. [Appendix A.2] The text refers to 'the BSC model' after introducing SC; this appears to be a typo for 'the SC model.'
  2. [Table 4] The table header spells 'Verbalizer' as 'Verbizer'; please correct the typo.
  3. [References] The reference to 'Tefnik and Kadlcik' appears to be a misspelling of 'Teplitsky' or the intended author name; please verify the citation.
  4. [Section 3.2] The text defines the recorded surprise as '-log p(y|e,z)', but the formal definition in Eq. (4) uses p(y_j | e_j, D_{j-1}); using the same notation would avoid confusion.
  5. [Table 1] The first row appears as 'BCT' rather than 'BC T' or a properly spaced entry; please format the table consistently.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the surprise signal is computed from pre-label predictive probability and the calibration model is trained against external ground-truth labels.

full rationale

The paper's derivation chain is not circular. The central signal, Surprise(yj | ej, Dj-1) = -log p(yj | ej, Dj-1) (Eq. 4), is defined from the LLM's predictive distribution before the label is revealed, and the calibration target is the ground-truth label of the query, so the supervised training signal is external to the surprise definition. The Bayesian update in Eq. 5 and the prior-shift expression in Eq. 6 are algebraic identities given the latent-variable model. Eq. 7 is also an identity, p(yj | ej, Dj-1) = E[p(ej, yj | z) / p(ej | z)], and the paper explicitly states that the link to the covariance denominator is 'not formally guaranteed due to the normalization by p(ej | Dj-1)' (Section 3.1), i.e., it is offered as a hypothesis rather than as a derived theorem. The empirical support in Figure 2 correlates surprise with separately measured BC prior shifts; although the authors report label-conditioned correlations and acknowledge that aggregate correlations are 'occasionally unstable,' this is a weakness of the premise, not a definitional equivalence. SC's GRU is trained end-to-end on labeled examples and evaluated on held-out accuracy; no fitted parameter is renamed as a prediction, and no load-bearing claim is justified by self-citation. The accuracy improvements in Table 2 are independent empirical results, though they do not by themselves validate the surprise-prior interpretation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a standard latent-variable view of ICL, a heuristic link between surprise and Bayesian updates, and a learned sequence model whose hyperparameters are chosen on one dataset and transferred to others. The free parameters are primarily the trained GRU/decoder weights and the hand-selected training hyperparameters.

free parameters (3)
  • GRU and decoder parameters = trained on labeled calibration data
    The prior adjustment vector a in Eq 9 is produced by a GRU and decoding layers whose weights are learned end-to-end on each dataset's training labels. The central method depends on these fitted parameters.
  • Learning rate and training epochs = lr=1e-4, 200 epochs
    Chosen by early-stage experiments on MRPC (20% validation) and applied unchanged to all datasets. These hand-selected hyperparameters affect performance.
  • Training sample size = 125 to 2000 depending on setting
    The number of labeled samples used to train SC is varied in Figure 7; the exact size used for the main results in Table 2 is not stated precisely, but it is a data-dependent choice.
assumptions (4)
  • domain assumption In-context learning can be modeled as Bayesian inference over a latent concept z with posterior p(z|D), as in Eq 3.
    Invoked in Section 3.1 to motivate surprise. This is a modeling assumption inherited from Xie et al. (2021), not proven in this paper.
  • standard math Bayes' rule and the covariance decomposition in Eqs 5-6 are valid.
    Used in Section 3.1; these are standard probability identities and are applied correctly.
  • ad hoc to paper Low p(y_j|e_j,D_{j-1}) implies lower expected joint likelihood p(e_j,y_j|z), making surprise a proxy for prior shift.
    Explicitly hypothesized in Section 3.1 as "not formally guaranteed". This is the key heuristic on which the method's feature design relies.
  • domain assumption A GRU trained on surprise sequences can learn useful prior adjustments across datasets.
    The calibration model is trained, but the representational sufficiency of surprise features is assumed; it is supported only by empirical validation on the benchmark tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Surprise Calibration for Better In-Context Learning." pith.science (2026). https://pith.science/paper/HTGF7DRL

@misc{pith2026250612796,
  author       = {Pith},
  title        = {Pith review of: Surprise Calibration for Better In-Context Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HTGF7DRL}},
  note         = {Machine review of arXiv:2506.12796}
}
read the original abstract

In-context learning (ICL) has emerged as a powerful paradigm for task adaptation in large language models (LLMs), where models infer underlying task structures from a few demonstrations. However, ICL remains susceptible to biases that arise from prior knowledge and contextual demonstrations, which can degrade the performance of LLMs. Existing bias calibration methods typically apply fixed class priors across all inputs, limiting their efficacy in dynamic ICL settings where the context for each query differs. To address these limitations, we adopt implicit sequential Bayesian inference as a framework for interpreting ICL, identify "surprise" as an informative signal for class prior shift, and introduce a novel method--Surprise Calibration (SC). SC leverages the notion of surprise to capture the temporal dynamics of class priors, providing a more adaptive and computationally efficient solution for in-context learning. We empirically demonstrate the superiority of SC over existing bias calibration techniques across a range of benchmark natural language processing tasks.

Figures

Figures reproduced from arXiv: 2506.12796 by the authors.

Figure 1
Figure 1. Illustration of surprise signal. The yellow rect [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Spearman correlation between surprise (− log p(y|e, D)) and the prior probability of the positive class across four binary classification datasets (AIGA, MRPC, QNLI, WiC). Each scatter plot shows the relationship of surprise and positive class prior for positive (orange) and negative (purple) demonstration insertions, with corresponding Spearman ρ and p-values. The gray dashed line marks the estimated prior before i… view at source ↗
Figure 3
Figure 3. Illustration of the proposed Surprise Calibra [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Accuracy comparison between SC and ICL across 3 demonstration selection and demonstration or￾dering strategies with other settings kept consistent as shown in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Average performance across 8 datasets varies [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Performance comparison of calibration meth [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Scatter plot comparing the calibrated prob [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 7
Figure 7. Figure 7: Performance across varying number of train [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 9
Figure 9. Figure 9: Spearman correlation between surprise (− log p(y|e, D)) and the prior probability of the positive class across various in-context settings (3-, 5-, 10-, 20-shot) with MRPC dataset. Each scatter plot shows the relationship for positive (orange) and negative (purple) dem…
Figure 10
Figure 10. Figure 10: Accuracy comparisons between SC and various methods (BC, LinC, CC+, BC+, LinC+) across three [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Accuracy comparisons between SC and various methods (BC, LinC, CC+, BC+, LinC+) across three [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 22 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Momin Abbas, Yi Zhou, Parikshit Ram, Nathalie Baracaldo, Horst Samulowitz, Theodoros Salonidis, and Tianyi Chen. 2024. Enhancing in-context learning via linear probe calibration. In International Conference on Artificial Intelligence and Statistics, pages 307--315. PMLR

    work page 2024

  4. [4]

    Rishabh Agarwal, Avi Singh, Lei Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, et al. 2024. Many-shot in-context learning. Advances in Neural Information Processing Systems, 37:76930--76966

    work page 2024

  5. [5]

    T \'u lio C Alberto, Johannes V Lochter, and Tiago A Almeida. 2015. Tubespam: Comment spam filtering on youtube. In 2015 IEEE 14th international conference on machine learning and applications (ICMLA), pages 138--143. IEEE

    work page 2015

  6. [6]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  7. [7]

    Bill Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Third international workshop on paraphrasing (IWP2005)

    2005

  8. [8]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. 2022. A survey on in-context learning. arXiv preprint arXiv:2301.00234

    arXiv 2022

Show all 47 references
  1. [9]

    Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselut. 2023. Mitigating label biases for in-context learning. In Proceedings Of The 61St Annual Meeting Of The Association For Computational Linguistics (ACL 2023): Long Papers, Vol 1, pages 14014--14031

    2023

  2. [10]

    Qi Guo, Leiyu Wang, Yidong Wang, Wei Ye, and Shikun Zhang. 2024. What makes a good order of examples in in-context learning. In Findings of the Association for Computational Linguistics: ACL 2024, pages 14892--14904

    2024

  3. [11]

    Seungwook Han, Jinyeop Song, Jeff Gore, and Pulkit Agrawal. 2024. Emergence of abstractions: Concept encoding and decoding mechanism for in-context learning in transformers. arXiv preprint arXiv:2412.12276

    2024 arXiv

  4. [12]

    Zhixiong Han, Yaru Hao, Li Dong, Yutao Sun, and Furu Wei. 2022. Prototypical calibration for few-shot learning of language models. arXiv preprint arXiv:2205.10183

    2022 arXiv

  5. [13]

    Roee Hendel, Mor Geva, and Amir Globerson. 2023. In-context learning creates task vectors. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9318--9333

    2023

  6. [14]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. NeurIPS

    2021

  7. [15]

    Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. Surface form competition: Why the highest probability answer isn’t always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7038--7051

    2021

  8. [16]

    Jannik Kossen, Yarin Gal, and Tom Rainforth. 2024. In-context learning learns label relationships but is not conventional learning. In The Twelfth International Conference on Learning Representations

    2024

  9. [17]

    Yoav Levine, Noam Wies, Daniel Jannai, Dan Navon, Yedid Hoshen, and Amnon Shashua. 2021. The inductive bias of in-context learning: Rethinking pretraining example design. arXiv preprint arXiv:2110.04541

    2021 arXiv

  10. [18]

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Syste...

    2022

  11. [19]

    Xiaonan Li and Xipeng Qiu. 2023. Finding supporting examples for in-context learning. CoRR

    2023

  12. [20]

    Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281

    2023 arXiv

  13. [21]

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. https://doi.org/10.18653/v1/2022.deelio-1.10 What makes good in-context examples for GPT -3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extr...

    2022 doi

  14. [22]

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1--35

    2023

  15. [23]

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. https://doi.org/10.18653/v1/2022.acl-long.556 Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity . In Proceedings of the 60th Annual Meeting of th...

    2022 doi

  16. [24]

    Man Luo, Xin Xu, Yue Liu, Panupong Pasupat, and Mehran Kazemi. 2024. https://arxiv.org/abs/2401.11624 In-context learning with retrieved demonstrations for language models: A survey . Preprint, arXiv:2401.11624

    2024 arXiv

  17. [25]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing...

    2022

  18. [26]

    Keqin Peng, Liang Ding, Yancheng Yuan, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2024. Revisiting demonstration selection strategies in in-context learning. arXiv preprint arXiv:2401.12087

    2024 arXiv

  19. [27]

    Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. Wic: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu...

    2019

  20. [28]

    Chengwei Qin, Aston Zhang, Chen Chen, Anirudh Dagar, and Wenming Ye. 2023. In-context learning with iterative demonstration selection. arXiv preprint arXiv:2310.09881

    2023 arXiv

  21. [29]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  22. [30]

    Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655--2671

    2022

  23. [31]

    Dong Shu and Mengnan Du. 2024. Comparative analysis of demonstration selection algorithms for llm in-context learning. arXiv preprint arXiv:2410.23099

    2024 arXiv

  24. [32]

    Chenglei Si, Dan Friedman, Nitish Joshi, Shi Feng, Danqi Chen, and He He. 2023. Measuring inductive biases of in-context learning with underspecified demonstrations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape...

    2023

  25. [33]

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language...

    2013

  26. [34]

    Michal Tefnik and Marek Kadlcik. 2023. Can in-context learners learn a reasoning concept from demonstrations? In The 61st Annual Meeting Of The Association For Computational Linguistics

    2023

  27. [35]

    Theocharopoulos, Panagiotis Anagnostou, Anastasia Tsoukala, Spiros V

    Panagiotis C. Theocharopoulos, Panagiotis Anagnostou, Anastasia Tsoukala, Spiros V. Georgakopoulos, Sotiris K. Tasoulis, and Vassilis P. Plagianakos. 2023. https://doi.org/10.1109/BigDataService58306.2023.00011 Detection of fake generated scientific abstracts . In 2023 IEEE Ni...

    2023

  28. [36]

    Eric Todd, Millicent Li, Arnab Sharma, Aaron Mueller, Byron C Wallace, and David Bau. 2024. Function vectors in large language models. In International Conference on Learning Representations. ICLR

    2024

  29. [37]

    Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. 2024. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. Advances in Neural Information Processing Systems, 36

    2024

  30. [38]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  31. [39]

    Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2...

    2018

  32. [40]

    Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Lingpeng Kong. 2023. Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ...

    2023

  33. [41]

    Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2021. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations

    2021

  34. [42]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  35. [43]

    Yiming Zhang, Shi Feng, and Chenhao Tan. 2022. Active example selection for in-context learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9134--9148

    2022

  36. [44]

    Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. 2024. What makes good examples for visual in-context learning? Advances in Neural Information Processing Systems, 36

    2024

  37. [45]

    Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning, pages 12697--12706. PMLR

    2021

  38. [46]

    Han Zhou, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine Heller, and Subhrajit Roy. 2023. Batch calibration: Rethinking calibration for in-context learning and prompt engineering. arXiv preprint arXiv:2309.17249

    2023 arXiv

  39. [47]

    Yuxiang Zhou, Jiazheng Li, Yanzheng Xiang, Hanqi Yan, Lin Gui, and Yulan He. 2024. The mystery of in-context learning: A comprehensive survey on interpretation and analysis. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14365--14378

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.