Pith. sign in

REVIEW 5 major objections 6 minor 55 references

Investigating the Zone of Proximal Development of Language Models for In-Context Learning

T0 review · 5 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that whether a query benefits from in-context demonstrations is a per-query, per-model property that can be predicted from the query text alone, and that this prediction enables cheaper inference and better fine-tuning…

desk verdict Oracle demonstrations leak the answer, but the framework and MIRT-ICL are genuinely useful; the paper needs a reported threshold and a less answer-tuned Oracle before the ZPD claim holds. read the letter →

arxiv 2502.06990 v1 pith:DJJBCHKZ submitted 2025-02-10 cs.CL

Peng Cui , Mrinmaya Sachan This is my paper · ORCID
classification cs.CL
keywords in-contextlearningzoneofproximaldevelopmentitemresponsetheoryselectiveinferencecurriculumLLMevaluationdemonstrationselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the benefit a large language model gets from in-context demonstrations is a per-query, per-model property that can be measured and predicted without seeing the demonstrations themselves. It adapts the educational idea of the zone of proximal development to sort individual queries into three zones: already solved by direct prompting, solvable only with demonstrations, and unsolvable even with demonstrations. The authors train a variant of item response theory with separate latent traits for direct knowledge and in-context learnability, and show that this model can predict the zone of held-out queries with useful accuracy. If right, the framework makes inference cheaper by applying demonstrations only where a given model will benefit, and makes fine-tuning better by presenting examples inside each model's zone of proximal development first.

What carries the argument

The load-bearing object is the gated multidimensional item response theory model MIRT ICL (Eq. 12), which treats each LLM as a respondent with two latent skill vectors and each query as an item with two discrimination vectors and one difficulty scalar. Direct prompting activates only the knowledge term θ^T α − d, while in-context prompting adds the learnability term θ_c^T α_c; the shared difficulty d is what lets the model explain a query moving from wrong to right. The second load-bearing piece is the oracle demonstration construction of Eq. 5, a greedy search that maximizes the model's own likelihood of the ground-truth answer over a retrieval candidate pool, turning the unobservable 'ideal ICL' into concrete zone labels.

What would settle it

Take the queries that the oracle pipeline labels Z✗→✗, run a much stronger demonstration search over the same pool (for example, all subsets of size 8 for a few hundred candidates per query, or demonstrations drafted by a larger model), and count how many flip to correct; a large flip rate would show the measured zone is an artifact of the greedy construction rather than a property of the query–model pair.

Watch

Extended reading notes

Core claim

The central claim is that in-context learning behavior is generally predictable from the query text and the model identity alone, even before any demonstration is chosen. The paper's three-zone taxonomy labels each query as Z✓ (correct under direct prompting), Z✗→✓ (wrong directly, correct with oracle-quality demonstrations: the model's zone of proximal development), or Z✗→✗ (wrong even with demonstrations). To make these labels measurable, the authors construct oracle demonstrations by greedy likelihood maximization: for each query, they retrieve candidates with BM25 and sentence embeddings, then iteratively add the demonstration that most increases the model's probability of the ground-truth answer. The predictor, gated multidimensional item response theory, gives each model a knowledge-skill vector and an in-context-learnability vector, and each query two discrimination vectors plus a shared difficulty scalar, so one model produces correctness probabilities for both the direct and in-context settings. Trained on eight LLaMA models over a math word-problem dataset and a stance-detection dataset, the predictor reaches held-out AUC around 0.86 and 0.80 respectively, supporting the claim that demonstration-agnostic zone prediction is possible to a practically useful degree, though the degree varies by task.

Load-bearing premise

The load-bearing premise is that the greedily constructed oracle demonstrations—selected to maximize the model's own likelihood of the ground-truth answer—faithfully stand in for ideal in-context support; if that search is too weak, many genuinely learnable queries get mislabeled as hopeless, and every downstream prediction and curriculum inherits the mislabel.

Editorial extensions

If this is right

  • Selective ICL can reduce inference cost without losing accuracy: on 6 of 8 LLaMA models, applying demonstrations only to predicted ZPD queries dominates full ICL in both accuracy and token count.
  • Fine-tuning can be ordered as a ZPD-based curriculum: ranking examples by predicted ICL gain and feeding them in increasing buckets gives faster convergence and better final accuracy than a random order.
  • Demonstration selection must be model-aware, not just similarity-aware, because the same query falls in different zones for different models; maximally shared zones are Z✓, while Z✗→✓ and Z✗→✗ overlap only partially.
  • A query's difficulty under direct prompting and its learnability in context are separate axes; for stance detection the correlation is negative, so hard-for-the-model queries are often exactly the ones where demonstrations help.
  • In-context learning and fine-tuning draw on a consistent notion of learnability: examples that are ICL-learnable also show lower and more stable fine-tuning loss, which is why ordering fine-tuning data by predicted ICL gain works.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to make the zone predictor a latent 'learnability score' used at deployment time: run a small model family on a few hundred labeled queries from a new task, train MIRT ICL, and then decide per query and per model whether to spend demonstration tokens; the paper reports the Pareto curve but does not test the fully deployed online version.
  • The paper merges Z✓→✗ (demonstrations that actively hurt) into Z✓ because it measures oracle behavior; a direct extension would predict this harmful zone explicitly and use it to block ICL, which could matter more than recalling ZPD examples for models like LLaMA-3-70B where the negative effect outweighs the benefit.
  • Since zone labels come from greedy oracle search, the framework's value depends on that search; a testable extension is to compare zones measured with oracle demonstrations against zones measured with a small number of random demonstration sets, to see whether predicted zones are stable or an artifact of the construction.
  • The curriculum application assumes ZPD examples are 'learnable and not yet learned'; a further test is whether the same ZPD-based ordering transfers to multi-task fine-tuning or to model families beyond LLaMA, since the paper only reports single-task results on one 8B instruction-tuned model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces a learning-analytics framework that adapts the educational concept of the Zone of Proximal Development (ZPD) to in-context learning (ICL) in LLMs. It partitions queries into three zones—Z✓ (solved without demonstrations), Z✗→✓ (solved only with demonstrations), and Z✗→✗ (unsolvable even with demonstrations)—using an Oracle demonstration construction (Eq. 5). It then proposes a multidimensional item response theory model with an ICL-specific skill vector (MIRT ICL, Eq. 12) to predict these zones from query text and model identity without seeing demonstrations. Two applications are presented: SELICL, a selective ICL strategy that chooses whether to prepend demonstrations based on predicted probabilities, and a ZPD-based curriculum that prioritizes Z✗→✓ examples during fine-tuning. The main claim is that ICL behavior is generally predictable without demonstration information, and that this predictability enables cost/accuracy trade-offs and improved fine-tuning curricula.

Significance. If the zone labels are valid, the paper offers a useful formalization of per-query, per-model in-context learnability and demonstrates two practical applications. The IRT variant is a sensible way to jointly model performance with and without ICL, and the inclusion of code and reproducible experimental setup is a strength. However, the central claim rests entirely on the Oracle demonstration construction in Section 3.2, which is model-specific and ground-truth-tuned. The applications inherit the validity of these labels. The paper also makes a falsifiable prediction (ICL behavior is predictable without demonstrations) and reports quantitative AUC and accuracy results, though the gains over strong baselines are modest and uneven across datasets. The curriculum result is interesting but currently only compared against a random baseline. Overall, the framework is potentially valuable, but its empirical support is conditional on resolving the Oracle-construction concern.

major comments (5)
  1. [§3.2, Eq. (5)] The Oracle demonstration construction is the load-bearing component of the entire pipeline, but it is self-referential and potentially upwardly biased. Equation (5) greedily selects demonstrations to maximize Prob_M(y | c1...ci ⊕ x), i.e., the same model's likelihood of the ground-truth answer, and the candidate pool is partly built by retrieving on the ground-truth answer (Section 3.2, '2×2×K candidates' includes similarity based on y only). Thus a query whose gold answer can be copied or inferred from a similar solved example will be labeled Z✗→✓ even if the model acquired no task ability. For GSM8K, demonstrations contain full step-by-step solutions, so high Prob_M(y|...) can reflect answer leakage. The Limitations section acknowledges that the Oracle is a lower bound, but that addresses incompleteness, not the upward bias from ground-truth-tuned selection. This concern propagates to the IRT training labels, the SELICL decisions, and the curriculum ordering. I recommend validating the zone labels against external demonstration-selection strategies (e.g., random or KATE) and controlling for answer leakage, or explicitly showing that the conclusions are robust to the choice of Oracle definition.
  2. [§3.1, Eq. (1)] The threshold τ in the definition of Z✗→✓ is never reported in the paper, and no sensitivity analysis is provided. Since the entire zone split depends on this threshold, the main empirical results (zone distributions, Table 1, and the IRT training labels) are not reproducible without knowing τ. The threshold should be explicitly reported for each dataset, and the paper should show how the zone distributions and downstream results vary with τ (e.g., a small sweep).
  3. [§5.2, Table 2] The AUC results do not uniformly support the claim that ICL behavior is predictable without demonstration information. On EZStance, MIRT ICL achieves the same overall AUC as the MIRT baseline (0.799 vs 0.799) and only +0.010 on DP AUC, while on GSM8K the gain is +0.119 overall. The baselines are trained only on DP data, so the comparison is not apples-to-apples, and there are no confidence intervals or significance tests. Given that the paper's headline claim is 'generally predictable,' the dataset-dependent and modest gains need statistical grounding; otherwise the claim should be qualified as strong only for some datasets.
  4. [§5.3.1, Figure 5] The SELICL Pareto plots in Figures 5 and 8 show point estimates without error bars, and the thresholds τ1 and τ2 are selected by grid search on the IRT validation set. This raises the risk of overfitting to the validation set; the reported '★' points that dominate FULICL could reflect threshold selection bias. Reporting variance across multiple validation splits or bootstrap resamples, and perhaps fixing thresholds on a separate development set, would strengthen the cost/accuracy claim.
  5. [§5.3.2, Figure 6] The curriculum learning experiment is compared only against a random baseline. While the paper notes that random is a common baseline, the claim that ZPD-based curriculum 'results in improved performance' would be more convincing with additional baselines, such as an anti-curriculum (hardest-first), a difficulty-only curriculum, or a curriculum based on the predicted gain pc − p∅ without zone labels. Moreover, footnote 3 in Figure 7 admits that some Z✗→✓ examples are misclassified into Z✗→✗ due to Oracle sub-optimality, which weakens the interpretation of the training-dynamics analysis. These issues do not invalidate the result, but they currently leave the advantage attributable to the ZPD mechanism rather than to the specific ranking criterion unclear.
minor comments (6)
  1. [§5.3.1] There is a typo: 'paticularly' should be 'particularly'.
  2. [Eq. (8)] The loss notation says CE(P(ri,j), yj), but yj is the ground-truth label, not the query; the convention is inconsistent with the earlier definition of xi and yi. Please align notation.
  3. [Table 3] The model abbreviations 'L2-7B', 'L2-13B', 'L3-8B', 'L3-70B' are not introduced in the table caption; please define them or use consistent model names.
  4. [§A.2] The dataset construction says N_GSM8K = 1319 and N_EZStance = 6703, but the main text refers to the GSM8K test set and EZStance validation set; please clarify exactly which splits are used and how the test splits for IRT are formed.
  5. [§5.1, Table 1] The header 'Max Min A vg' contains a formatting artifact ('A vg' should be 'Avg'); please correct.
  6. [Appendix B.2] The EZStance SELICL results are only shown in the appendix figure, but the main text in §5.3.1 claims results for both datasets without referencing the appendix figure at the appropriate place; please add a cross-reference.

Circularity Check

1 steps flagged · score 4.0 of 10

Oracle demonstrations in Eq. 5 are selected to maximize the model's own probability of the ground-truth answer, making the measured ZPD labels self-referential; the IRT prediction itself is held-out and retains independent content.

  1. self definitional [Section 3.1 Eq. (1); Section 3.2 Eq. (5)]
    "Z✗→✓≜ {x∣F(y∅)< τ, F(yc)> τ}, (1) ... c_i= argmax_{C\{c1,..,ci−1\}} ProbM(y∣c1⊕ ...ci⊕ x), (5) In other words, we greedily choose demonstrations that can maximize the likelihood of the ground-truth answer."

    Eq. (1) labels a query as in the ZPD exactly when F(y_c)>τ, where y_c is the same model's output under demonstrations c. Those demonstrations are not independent scaffolding: Eq. (5) selects each c_i to maximize ProbM(y∣...), i.e., the same model's probability of the same ground-truth y used by F. Thus 'can be solved with ICL' is, by construction, 'can reproduce the gold answer when demonstrations are greedily chosen to make that answer likely for this exact model.' The zone label is therefore an artifact of the answer-tuned Oracle search rather than a stable query-model learnability trait, and the IRT labels, SELICL decisions, and curriculum ordering all inherit this self-referential target.

full rationale

The central prediction claim, 'We find that the ICL behavior of LLMs is generally predictable even without demonstration information,' is supported by a content-aware IRT model that is trained on Oracle-derived zone labels and evaluated on a held-out 10% test split of queries. That is a genuine supervised prediction, not a fitted parameter renamed as a prediction, so the paper does not reach score 6-10. However, the target labels themselves are self-referential: Eq. (5) builds the Oracle demonstrations by maximizing the same model's likelihood of the gold answer, and Eq. (1) then scores the model on those demonstrations. The paper acknowledges only that the Oracle is a lower bound, which addresses incompleteness but not the upward bias introduced by gold-answer-tuned demonstration selection; this is a partial circularity in the measurement of ZPD. The self-citations (Sachan and Xing 2016; Li et al. 2024, a co-authored paper) are contextual and not load-bearing, so they do not contribute to the score. The unreported threshold τ in Eq. (1) is a transparency issue, but it is not a circularity. Overall, the derivation chain is mostly independent; the score of 4 reflects the self-referential definition of the measured zone, not a collapse of the prediction into its inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The framework adds a conceptual taxonomy (three zones) and a latent-variable model. Its main free parameter is an unreported threshold τ, plus SELICL thresholds tuned on validation. Its key axiom is that Oracle demonstrations built by greedy likelihood maximization faithfully approximate ideal ICL, which the authors themselves qualify as a lower bound. The curriculum assumes that ICL-learnable examples are the best fine-tuning examples, supported only by a post-hoc loss analysis. The MIRT-ICL latent vectors are invented latent variables with no independent measurement.

free parameters (4)
  • Zone threshold τ = not reported
    Defines Z✓, Z✗→✓ and Z✗→✗ in Eq. 1; all zone measurements and downstream labels depend on it, yet no value or sensitivity analysis is given.
  • SELICL thresholds τ1 and τ2 = grid-searched over [0.01, 0.99]; final per-model values not reported
    Selected on the IRT validation set to build the Pareto curves in Figure 5; these are fitted to data, not derived.
  • IRT latent trait dimension H = 32
    Model capacity choice for θ, α, θc, αc; affects how well zones can be predicted.
  • Number of demonstrations k = 8
    Fixed following Li et al. 2023 and Rubin et al. 2021; zone measurements and SELICL cost depend on it.
assumptions (4)
  • domain assumption Oracle demonstrations chosen by greedy likelihood maximization (Eq. 5) provide a good approximation of ideal ICL support for each query.
    This premise anchors all zone labels and all downstream predictions; the paper acknowledges in Limitations that the measured ZPD is only a lower bound and the optimization is practical and finite.
  • domain assumption A query's benefit from ICL can be predicted from its text embedding and model identity alone, without knowing the demonstrations.
    The MIRT-ICL model has no demonstration representation; any unpredictable ICL behavior due to demonstration choice is outside the model, which the paper flags by reporting varying AUC across datasets.
  • ad hoc to paper Queries in Z✗→✓ are more learnable and more valuable for fine-tuning than queries in Z✓ or Z✗→✗.
    Used to justify the curriculum; the evidence in Figure 7 is a correlation between zone labels and training loss on one model and dataset, not a causal demonstration.
  • standard math Standard item response theory monotonicity and conditional-independence assumptions hold for LLM correctness data.
    The IRT model in Eq. 12 is estimated as a standard latent-variable model; these assumptions are standard but untested on LLM outputs.
invented entities (2)
  • ICL skill vector θ^c and ICL discrimination vector α^c
    purpose: Latent factors that shift correctness probability when demonstrations are added (Eq. 10), intended to capture per-model and per-query in-context learnability.
    These are unobserved latent variables introduced by the MIRT-ICL model; there is no external measurement or prediction outside the fitted model that validates them.
  • Zone of Proximal Development as a per-query property of an LLM
    purpose: Used to categorize queries into Z✓, Z✗→✓, Z✗→✗ and to drive selective ICL and curriculum.
    This is a construct defined operationally by the paper's thresholded performance comparison; it is not an independently measured quantity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Investigating the Zone of Proximal Development of Language Models for In-Context Learning." pith.science (2026). https://pith.science/paper/DJJBCHKZ

@misc{pith2026250206990,
  author       = {Pith},
  title        = {Pith review of: Investigating the Zone of Proximal Development of Language Models for In-Context Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DJJBCHKZ}},
  note         = {Machine review of arXiv:2502.06990}
}
read the original abstract

In this paper, we introduce a learning analytics framework to analyze the in-context learning (ICL) behavior of large language models (LLMs) through the lens of the Zone of Proximal Development (ZPD), an established theory in educational psychology. ZPD delineates the space between what a learner is capable of doing unsupported and what the learner cannot do even with support. We adapt this concept to ICL, measuring the ZPD of LLMs based on model performance on individual examples with and without ICL. Furthermore, we propose an item response theory (IRT) model to predict the distribution of zones for LLMs. Our findings reveal a series of intricate and multifaceted behaviors of ICL, providing new insights into understanding and leveraging this technique. Finally, we demonstrate how our framework can enhance LLM in both inference and fine-tuning scenarios: (1) By predicting a model's zone of proximal development, we selectively apply ICL to queries that are most likely to benefit from demonstrations, achieving a better balance between inference cost and performance; (2) We propose a human-like curriculum for fine-tuning, which prioritizes examples within the model's ZPD. The curriculum results in improved performance, and we explain its effectiveness through an analysis of the training dynamics of LLMs.

Figures

Figures reproduced from arXiv: 2502.06990 by the authors.

Figure 1
Figure 1. We conceptualize an LLM’s Zone of Proximal [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. 4.1 Background of IRT IRT is a statistical model that predicts the probabil￾ity of individual respondents correctly answering a set of queries (or items). In this work, we take a collection of LLMs {M1,M2, ...,Mm} as re￾spondents. The basic 1 Parameter Logistic (1PL) IRT is defined as: P(ri,j = 1∣Mi , xj) = σ(θi − dj), (6) where ri,j is the binary correctness label of M’s prediction on xi . σ is the sigmoid function… view at source ↗
Figure 3
Figure 3. Zone distribution of various LLMs on the two [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Increased and decreased accuracy by KATE on GSM8K (left) and EZStance (right). • In-context demonstrations can be harmful. In Section § 3, we divide a query set into three zones according to the model’s performance difference with and without ICL. However, sometimes, I…
Figure 5
Figure 5. Figure 5: Accuracy and inference cost (number of input tokens) of different ICL strategies on the GSM8K dataset. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Comparison between random and our ZPD￾based curriculum on two datasets. tively introduces new buckets. The overall process is outlined in Algorithm 1. Results and Analysis. We compare our algo￾rithm against a random baseline. Although simple, random is the most widely …
Figure 8
Figure 8. Figure 8: Results of SELICL on EZStance. See detailed explanations in [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 19 canonical work pages

  1. [1]

    Ekin Aky \"u rek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. 2023. https://openreview.net/forum?id=0g0X4H8yN4I What learning algorithm is in-context learning? investigations with linear models . In The Eleventh International Conference on Learning Representations

    2023

  2. [2]

    Yoshua Bengio, J \'e r \^o me Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41--48

    2009

  3. [3]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  4. [4]

    Matthew Byrd and Shashank Srivastava. 2022. Predicting difficulty and discrimination of natural language questions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 119--130

    work page 2022

  5. [5]

    Daniel Campos. 2021. Curriculum learning for language modeling. arXiv preprint arXiv:2108.02170

    arXiv 2021

  6. [6]

    Seth Chaiklin et al. 2003. The zone of proximal development in vygotsky’s analysis of learning and instruction. Vygotsky’s educational theory in cultural context, 1(2):39--64

    work page 2003

  7. [7]

    R Philip Chalmers. 2012. mirt: A multidimensional item response theory package for the r environment. Journal of statistical Software, 48:1--29

    work page 2012

  8. [8]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

Show all 55 references
  1. [9]

    Kalyanmoy Deb. 2011. Multi-objective optimisation using evolutionary algorithms: an introduction. In Multi-objective evolutionary optimisation for product design and manufacturing, pages 3--34. Springer

    2011

  2. [10]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  3. [11]

    Simin Fan and Martin Jaggi. 2023. Irreducible curriculum for language model pretraining. arXiv preprint arXiv:2310.15389

    2023 arXiv

  4. [12]

    Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. 2022. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583--30598

    2022

  5. [13]

    Maharshi Gor, Hal Daum \'e III, Tianyi Zhou, and Jordan Boyd-Graber. 2024. Do great minds think alike? investigating human-ai complementarity in question answering with caimira. arXiv preprint arXiv:2410.06524

    2024 arXiv

  6. [14]

    SouYoung Jin, Aruni RoyChowdhury, Huaizu Jiang, Ashish Singh, Aditya Prasad, Deep Chakraborty, and Erik Learned-Miller. 2018. Unsupervised hard example mining from videos for improved object detection. In Proceedings of the European Conference on Computer Vision (ECCV), pages 307--324

    2018

  7. [15]

    Diederik P Kingma. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

    2014 arXiv

  8. [16]

    John Patrick Lalor and Pedro Rodriguez. 2023. py-irt: A scalable item response theory library for python. INFORMS Journal on Computing, 35(1):5--13

    2023

  9. [17]

    Jiaoda Li, Yifan Hou, Mrinmaya Sachan, and Ryan Cotterell. 2024. https://doi.org/10.18653/v1/2024.acl-long.669 What do language models learn in context? the structured task hypothesis. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (...

    2024 doi

  10. [18]

    Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. 2023. Unified demonstration retriever for in-context learning. arXiv preprint arXiv:2305.04320

    2023 arXiv

  11. [19]

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt- 3 ? arXiv preprint arXiv:2101.06804

    2021 arXiv

  12. [20]

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. https://doi.org/10.18653/v1/2022.deelio-1.10 What makes good in-context examples for GPT -3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extr...

    2022 doi

  13. [21]

    Frederic M Lord and Melvin R Novick. 2008. Statistical theories of mental test scores. IAP

    2008

  14. [22]

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. https://doi.org/10.18653/v1/2022.acl-long.556 Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity . In Proceedings of the 60th Annual Meeting of th...

    2022 doi

  15. [23]

    o ren Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt H \

    S \"o ren Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt H \"o ltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. 2022. Prioritized training on points that are learnable, worth learning, and not yet learnt. In In...

    2022

  16. [24]

    Keqin Peng, Liang Ding, Yancheng Yuan, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2024. https://doi.org/10.18653/v1/2024.acl-long.492 Revisiting demonstration selection strategies in in-context learning . In Proceedings of the 62nd Annual Meeting of the Association...

    2024 doi

  17. [25]

    Jean Piaget. 1977. The development of thought: Equilibration of cognitive structures.(Trans A. Rosin). Viking

    1977

  18. [26]

    Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabas Poczos, and Tom M Mitchell. 2019. Competence-based curriculum learning for neural machine translation. arXiv preprint arXiv:1903.09848

    2019 arXiv

  19. [27]

    Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. 2024. tinybenchmarks: evaluating llms with fewer examples. arXiv preprint arXiv:2402.14992

    2024 arXiv

  20. [28]

    Chengwei Qin, Aston Zhang, Anirudh Dagar, and Wenming Ye. 2023. In-context learning with iterative demonstration selection. arXiv preprint arXiv:2310.09881

    2023 arXiv

  21. [29]

    Mark D Reckase. 2006. 18 multidimensional item response theory. Handbook of statistics, 26:607--642

    2006

  22. [30]

    N Reimers. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084

    2019 arXiv

  23. [31]

    Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval , 3(4):333--389

    2009

  24. [32]

    Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2021. Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:2112.08633

    2021 arXiv

  25. [33]

    Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. https://doi.org/10.18653/v1/2022.naacl-main.191 Learning to retrieve prompts for in-context learning . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: ...

    2022 doi

  26. [34]

    Mrinmaya Sachan and Eric Xing. 2016. Easy questions first? a case study on curriculum learning for question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 453--463

    2016

  27. [35]

    Darcy A Santor and James O Ramsay. 1998. Progress in the technology of measurement: Applications of item response models. Psychological assessment, 10(4):345

    1998

  28. [36]

    Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. 2016. Training region-based object detectors with online hard example mining. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 761--769

    2016

  29. [37]

    less is more

    Valentin I Spitkovsky, Hiyan Alshawi, and Dan Jurafsky. 2010. From baby steps to leapfrog: How “less is more” in unsupervised dependency parsing. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Lingu...

    2010

  30. [38]

    Pragya Srivastava, Satvik Golechha, Amit Deshpande, and Amit Sharma. 2024. https://doi.org/10.18653/v1/2024.acl-long.300 NICE : To optimize in-context examples or not? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa...

    2024 doi

  31. [39]

    Smith, and Yejin Choi

    Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.746 Dataset cartography: Mapping and diagnosing datasets with training dynamics . In Proceedings of the 2020 C...

    2020 doi

  32. [40]

    Yi Tay, Shuohang Wang, Luu Anh Tuan, Jie Fu, Minh C Phan, Xingdi Yuan, Jinfeng Rao, Siu Cheung Hui, and Aston Zhang. 2019. Simple and effective curriculum pointer-generator networks for reading comprehension over long narratives. arXiv preprint arXiv:1905.10847

    2019 arXiv

  33. [41]

    Roland G Tharp and Ronald Gallimore. 1991. Rousing minds to life: Teaching, learning, and schooling in social context. Cambridge University Press

    1991

  34. [42]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  35. [43]

    Lev Semenovich Vygotsky. 1978. Mind in society: The development of higher psychological processes, volume 86. Harvard university press

    1978

  36. [44]

    Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. 2024. Large language models are latent variable models: explaining and finding good demonstrations for in-context learning. Red Hook, NY, USA. Curran Associates Inc

    2024

  37. [45]

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682

    2022 arXiv

  38. [46]

    Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. 2023. Larger language models do in-context learning differently, 2023. URL https://arxiv. org/abs/2303.03846

    2023 arXiv

  39. [47]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical met...

    2020

  40. [48]

    Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2022. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations (ICLR)

    2022

  41. [49]

    Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. Compositional exemplars for in-context learning. In International Conference on Machine Learning, pages 39818--39833. PMLR

    2023

  42. [50]

    Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, et al. 2024. A careful examination of large language model performance on grade school arithmetic. arXiv preprint arXiv:2405.00332

    2024 arXiv

  43. [51]

    Chenye Zhao and Cornelia Caragea. 2023. Ez-stance: A large dataset for zero-shot stance detection. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 897--911

    2023

  44. [52]

    Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021 a . https://proceedings.mlr.press/v139/zhao21c.html Calibrate before use: Improving few-shot performance of language models . In Proceedings of the 38th International Conference on Machine Learning, volume 1...

    2021

  45. [53]

    Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021 b . Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning, pages 12697--12706. PMLR

    2021

  46. [54]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  47. [55]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.