Pith. sign in

REVIEW 4 major objections 6 minor 47 references

In-Context Learning (and Unlearning) of Length Biases

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Large language models shift their predictions toward the length distribution of the demonstrations in the prompt, and balanced demonstrations can cancel a length bias learned during finetuning.

desk verdict Useful empirical sweep with a central claim that outruns its design: the tail-sampling manipulation confounds length with content, so 'length bias' is not yet isolated. read the letter →

arxiv 2502.06653 v1 pith:CEOBWHYV submitted 2025-02-10 cs.CL

Stephanie Schoch , Yangfeng Ji This is my paper · ORCID
classification cs.CL
keywords in-contextlearninglengthbiasstatisticalpromptdemonstrationsdebiasinglargelanguagemodelsfew-shotannotationartifacts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether giving a large language model a few input-output example pairs in its prompt, a process called in-context learning, lets it absorb statistical biases about text length just as finetuning does. Across seven binary classification datasets and five model families, the authors show the answer is yes: when one class is demonstrated with short inputs and the other with long inputs, predictions on validation examples track the demonstrated length, with accuracy on matching-length examples above and opposite-length examples below. They find the bias grows with more demonstrations, appears even when the average lengths differ by only a few tokens, and does not scale consistently with model size. They then show that randomly sampled demonstrations can erase a length bias a model acquired during finetuning, with no parameter updates. The practical consequence is that example selection in a prompt is not only an accuracy knob but also a bias knob.

What carries the argument

The load-bearing object is a controlled tail-sampling intervention on the demonstration set. For each class in a binary task, the top $k/2$ longest or shortest inputs are selected, so the two classes' demonstrated input lengths are maximally separated; this creates a length shortcut in the context window. Its effect is read out with six length bins over the validation set, comparing per-class accuracy on short versus long bins to detect whether predictions moved with demonstrated length. The same tail sampling is applied to the 200 LoRA finetuning examples for a direct comparison, and debiasing flips or randomizes the tail assignment in the demonstrations.

What would settle it

Run the same tail-sampling experiment with a matched-length control in which only length changes, for example by inserting task-neutral filler tokens into otherwise identical inputs; if the bin-dependent accuracy gap disappears when vocabulary and syntax are held fixed, the reported effect is not purely about length.

Watch

Extended reading notes

Core claim

The paper's central claim, stated in Section 4.2, is that models can pick up length biases in-context. The experimental recipe introduces a worst-case length bias by sampling demonstrations from the top of one class's input-length distribution and the bottom of the other, then bins the validation set into six length bins. The observed pattern matches the finetuning comparison: per-class accuracy is higher on validation examples whose length resembles the demonstrations for that class and lower on examples of the opposite length. The paper's third contribution is that in-context learning can debias a model that contains existing length biases, with length-balanced or randomly sampled demonstrations counteracting a finetuned bias. A notable finding is that models then follow the in-context length signal even when it contradicts what they saw during finetuning.

Load-bearing premise

The tail-sampling procedure is assumed to vary only input length, so if the longest and shortest examples of a class also differ in vocabulary, syntax, or other heuristics, the bin-dependent accuracy could come from those confounds rather than length.

Editorial extensions

If this is right

  • A prompt's demonstration set can silently inject a length shortcut: with 16 examples, models favor the class whose demonstrated inputs match the test input's length.
  • The effect typically appears around 8 demonstrations and strengthens with more, so the current push toward longer contexts increases the risk of learning statistical biases in-context.
  • Length bias is learnable from a very small signal, since the authors observe it when average class lengths differ by roughly three tokens on HANS.
  • Randomly sampled or length-balanced demonstrations removed a finetuned length bias on the tested datasets without parameter updates, suggesting balanced sampling as a practical debiasing intervention.
  • Model size does not consistently predict bias susceptibility, so the phenomenon is not confined to the largest models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One implication the authors leave implicit is a deployment rule: audit the length distribution of any pool of few-shot examples, because random sampling only cancels bias when the pool itself is balanced.
  • The finding suggests benchmark comparisons of in-context learning methods should control for the length distribution of demonstrations, since otherwise a method's reported gain could be partly a length-shortcut effect rather than better task learning.
  • If the mechanism is distributional rather than lexical, the same tail-sampling protocol should reveal analogous shortcuts for other continuous features such as word frequency, sentence complexity, or formality, which would be a direct testable extension.
  • A further testable extension is to scale the protocol to very long contexts and see whether the bias saturates or keeps growing with demonstration count.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper empirically investigates whether large language models learn length biases from demonstrations during in-context learning. It introduces a length bias by sampling the longest examples of one class and the shortest examples of the other class as demonstrations, then measures per-length-bin class accuracy on validation sets. The authors report that models up to 8B parameters exhibit higher accuracy for validation examples whose length matches the length of the demonstrations for that class, across seven binary classification datasets. They further analyze the effect of model size, number of demonstrations, and the magnitude of class-length difference, and they propose using in-context demonstrations with random or oppositely biased sampling to mitigate a length bias acquired through finetuning. The central claims are that models can learn length biases in-context and that in-context learning can debias finetuned models without parameter updates.

Significance. If the central claim holds, the paper addresses a genuine gap: statistical biases such as input length have been studied in finetuned models but are underexplored in in-context learning. The study has notable strengths: it covers multiple model families (Llama, Mistral, OPT, GPT-Neo), several datasets, and transparently specifies the sampling and binning procedure. The potential practical implication that random or balanced demonstration sampling can counteract a finetuned length bias is interesting and actionable. However, the empirical foundation of the claim depends on the assumption that the tail-sampling procedure varies only length, which is not established. Because the main contribution is an empirical attribution of accuracy differences to length, this confound is load-bearing; the paper also lacks statistical tests and uses only four trials per condition.

major comments (4)
  1. [§4.1, Figures 3–6] The tail-sampling manipulation selects the longest examples of one class and the shortest of the other based on word count. In natural-language datasets, the longest and shortest examples of a class systematically differ in vocabulary, syntax, and content beyond their length: for HANS, short hypotheses tend to be simple noun-verb phrases while long ones contain relative clauses or conjunctions; for SST-2, short reviews often contain overt sentiment words while longer reviews are more nuanced; for PAWS-XEN, longer sentence pairs may exhibit different scrambling patterns. The observed bin-dependent accuracy therefore could reflect lexical or syntactic correlates of length rather than a learned length bias. The paper provides no matched-pair control (for example, the same content presented with padding or with length altered by paraphrase) and no covariate analysis. Since the claim in Section 4.2 that 'models can pick up length biases in-context' rests entirely on this manipulation, a control or alternative design is necessary to support the attribution.
  2. [§6, Figures 10–12] The debiasing claim that random in-context demonstrations can 'unlearn' a finetuned length bias is measured with the same per-bin class-accuracy comparison that defines the bias. If the finetuned bias itself is confounded with content correlates, as described above, then the reduction under random sampling may reflect a shift in those correlates rather than a genuine manipulation of length information. The opposite-tail intervention is more informative because it shows that the model tracks the in-context length structure even when it contradicts the finetuned structure, but the same confound applies to the interpretation of 'length' in the demonstrations. The conclusion in Section 6 that 'balanced sampling is critical' should be supported by an analysis that isolates length from content.
  3. [§3, §5, Figures 7–9] Each result is the mean of four trials and no statistical significance tests, confidence intervals, or effect sizes are reported. Many of the supporting figures use very small y-axis ranges (for example, Figure 7 and Figure 8 use a range of about 0.04), and the claims that 'bias increases with increased numbers of examples' and that a bias can be learned from 'as few as 3 tokens' of average length difference would be more persuasive with a significance test across the trials or a bootstrapped interval. The absence of such tests leaves the strength of several factor analyses unclear.
  4. [§5.3, Table 7] The length-difference analysis varies the sampling percentage from the tails, but the measured 'difference in average demonstration length' is reported only in aggregate and only for validation/training sets, not for the specific demonstration sets used in each condition. As a result, the relationship claimed between demonstration length difference and the observed bias is not directly established by the reported numbers; reporting the average length of the actual k demonstrations in each condition would make the comparison concrete.
minor comments (6)
  1. [§2] There is a duplicated citation 'Fei et al., 2023' in the second paragraph, and the text contains a typo, 'simlarities', in the introduction.
  2. [Appendix A.4–A.8] Several appendix figures contain unlabeled or truncated axes, and the y-axis scale is not always visible; providing explicit numeric axes and consistent bin labels would improve reproducibility and readability.
  3. [Table 5] The QNLI 'Random' row contains the value '42.5.00', which appears to be a formatting error; the table should be checked for other typos.
  4. [Appendix A.5] The appendix text refers to 'MPRC' in the figure caption for the OPT results, while the dataset is named MRPC elsewhere; the caption should be corrected.
  5. [Section 9] The limitation statement 'We believe our results scale to larger models' is an assertion without supporting evidence; it should either be removed or explicitly framed as a conjecture rather than a belief.
  6. [General] No code, prompts, or experiment configuration details are provided in a public repository; releasing these would strengthen reproducibility, especially given the small number of trials.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports controlled experiments with no fitted parameters, no self-citation chain, and no claim that is true by construction.

full rationale

This paper is an empirical study; its central claims, that models can pick up length biases in-context and that in-context learning can debias finetuned models, are supported by controlled comparisons between tail-sampled and randomly sampled demonstrations. There are no fitted parameters, no derived prediction that reduces to an input, and no self-citations that carry argumentative weight. The tail-sampling procedure in Section 4.1 manipulates the length distribution of demonstrations, and the binning procedure defines the measurement; the observed bin-dependent accuracy is an empirical outcome that could have failed to appear, so the claim is not true by construction. The Section 6 'unlearning' claim uses the same bin comparison to measure the finetuned bias and the intervention effect; this is an operational definition of the bias being targeted, not an assumption that the effect exists. The possible confound between input length and vocabulary or syntax in tail sampling is a validity concern, not a circularity concern, and per the review rules it does not raise the circularity score.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper introduces no fitted parameters in a model and no invented entities. Its central claim rests on experimental control assumptions: tail sampling isolates length, four-trial averaging controls ordering, bin comparisons detect bias, and small-model results generalize. The unsupported scaling claim is recorded as an ad hoc assumption.

free parameters (1)
  • number of validation length bins b = 6
    Chosen by hand to define short versus long validation groups; the bias metric compares bin 1 against bin 6, so the choice affects the reported effect.
assumptions (4)
  • domain assumption Tail sampling of demonstrations by length isolates length as the only systematically varied property between classes.
    Section 4.1 samples from tails of length distributions, but tail examples may also differ in vocabulary, syntax, or dataset-specific heuristics, especially in HANS.
  • domain assumption Averaging 4 trials without randomizing demonstration order minimizes ordering effects.
    Section 3 states 4-trial means; if order is fixed across trials, averaging identical orders does not address order sensitivity.
  • domain assumption Binning validation inputs into 6 equal-frequency length bins and comparing accuracy on bin 1 versus bin 6 is a valid detector of learned length bias.
    This operational definition is used throughout Sections 4 to 6 to conclude that a bias exists or was reduced.
  • ad hoc to paper Results on models up to 8B parameters generalize to larger models.
    The Limitations section asserts 'We believe our results scale to larger models' without providing evidence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of In-Context Learning (and Unlearning) of Length Biases." pith.science (2026). https://pith.science/paper/CEOBWHYV

@misc{pith2026250206653,
  author       = {Pith},
  title        = {Pith review of: In-Context Learning (and Unlearning) of Length Biases},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CEOBWHYV}},
  note         = {Machine review of arXiv:2502.06653}
}
read the original abstract

Large language models have demonstrated strong capabilities to learn in-context, where exemplar input-output pairings are appended to the prompt for demonstration. However, existing work has demonstrated the ability of models to learn lexical and label biases in-context, which negatively impacts both performance and robustness of models. The impact of other statistical data biases remains under-explored, which this work aims to address. We specifically investigate the impact of length biases on in-context learning. We demonstrate that models do learn length biases in the context window for their predictions, and further empirically analyze the factors that modulate the level of bias exhibited by the model. In addition, we show that learning length information in-context can be used to counter the length bias that has been encoded in models (e.g., via fine-tuning). This reveals the power of in-context learning in debiasing model prediction behaviors without the need for costly parameter updates.

Figures

Figures reproduced from arXiv: 2502.06653 by the authors.

Figure 1
Figure 1. An illustration of our experiment setup and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An overview of in-context learning using [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. In-context learning validation performance across different models on the Hans dataset. For each graph, [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (59 more)
Figure 4
Figure 4. Figure 4: Finetuning validation performance across different models on the Hans dataset. For each graph, [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: In-context learning validation performance across different models on the PAWS-X [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Finetuning validation performance across different models on the PAWS-X [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]
Figure 7
Figure 7. Figure 7: Validation performance across different num [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 8
Figure 8. Figure 8: Validation performance of Llama3 (8B) across different numbers of demonstrations on the PAWS-XEN dataset. In this figure, in-context exam￾ples from y2 were sampled from long instances, and in-context examples from y1 were sampled from short instances. Each subfigure sh…
Figure 10
Figure 10. Figure 10: HANS validation set performance on a finetuned Llama 3 (8B) model exhibiting a length bias (see [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]
Figure 11
Figure 11. Figure 11: PAWS-XEN validation set performance on a finetuned Llama 3 (8B) model exhibiting a length bias (see Figure 11a for finetuning performance prior to intervention). Figure 11b and Figure 11c (respectively) show results on two debiasing conditions: ICL demonstrations (k =…
Figure 12
Figure 12. Figure 12: WNLI validation set performance on a finetuned Llama 3 (8B) model exhibiting a length bias (see [PITH_FULL_IMAGE:figures/full_fig_p007_12.png]
Figure 13
Figure 13. Figure 13: ICL performance of Llama3 8B, Llama2 7B, Mistral 7B, OPT 6.7B, and GPT Neo 2.7B (from left to [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 14
Figure 14. Figure 14: Finetuning performance of Llama3 8B, Llama2 7B, Mistral 7B, OPT 6.7B, and GPT Neo 2.7B (from left [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]
Figure 15
Figure 15. Figure 15: ICL performance of Llama3 8B, Llama2 7B, Mistral 7B, OPT 6.7B, and GPT Neo 2.7B (from left to [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]
Figure 16
Figure 16. Figure 16: Finetuning performance of Llama3 8B, Llama2 7B, Mistral 7B, OPT 6.7B, and GPT Neo 2.7B (from left [PITH_FULL_IMAGE:figures/full_fig_p023_16.png]
Figure 17
Figure 17. Figure 17: ICL performance of Llama3 8B, Llama2 7B, Mistral 7B, OPT 6.7B, and GPT Neo 2.7B (from left to [PITH_FULL_IMAGE:figures/full_fig_p024_17.png]
Figure 18
Figure 18. Figure 18: Finetuning performance of Llama3 8B, Llama2 7B, Mistral 7B, OPT 6.7B, and GPT Neo 2.7B (from left [PITH_FULL_IMAGE:figures/full_fig_p025_18.png]
Figure 22
Figure 22. Figure 22: QNLI dataset (OPT) (a) y1 0.04 0.02 (b) 0.00 y2 0.02 0.04 0.04 0.02 0.00 0.02 0.04 Bin 0 Bin 1 Bin 2 Bin 3 Bin 4 Bin 5 [PITH_FULL_IMAGE:figures/full_fig_p026_22.png]
Figure 19
Figure 19. Figure 19: Hans dataset (OPT) (a) y1 0.04 0.02 (b) 0.00 y2 0.02 0.04 0.04 0.02 0.00 0.02 0.04 Bin 0 Bin 1 Bin 2 Bin 3 Bin 4 Bin 5 [PITH_FULL_IMAGE:figures/full_fig_p026_19.png]
Figure 20
Figure 20. Figure 20: PAWS-XEN dataset (OPT) (a) y1 0.04 0.02 (b) 0.00 y2 0.02 0.04 0.04 0.02 0.00 0.02 0.04 Bin 0 Bin 1 Bin 2 Bin 3 Bin 4 Bin 5 [PITH_FULL_IMAGE:figures/full_fig_p026_20.png]
Figure 21
Figure 21. Figure 21: RTE dataset (OPT) (a) y1 0.04 0.02 (b) 0.00 y2 0.02 0.04 0.04 0.02 0.00 0.02 0.04 Bin 0 Bin 1 Bin 2 Bin 3 Bin 4 Bin 5 [PITH_FULL_IMAGE:figures/full_fig_p026_21.png]
Figure 29
Figure 29. Figure 29: QNLI dataset (Llama 3 8B) (a) y1 0.04 0.02 (b) 0.00 y2 0.02 0.04 0.04 0.02 0.00 0.02 0.04 Bin 0 Bin 1 Bin 2 Bin 3 Bin 4 Bin 5 [PITH_FULL_IMAGE:figures/full_fig_p027_29.png]
Figure 26
Figure 26. Figure 26: Hans dataset (Llama 3 8B) (a) y1 0.04 0.02 (b) 0.00 y2 0.02 0.04 0.04 0.02 0.00 0.02 0.04 Bin 0 Bin 1 Bin 2 Bin 3 Bin 4 Bin 5 [PITH_FULL_IMAGE:figures/full_fig_p027_26.png]
Figure 27
Figure 27. Figure 27: PAWS-XEN dataset (Llama 3 8B) (a) y1 0.04 0.02 (b) 0.00 y2 0.02 0.04 0.04 0.02 0.00 0.02 0.04 Bin 0 Bin 1 Bin 2 Bin 3 Bin 4 Bin 5 [PITH_FULL_IMAGE:figures/full_fig_p027_27.png]
Figure 28
Figure 28. Figure 28: RTE dataset (Llama 3 8B) (a) y1 0.04 0.02 (b) 0.00 y2 0.02 0.04 0.04 0.02 0.00 0.02 0.04 Bin 0 Bin 1 Bin 2 Bin 3 Bin 4 Bin 5 [PITH_FULL_IMAGE:figures/full_fig_p027_28.png]
Figure 33
Figure 33. Figure 33: Hans dataset (GPT Neo 2.7B) (a) y1 0.04 0.02 (b) 0.00 y2 0.02 0.04 0.04 0.02 0.00 0.02 0.04 Bin 0 Bin 1 Bin 2 Bin 3 Bin 4 Bin 5 [PITH_FULL_IMAGE:figures/full_fig_p028_33.png]
Figure 34
Figure 34. Figure 34: PAWS-XEN dataset (GPT Neo 2.7B) (a) y1 0.04 0.02 (b) 0.00 y2 0.02 0.04 0.04 0.02 0.00 0.02 0.04 Bin 0 Bin 1 Bin 2 Bin 3 Bin 4 Bin 5 [PITH_FULL_IMAGE:figures/full_fig_p028_34.png]
Figure 39
Figure 39. Figure 39: SST-2 dataset (GPT Neo 2.7B) [PITH_FULL_IMAGE:figures/full_fig_p028_39.png]
Figure 42
Figure 42. Figure 42: RTE dataset (Llama 3 8B) (a) y1 0.04 0.02 (b) 0.00 y2 0.02 0.04 0.04 0.02 0.00 0.02 0.04 Bin 0 Bin 1 Bin 2 Bin 3 Bin 4 Bin 5 [PITH_FULL_IMAGE:figures/full_fig_p029_42.png]
Figure 40
Figure 40. Figure 40: Hans dataset (Llama 3 8B) (a) y1 0.04 0.02 (b) 0.00 y2 0.02 0.04 0.04 0.02 0.00 0.02 0.04 Bin 0 Bin 1 Bin 2 Bin 3 Bin 4 Bin 5 [PITH_FULL_IMAGE:figures/full_fig_p029_40.png]
Figure 41
Figure 41. Figure 41: PAWS-XEN dataset (Llama 3 8B) (a) y1 0.04 0.02 (b) 0.00 y2 0.02 0.04 0.04 0.02 0.00 0.02 0.04 Bin 0 Bin 1 Bin 2 Bin 3 Bin 4 Bin 5 [PITH_FULL_IMAGE:figures/full_fig_p029_41.png]
Figure 45
Figure 45. Figure 45: MPRC dataset (Llama 3 8B) (a) y1 0.04 0.02 (b) 0.00 y2 0.02 0.04 0.04 0.02 0.00 0.02 0.04 Bin 0 Bin 1 Bin 2 Bin 3 Bin 4 Bin 5 [PITH_FULL_IMAGE:figures/full_fig_p030_45.png]
Figure 46
Figure 46. Figure 46: SST-2 dataset (Llama 3 8B) (a) y1 0.04 0.02 (b) 0.00 y2 0.02 0.04 0.04 0.02 0.00 0.02 0.04 Bin 0 Bin 1 Bin 2 Bin 3 Bin 4 Bin 5 [PITH_FULL_IMAGE:figures/full_fig_p030_46.png]
Figure 50
Figure 50. Figure 50: QNLI dataset (GPT Neo 2.7B) [PITH_FULL_IMAGE:figures/full_fig_p030_50.png]
Figure 51
Figure 51. Figure 51: WNLI dataset (GPT Neo 2.7B) (a) y1 0.04 0.02 (b) 0.00 y2 0.02 0.04 0.04 0.02 0.00 0.02 0.04 Bin 0 Bin 1 Bin 2 Bin 3 Bin 4 Bin 5 [PITH_FULL_IMAGE:figures/full_fig_p031_51.png]
Figure 52
Figure 52. Figure 52: MPRC dataset (GPT Neo 2.7B) (a) y1 0.04 0.02 (b) 0.00 y2 0.02 0.04 0.04 0.02 0.00 0.02 0.04 Bin 0 Bin 1 Bin 2 Bin 3 Bin 4 Bin 5 [PITH_FULL_IMAGE:figures/full_fig_p031_52.png]
Figure 53
Figure 53. Figure 53: SST-2 dataset (GPT Neo 2.7B) [PITH_FULL_IMAGE:figures/full_fig_p031_53.png]
Figure 54
Figure 54. Figure 54: RTE (Llama 3 8B) (a) Finetuning: y1 (Blue) short demon￾strations, y2 (Orange) long demonstra￾tions. (b) Intervention: y1 (Blue) long demon￾strations, y2 (Orange) short demonstra￾tions. (c) Intervention: y1 (Blue) and y2 (Or￾ange) demonstrations randomly sam￾pled [PIT…
Figure 55
Figure 55. Figure 55: QNLI (Llama 3 8B) (a) Finetuning: y1 (Blue) short demon￾strations, y2 (Orange) long demonstra￾tions. (b) Intervention: y1 (Blue) long demon￾strations, y2 (Orange) short demonstra￾tions. (c) Intervention: y1 (Blue) and y2 (Or￾ange) demonstrations randomly sam￾pled [PI…
Figure 56
Figure 56. Figure 56: MRPC (Llama 3 8B) (a) Finetuning: y1 (Blue) short demon￾strations, y2 (Orange) long demonstra￾tions. (b) Intervention: y1 (Blue) long demon￾strations, y2 (Orange) short demonstra￾tions. (c) Intervention: y1 (Blue) and y2 (Or￾ange) demonstrations randomly sam￾pled [PI…
Figure 57
Figure 57. Figure 57: SST-2 (Llama 3 8B) [PITH_FULL_IMAGE:figures/full_fig_p033_57.png]
Figure 58
Figure 58. Figure 58: Hans (Llama 3 8B) (a) Finetuning: y1 (Blue) long demon￾strations, y2 (Orange) short demonstra￾tions. (b) Intervention: y1 (Blue) short demon￾strations, y2 (Orange) long demonstra￾tions. (c) Intervention: y1 (Blue) and y2 (Or￾ange) demonstrations randomly sam￾pled [PI…
Figure 59
Figure 59. Figure 59: PAWS-XEN (Llama 3 8B) (a) Finetuning: y1 (Blue) long demon￾strations, y2 (Orange) short demonstra￾tions. (b) Intervention: y1 (Blue) short demon￾strations, y2 (Orange) long demonstra￾tions. (c) Intervention: y1 (Blue) and y2 (Or￾ange) demonstrations randomly sam￾pled …
Figure 60
Figure 60. Figure 60: RTE (Llama 3 8B) (a) Finetuning: y1 (Blue) long demon￾strations, y2 (Orange) short demonstra￾tions. (b) Intervention: y1 (Blue) short demon￾strations, y2 (Orange) long demonstra￾tions. (c) Intervention: y1 (Blue) and y2 (Or￾ange) demonstrations randomly sam￾pled [PIT…
Figure 61
Figure 61. Figure 61: QNLI (Llama 3 8B) [PITH_FULL_IMAGE:figures/full_fig_p034_61.png]
Figure 62
Figure 62. Figure 62: WNLI (Llama 3 8B) (a) Finetuning: y1 (Blue) long demon￾strations, y2 (Orange) short demonstra￾tions. (b) Intervention: y1 (Blue) short demon￾strations, y2 (Orange) long demonstra￾tions. (c) Intervention: y1 (Blue) and y2 (Or￾ange) demonstrations randomly sam￾pled [PI…
Figure 63
Figure 63. Figure 63: MRPC (Llama 3 8B) (a) Finetuning: y1 (Blue) long demon￾strations, y2 (Orange) short demonstra￾tions. (b) Intervention: y1 (Blue) short demon￾strations, y2 (Orange) long demonstra￾tions. (c) Intervention: y1 (Blue) and y2 (Or￾ange) demonstrations randomly sam￾pled [PI…
Figure 64
Figure 64. Figure 64: SST-2 (Llama 3 8B) (a) Finetuning: y1 (Blue) short demon￾strations, y2 (Orange) long demonstra￾tions. (b) Intervention: y1 (Blue) long demon￾strations, y2 (Orange) short demonstra￾tions. (c) Intervention: y1 (Blue) and y2 (Or￾ange) demonstrations randomly sam￾pled [P…
Figure 65
Figure 65. Figure 65: Hans (GPT Neo 2.7B) [PITH_FULL_IMAGE:figures/full_fig_p035_65.png]
Figure 66
Figure 66. Figure 66: PAWS-XEN (GPT Neo 2.7B) (a) Finetuning: y1 (Blue) short demon￾strations, y2 (Orange) long demonstra￾tions. (b) Intervention: y1 (Blue) long demon￾strations, y2 (Orange) short demonstra￾tions. (c) Intervention: y1 (Blue) and y2 (Or￾ange) demonstrations randomly sam￾ple…
Figure 67
Figure 67. Figure 67: RTE (GPT Neo 2.7B) (a) Finetuning: y1 (Blue) short demon￾strations, y2 (Orange) long demonstra￾tions. (b) Intervention: y1 (Blue) long demon￾strations, y2 (Orange) short demonstra￾tions. (c) Intervention: y1 (Blue) and y2 (Or￾ange) demonstrations randomly sam￾pled [P…
Figure 68
Figure 68. Figure 68: QNLI (GPT Neo 2.7B) (a) Finetuning: y1 (Blue) short demon￾strations, y2 (Orange) long demonstra￾tions. (b) Intervention: y1 (Blue) long demon￾strations, y2 (Orange) short demonstra￾tions. (c) Intervention: y1 (Blue) and y2 (Or￾ange) demonstrations randomly sam￾pled […
Figure 69
Figure 69. Figure 69: WNLI (GPT Neo 2.7B) [PITH_FULL_IMAGE:figures/full_fig_p036_69.png]
Figure 70
Figure 70. Figure 70: MRPC (GPT Neo 2.7B) (a) Finetuning: y1 (Blue) short demon￾strations, y2 (Orange) long demonstra￾tions. (b) Intervention: y1 (Blue) long demon￾strations, y2 (Orange) short demonstra￾tions. (c) Intervention: y1 (Blue) and y2 (Or￾ange) demonstrations randomly sam￾pled […
Figure 71
Figure 71. Figure 71: SST-2 (GPT Neo 2.7B) (a) Finetuning: y1 (Blue) long demon￾strations, y2 (Orange) short demonstra￾tions. (b) Intervention: y1 (Blue) short demon￾strations, y2 (Orange) long demonstra￾tions. (c) Intervention: y1 (Blue) and y2 (Or￾ange) demonstrations randomly sam￾pled …
Figure 72
Figure 72. Figure 72: Hans (GPT Neo 2.7B) (a) Finetuning: y1 (Blue) long demon￾strations, y2 (Orange) short demonstra￾tions. (b) Intervention: y1 (Blue) short demon￾strations, y2 (Orange) long demonstra￾tions. (c) Intervention: y1 (Blue) and y2 (Or￾ange) demonstrations randomly sam￾pled […
Figure 73
Figure 73. Figure 73: PAWS-XEN (GPT Neo 2.7B) [PITH_FULL_IMAGE:figures/full_fig_p037_73.png]
Figure 74
Figure 74. Figure 74: RTE (GPT Neo 2.7B) (a) Finetuning: y1 (Blue) long demon￾strations, y2 (Orange) short demonstra￾tions. (b) Intervention: y1 (Blue) short demon￾strations, y2 (Orange) long demonstra￾tions. (c) Intervention: y1 (Blue) and y2 (Or￾ange) demonstrations randomly sam￾pled [P…
Figure 75
Figure 75. Figure 75: QNLI (GPT Neo 2.7B) (a) Finetuning: y1 (Blue) long demon￾strations, y2 (Orange) short demonstra￾tions. (b) Intervention: y1 (Blue) short demon￾strations, y2 (Orange) long demonstra￾tions. (c) Intervention: y1 (Blue) and y2 (Or￾ange) demonstrations randomly sam￾pled […
Figure 76
Figure 76. Figure 76: WNLI (GPT Neo 2.7B) (a) Finetuning: y1 (Blue) long demon￾strations, y2 (Orange) short demonstra￾tions. (b) Intervention: y1 (Blue) short demon￾strations, y2 (Orange) long demonstra￾tions. (c) Intervention: y1 (Blue) and y2 (Or￾ange) demonstrations randomly sam￾pled […
Figure 77
Figure 77. Figure 77: MRPC (GPT Neo 2.7B) [PITH_FULL_IMAGE:figures/full_fig_p038_77.png]
Figure 78
Figure 78. Figure 78: SST-2 (GPT Neo 2.7B) [PITH_FULL_IMAGE:figures/full_fig_p039_78.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 6 canonical work pages

  1. [1]

    Ameen Ali, Lior Wolf, and Ivan Titov. 2024. Mitigating copy bias in in-context learning through neuron pruning. arXiv preprint arXiv:2410.01288

    arXiv 2024

  2. [2]

    Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second PASCAL recognising textual entailment challenge

    2006

  3. [3]

    Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009. The fifth PASCAL recognizing textual entailment challenge

    2009

  4. [4]

    Steven Bird and Edward Loper. 2004. https://aclanthology.org/P04-3031 NLTK : The natural language toolkit . In Proceedings of the ACL Interactive Poster and Demonstration Sessions , pages 214--217, Barcelona, Spain. Association for Computational Linguistics

    2004

  5. [5]

    Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. https://doi.org/10.5281/zenodo.5297715 GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow

    doi:10.5281/zenodo.5297715 2021

  6. [6]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  7. [7]

    Tianle Cai, Kaixuan Huang, Jason D Lee, and Mengdi Wang. 2023. Scaling in-context demonstrations with structured attention. arXiv preprint arXiv:2307.02690

    arXiv 2023

  8. [8]

    Zheng Cai, Lifu Tu, and Kevin Gimpel. 2017. https://doi.org/10.18653/v1/P17-2097 Pay attention to the ending:strong neural baselines for the ROC story cloze task . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 616--622, Vancouver, Canada. Association for Computational Linguistics

    work page doi:10.18653/v1/p17-2097 2017

Show all 47 references
  1. [9]

    Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, and He He. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.12 On the relation between sensitivity and accuracy in in-context learning . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 155--...

    2023 doi

  2. [10]

    Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pages 177--190. Springer

    2006

  3. [11]

    William Dolan, Chris Quirk, Chris Brockett, and Bill Dolan. 2004. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources

    2004

  4. [12]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

    2024 arXiv

  5. [13]

    Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselut. 2023. https://doi.org/10.18653/v1/2023.acl-long.783 Mitigating label biases for in-context learning . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages...

    2023 doi

  6. [14]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2023 doi

  7. [15]

    Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. https://doi.org/10.18653/v1/2021.acl-long.295 Making pre-trained language models better few-shot learners . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint...

    2021 doi

  8. [16]

    Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1--9. Association for Computational Linguistics

    2007

  9. [17]

    Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. https://doi.org/10.18653/v1/N18-2017 Annotation artifacts in natural language inference data . In Proceedings of the 2018 Conference of the North A merican Chapter of the As...

    2018 doi

  10. [18]

    Yaru Hao, Yutao Sun, Li Dong, Zhixiong Han, Yuxian Gu, and Furu Wei. 2022. Structured prompting: Scaling in-context learning to 1,000 examples. arXiv preprint arXiv:2212.06713

    2022 arXiv

  11. [19]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lo RA : Low-rank adaptation of large language models . In International Conference on Learning Representations

    2022

  12. [20]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  13. [21]

    Hector J Levesque, Ernest Davis, and Leora Morgenstern. 2011. The W inograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning , volume 46, page 47

    2011

  14. [22]

    Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario S a s ko, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh,...

    2021 arXiv

  15. [23]

    Lvxue Li, Jiaqi Chen, Xinyu Lu, Yaojie Lu, Hongyu Lin, Shuheng Zhou, Huijia Zhu, Weiqiang Wang, Zhongyi Liu, Xianpei Han, and Le Sun. 2024. https://doi.org/10.18653/v1/2024.findings-acl.430 Debiasing in-context learning by instructing LLM s how to follow demonstrations . In Fi...

    2024 doi

  16. [24]

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. https://doi.org/10.18653/v1/2022.deelio-1.10 What makes good in-context examples for GPT -3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extr...

    2022 doi

  17. [25]

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. https://doi.org/10.18653/v1/2022.acl-long.556 Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity . In Proceedings of the 60th Annual Meeting of th...

    2022 doi

  18. [26]

    Thomas McCoy, Ellie Pavlick, and Tal Linzen

    R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019 a . https://arxiv.org/abs/1902.01007 Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference . CoRR, abs/1902.01007

    2019 arXiv

  19. [27]

    Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019 b . https://doi.org/10.18653/v1/P19-1334 Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, page...

    2019 doi

  20. [28]

    Aristides Milios, Siva Reddy, and Dzmitry Bahdanau. 2023. https://doi.org/10.18653/v1/2023.genbench-1.14 In-context learning for text classification with many labels . In Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP, pages 173--184, Singapor...

    2023 doi

  21. [29]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.759 Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on ...

    2022 doi

  22. [30]

    Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. https://doi.org/10.18653/v1/S18-2023 Hypothesis only baselines in natural language inference . In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics,...

    2018 doi

  23. [31]

    Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. https://doi.org/10.18653/v1/2022.naacl-main.191 Learning to retrieve prompts for in-context learning . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: ...

    2022 doi

  24. [32]

    Timo Schick and Hinrich Sch \"u tze. 2021. https://doi.org/10.18653/v1/2021.naacl-main.185 It ' s not just size that matters: Small language models are also few-shot learners . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computati...

    2021 doi

  25. [33]

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language...

    2013

  26. [34]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  27. [35]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/W18-5446 GLUE : A multi-task benchmark and analysis platform for natural language understanding . In Proceedings of the 2018 EMNLP Workshop B lackbox NLP : A...

    2018 doi

  28. [36]

    Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. https://openreview.net/forum?id=yzkSU5zd...

    2022

  29. [37]

    Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Lingpeng Kong. 2023. https://doi.org/10.18653/v1/2023.acl-long.79 Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering . In Proceedings of the 61st Annual Meeting of ...

    2023 doi

  30. [38]

    Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2022. https://openreview.net/forum?id=RdJVFCHjUMI An explanation of in-context learning as implicit bayesian inference . In International Conference on Learning Representations

    2022

  31. [39]

    Maria Yancheva and Frank Rudzicz. 2013. https://aclanthology.org/P13-1093 Automatic detection of deception in child-produced speech using syntactic complexity features . In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long ...

    2013

  32. [40]

    Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. https://doi.org/10.18653/v1/D19-1382 PAWS - X : A cross-lingual adversarial dataset for paraphrase identification . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the ...

    2019 doi

  33. [41]

    Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. https://proceedings.mlr.press/v202/ye23c.html Compositional exemplars for in-context learning . In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machin...

    2023

  34. [42]

    G Udny Yule. 1939. On sentence-length as a statistical characteristic of style in prose: With application to two cases of disputed authorship. Biometrika, 30(3/4):363--390

    1939

  35. [43]

    Hanlin Zhang, YiFan Zhang, Yaodong Yu, Dhruv Madeka, Dean Foster, Eric Xing, Himabindu Lakkaraju, and Sham Kakade. 2024. https://doi.org/10.18653/v1/2024.naacl-long.340 A study on the calibration of in-context learning . In Proceedings of the 2024 Conference of the North Ameri...

    2024 doi

  36. [44]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068

    2022 arXiv

  37. [45]

    Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. https://proceedings.mlr.press/v139/zhao21c.html Calibrate before use: Improving few-shot performance of language models . In Proceedings of the 38th International Conference on Machine Learning, volume 139 ...

    2021

  38. [46]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  39. [47]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.