Pith. sign in

REVIEW 3 major objections 5 minor 28 references

Refract ICL: Rethinking Example Selection in the Era of Million-Token Models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Refract ICL claims that repeating a model's own zero-shot mistakes inside a long prompt improves in-context learning, especially on tasks with few output classes.

desk verdict Useful scaling study for long-context ICL, but the Refract ICL mechanism claim is underdetermined by a missing token-matched control and an incomplete ablation. read the letter →

arxiv 2506.12346 v1 pith:LUJO3CWD submitted 2025-06-14 cs.CL cs.AI

Arjun R. Akula , Kazuma Hashimoto , Krishna Srinivasan , Aditi Chaudhary , Karthik Raman , Michael Bendersky This is my paper
classification cs.CLcs.AI
keywords in-contextlearninglong-contextlanguagemodelsdemonstrationselectionexamplerepetitionerrorsignalszero-shotpromptingretrievalbaselinesGemini1.5
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that in the era of million-token context windows, in-context learning does not improve simply by packing more demonstrations into the prompt; choosing the right demonstrations still matters. It introduces Refract ICL, a prompt-construction method that identifies examples the model gets wrong under zero-shot prompting, repeats those challenging examples later in the context, and appends the model's zero-shot predictions as explicit error signals. On Gemini 1.5 Pro, this recipe produces consistent accuracy gains across many classification benchmarks, with the largest improvements on tasks that have a small number of output classes. The result matters because it shows that long-context models can be steered by cheap, training-free prompt manipulations rather than by ever-larger demonstration sets.

What carries the argument

The load-bearing object is the Refract ICL context: a standard retrieved set of demonstrations d_1...d_n, followed by repeated copies d'_1... of the subset D' ⊂ D on which the model made wrong zero-shot predictions, with each demonstration and repetition followed by its zero-shot prediction z_i. This construction converts the model's own errors into training-like signals inside the prompt, and the physical repetition of hard examples is meant to break the recency bias of causal attention so that those examples get processed multiple times and interact with each other. The paper uses TF-IDF and T5x retrievers with class balancing as the base selection layer, and Refract ICL is applied on top.

What would settle it

Run Refract ICL against a length-matched control on the same benchmarks: replace the repeated challenging demonstrations and zero-shot predictions with the same number of tokens drawn from randomly chosen demonstrations or filler text. If the control matches Refract ICL's gains on EDOS-A or COUNTFACT, the proposed mechanism—repetition plus error signals—is not what drives the improvement; if it does not, the mechanism survives.

Watch

Extended reading notes

Core claim

The central claim is that scaling the number of demonstrations k in long-context LLMs has diminishing and sometimes negative returns unless the demonstrations are chosen carefully, and that a specific selection strategy—Refract ICL—can recover large gains by focusing the model's attention on hard examples. Refract ICL first obtains zero-shot predictions for a pool of demonstrations, flags those the model answers incorrectly as 'challenging,' appends repeated copies of those challenging demonstrations to the end of the context, and then inserts the zero-shot predictions after each demonstration as error signals. The paper reports that this approach significantly improves Gemini 1.5 Pro's performance relative to strong retrieval baselines, especially on classification tasks with few output classes such as EDOS-A, COUNTFACT, and DDI13, while having weaker effects on tasks with many classes or sequence labeling. Ablations in the paper attribute part of the gain to the repetition itself, which it says counteracts the sequential bias of causal language models.

Load-bearing premise

The paper never controls for the extra tokens that Refract ICL adds: the baseline has one copy of each demonstration with no appended text, while the method adds repeated demonstrations plus zero-shot predictions, so any gain attributed to repetition or error signals could in principle come from simply having more text or more copies in the prompt.

Editorial extensions

If this is right

  • With million-token models, prompt construction can matter as much as retrieval quality, so demonstration selection remains a live design problem even at scale.
  • Repetition inside the context is a cheap way to counteract the recency bias of causal attention, without retraining or fine-tuning the model.
  • Refract ICL's gains concentrate on tasks with a small number of output classes, suggesting the error signal acts as an effective prior in low-cardinality settings.
  • Scaling k alone is not sufficient: models with smaller context windows degrade past certain k values, and even Gemini 1.5 Pro does not benefit uniformly from more randomly chosen examples.
  • The method stacks on top of TF-IDF and T5x retrievers, indicating it can be combined with existing retrieval pipelines rather than replacing them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A length-matched control that repeats randomly chosen demonstrations, or inserts arbitrary filler tokens, would isolate whether the gain comes from repetition and error signals or merely from the extra prompt tokens that Refract ICL adds.
  • The same repetition-plus-error recipe could be tested on tasks with large label spaces by representing error signals differently, for example as the model's confidence or a corrected exemplar, rather than a single zero-shot label.
  • If repetition is what matters, Refract ICL should stack with any base retriever; the paper shows it on TF-IDF and T5x, which suggests it is orthogonal to retrieval quality.
  • The few-class advantage hints that zero-shot error signals act like a prior over a compact label set, so on high-cardinality tasks the error signal may need to carry more information than a wrong label.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper studies in-context learning (ICL) at large demonstration counts in long-context LLMs. Section 2 compares random, TF-IDF, T5x-based, and multi-task retrievers across 11 datasets and three model families (Flan-PaLM 2, Gemini, Gemini 1.5 Pro) with up to 2000 demonstrations, reporting that TF-IDF remains competitive and that scaling k alone does not guarantee gains. Section 3 introduces Refract ICL, which appends the model's zero-shot prediction to each demonstration and repeats demonstrations on which the zero-shot prediction is incorrect, and reports improvements on Gemini 1.5 Pro, especially for few-class classification tasks. The paper concludes that both repetition of challenging examples and integration of error signals drive the gains.

Significance. If the results hold, the paper makes two useful contributions: a broad empirical map of retrieval-based ICL at long context, and a simple, falsifiable prompt-construction trick that may improve long-context ICL on few-class tasks. The Section 2 comparison is valuable and largely convincing. The Refract ICL idea is clearly described and does not involve hidden fitting or circularity in the formal sense, since the target labels are not embedded in the input. However, the central mechanism claims for Refract ICL are currently underdetermined: the experiments do not control for added tokens, do not include the missing error-signal ablation, and do not report variance or significance. These are fixable with additional experiments, so the contribution is conditional rather than established.

major comments (3)
  1. [Section 3.1, Table 2] The Refract ICL condition is not token-matched to the baseline. The baseline rows 'TF-IDF bal' and 'T5x bal' contain k demonstrations, whereas Refract ICL appends a zero-shot prediction to every demonstration and repeats 'challenging' demonstrations at the end of the context. Since Section 2 and Figure 1 show that simply increasing the number of demonstrations (and hence the number of tokens) improves Gemini 1.5 Pro on several datasets, the deltas in Table 2 could be explained by extra context alone rather than by the proposed repetition/error-signal mechanism. The paper needs a control that adds the same number of tokens without the proposed content (e.g., repeated non-challenging demonstrations, or appended neutral text) and should report matched-token comparisons.
  2. [Section 3, Table 3] The only ablation removes repetition but retains the appended zero-shot predictions, so it cannot isolate the contribution of repetition; there is no ablation that removes the zero-shot predictions while keeping repetition, despite the conclusion in Section 4 attributing the gains to both components. Moreover, the differences in Table 3 are mostly 0.00-0.03 (e.g., COUNTFACT 0.77 vs 0.77, DDI13 0.48 vs 0.48, MTOP-intent 0.97 vs 0.97), which does not support the stated 'noticeable performance decrease' without error bars or significance tests. Please add a 2x2 ablation crossing repetition on/off with error-signal on/off.
  3. [Section 3.1, Tables 2-3] No error bars, confidence intervals, significance tests, or number of evaluation examples are reported. Many of the claimed gains in Table 2 are 0.01-0.05 on a 0-1 scale (e.g., EDOS-A +0.05 at k=50, COUNTFACT +0.02 to +0.05, DDI13 +0.02 to +0.06), and Table 3 differences fall in the same range. Without variance estimates or repeated runs, the headline claim that Refract ICL 'significantly' improves performance is not supported. Please report test-set sizes, seeds, and interval estimates for the main comparisons.
minor comments (5)
  1. [Section 3] The method description does not specify the number of repetitions of challenging demonstrations, the threshold used to define 'challenging', or whether D' is selected per test input or globally. Please provide these details together with a prompt template in an appendix for reproducibility.
  2. [Tables 2-3] Table 2 contains malformed entries such as '+−0.00' in the MTOP-intent rows, and Table 3's header 'w/ repeat w/o repeat' is unclear. Please clean up the formatting and clarify the metric used for ATIS-intent (95.8 vs 0.97 elsewhere suggests inconsistent scaling).
  3. [References] The bibliography contains stray spaces in 'Proceedings of the F ourth V olume' and a few other entries; please proofread the references.
  4. [Figures 1-2] Figure 1's y-axis is not labeled and the caption does not define the plotted metric or the aggregation across datasets. Please add axis labels and explicitly state the metric.
  5. [Sections 3.1 and 5] The future-work text in Section 3.1 is repeated nearly verbatim in Section 5 (Limitations); consider consolidating these passages.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Refract ICL is an empirical comparison whose claims rest on benchmark measurements, not on a derivation that reduces to its own inputs.

full rationale

I walked the claimed derivation chain: the paper makes no formal derivation; it introduces an ICL construction (repeat challenging demonstrations and append zero-shot predictions) and measures it on standard benchmarks against retrieval baselines. The 'challenging' subset is defined by the model's zero-shot predictions on training demonstrations, which is self-referential but not circular: the target quantities (test-set metrics) are not used to construct the prompt. Citations to prior work by overlapping authors (e.g., Gao et al., 2023) are introductory context for retrieval baselines, not load-bearing evidence for the method's efficacy. The only substantive concerns are experimental: Table 2 compares Refract prompts to shorter baselines without token-count matching, and Table 3 ablates repetition while retaining error signals, so the mechanism attribution is underdetermined. These are missing controls, not circular reductions, so they do not raise the circularity score.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities or fitted parameters. Its claims rest on assumptions about long-context attention, sequential bias, the usefulness of self-generated error signals, and the meaningfulness of small point estimates.

assumptions (4)
  • domain assumption Long-context models can effectively utilize repeated demonstrations.
    Refract ICL doubles or multiplies the count of some examples; this only works if the model can attend to repeated content in a 2M-token context, which is assumed throughout Section 3.
  • domain assumption Causal LMs have a sequential bias that repetition mitigates.
    The motivation in Section 3 relies on Gong et al. (2023) for this claim; if false, repetition would not serve the stated purpose.
  • ad hoc to paper Appending the model's own zero-shot prediction to a demonstration provides a beneficial error signal.
    This is the core mechanism of Refract ICL and is only evaluated indirectly; no baseline with arbitrary extra text is shown.
  • domain assumption Small metric deltas (0.01 to 0.05) are meaningful without statistical testing.
    Tables 2 and 3 report such deltas as evidence of improvement, but no variance or significance information is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Refract ICL: Rethinking Example Selection in the Era of Million-Token Models." pith.science (2026). https://pith.science/paper/LUJO3CWD

@misc{pith2026250612346,
  author       = {Pith},
  title        = {Pith review of: Refract ICL: Rethinking Example Selection in the Era of Million-Token Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LUJO3CWD}},
  note         = {Machine review of arXiv:2506.12346}
}
read the original abstract

The emergence of long-context large language models (LLMs) has enabled the use of hundreds, or even thousands, of demonstrations for in-context learning (ICL) - a previously impractical regime. This paper investigates whether traditional ICL selection strategies, which balance the similarity of ICL examples to the test input (using a text retriever) with diversity within the ICL set, remain effective when utilizing a large number of demonstrations. Our experiments demonstrate that, while longer contexts can accommodate more examples, simply increasing the number of demonstrations does not guarantee improved performance. Smart ICL selection remains crucial, even with thousands of demonstrations. To further enhance ICL in this setting, we introduce Refract ICL, a novel ICL selection algorithm specifically designed to focus LLM attention on challenging examples by strategically repeating them within the context and incorporating zero-shot predictions as error signals. Our results show that Refract ICL significantly improves the performance of extremely long-context models such as Gemini 1.5 Pro, particularly on tasks with a smaller number of output classes.

Figures

Figures reproduced from arXiv: 2506.12346 by the authors.

Figure 1
Figure 1. Performance of Gemini 1.5 Pro (2M context) with up to 2000 randomly retrieved demonstrations shows that increasing k alone does not guarantee improvement on all datasets [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Performance on ATIS and BC5CDR datasets with Gemini 1.5 Pro (2M context) shows that even with up to 2000 demonstrations, smart retrieval (TF-IDF and T5x with balancing) consistently outperforms random selection. lengths: Short Context: Flan-PaLM 2 (S) (Anil et al., 2023) (8K tokens). Long Context: Gemini (Team et al., 2023) (32K tokens). Extremely Long Context: Gemini 1.5 Pro (Reid et al., 2024) (2 Million tokens). … view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

28 extracted references · 7 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv preprint arXiv:2305.10403

    arXiv 2023

  4. [4]

    Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R Gormley, and Graham Neubig. 2024. In-context learning with long-context models: An in-depth exploration. arXiv preprint arXiv:2405.00200

    arXiv 2024

  5. [5]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert - Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litw...

    work page 2020

  6. [6]

    Rajarshi Das, Manzil Zaheer, Dung Thai, Ameya Godbole, Ethan Perez, Jay Yoon Lee, Lizhen Tan, Lazaros Polymenakos, and Andrew McCallum. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.755 Case-based reasoning for natural language queries over knowledge bases . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pag...

    doi:10.18653/v1/2021.emnlp-main.755 2021

  7. [7]

    Lingyu Gao, Aditi Chaudhary, Krishna Srinivasan, Kazuma Hashimoto, Karthik Raman, and Michael Bendersky. 2023. Ambiguity-aware in-context learning with large language models. arXiv preprint arXiv:2309.07900

    arXiv 2023

  8. [8]

    Zhuocheng Gong, Jiahao Liu, Qifan Wang, Jingang Wang, Xunliang Cai, Dongyan Zhao, and Rui Yan. 2023. Improving input-label mapping with demonstration replay for in-context learning. arXiv preprint arXiv:2310.19572

    work page Pith review arXiv 2023

Show all 28 references
  1. [9]

    Kazuma Hashimoto, Raffaella Buschiazzo, James Bradbury, Teresa Marshall, Richard Socher, and Caiming Xiong. 2019. https://doi.org/10.18653/v1/W19-5212 A High-Quality Multilingual Dataset for Structured Documentation Translation . In Proceedings of the Fourth Conference on Mach...

    2019 doi

  2. [10]

    María Herrero-Zazo, Isabel Segura-Bedmar, Paloma Martínez, and Thierry Declerck. 2013. https://doi.org/10.1016/j.jbi.2013.07.011 The DDI corpus: An annotated corpus with pharmacological substances and drug–drug interactions . Journal of Biomedical Informatics, 46(5):914--920

    2013 doi

  3. [11]

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118

    2021 arXiv

  4. [12]

    Hannah Kirk, Wenjie Yin, Bertie Vidgen, and Paul R \"o ttger. 2023. https://doi.org/10.18653/v1/2023.semeval-1.305 S em E val-2023 Task 10: Explainable Detection of Online Sexism . In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2...

    2023 doi

  5. [13]

    Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.243 The power of scale for parameter-efficient prompt tuning . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045--3059, Online a...

    2021 doi

  6. [14]

    Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. 2021. https://doi.org/10.18653/v1/2021.eacl-main.257 MTOP : A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark . In Proceedings of the 16th Conference of the European Chapt...

    2021 doi

  7. [15]

    Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J

    Jiao Li, Yueping Sun, Robin J. Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J. Mattingly, Thomas C. Wiegers, and Zhiyong Lu. 2016. https://api.semanticscholar.org/CorpusID:88817 BioCreative V CDR task corpus: a resource for chemical diseas...

    2016

  8. [16]

    Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. 2023. https://doi.org/10.18653/v1/2023.acl-long.256 Unified Demonstration Retriever for In-Context Learning . In Proceedings of the 61st Annual Meeting of the Association ...

    2023 doi

  9. [17]

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. https://doi.org/10.18653/v1/2022.deelio-1.10 What makes good in-context examples for GPT -3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extr...

    2022 doi

  10. [18]

    Katerina Margatina, Timo Schick, Nikolaos Aletras, and Jane Dwivedi-Yu. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.334 Active learning principles for in-context learning with large language models . In Findings of the Association for Computational Linguistics: EMNLP...

    2023 doi

  11. [19]

    Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. 2022. https://doi.org/10.18653/v1/2022.findings-acl.146 Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models . In Findings of the Association for Computa...

    2022 doi

  12. [20]

    James O'Neill, Polina Rozenshtein, Ryuichi Kiryo, Motoko Kubota, and Danushka Bollegala. 2021. https://arxiv.org/abs/2104.06893 I Wish I Would Have Loved This One, But I Didn't -- A Multilingual Dataset for Counterfactual Detection in Product Reviews . Preprint, arXiv:2104.06893

    2021 arXiv

  13. [21]

    Patti Price. 1990. Evaluation of spoken language systems: The ATIS domain . In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990

    1990

  14. [22]

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arX...

    2024 arXiv

  15. [23]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  16. [24]

    Liang Wang, Nan Yang, and Furu Wei. 2023. Learning to Retrieve In-Context Examples for Large Language Models . arXiv preprint cs.CL 2307.07164

    2023 arXiv

  17. [25]

    Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. https://doi.org/10.18653/v1/2021.naacl-main.41 mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer . In Proceedings of the 2021 Conference ...

    2021 doi

  18. [26]

    Zhao Yang, Yuanzhe Zhang, Dianbo Sui, Cao Liu, Jun Zhao, and Kang Liu. 2023. Representative demonstration selection for in-context learning with two-stage determinantal point process. In The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  19. [27]

    Yiming Zhang, Shi Feng, and Chenhao Tan. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.622 Active example selection for in-context learning . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9134--9148, Abu Dhabi, United Arab...

    2022 doi

  20. [28]

    Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models . In International Conference on Machine Learning, pages 12697--12706

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.