{"id":"e14af914-4b8d-4613-9da3-4c2ef60b27d1","arxiv_id":"2506.12346","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In long-context models, more demonstrations do not automatically help; repeating hard examples and appending the model's own zero-shot predictions gives small, inconsistent gains.","lead":"Example selection still matters when a model can read thousands of examples: adding more demonstrations does not always improve accuracy. The paper proposes Refract ICL, which repeats examples the model gets wrong and adds its own zero-shot guesses, yielding small gains on classification tasks with few labels.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Refract ICL gains are not token-matched against a longer-context control, and the Table 3 ablation does not isolate the error-signal component; the mechanism claim is underdetermined.","rationale":"The reader's weakest assumption is the absence of a control for extra tokens added by Refract ICL. I agree that this is the most load-bearing concern: the method's novelty is mechanistic (challenging repetition and error signals), yet the experimental design confounds the mechanism with prompt length and token count. The paper's own Figure 1 shows that simply adding demonstrations can improve performance in some settings, so the token-count confound is not ruled out by existing data. Furthermore, the only mechanism ablation (Table 3) removes repetition but retains error signals, and its differences are small and not statistically characterized; it does not support the strong claim that repetition is crucial. No ablation removes the error-signal component, so both components of the proposed mechanism are not separately validated. Because the reader already assigned a CONDITIONAL verdict based in part on this same concern, my stress-test does not move the verdict; it reinforces the need for the stated controls before the central claim can be accepted.","tokens_in":10091,"tokens_out":6489,"duration_ms":82681,"concrete_test":"Re-run the Refract ICL comparison on EDOS-A, COUNTFACT, and DDI13 at k=100 using T5x-bal retrieval, with a token-matched control in which the repeated 'challenging' demonstrations are replaced by an equal number of randomly selected demonstrations (keeping all zero-shot predictions appended). If this control reaches the same F1/accuracy as Refract ICL, the proposed mechanism is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Refract ICL's gains come from repeating challenging demonstrations and incorporating zero-shot error signals (Section 3) is not identified by the experiments. Table 2 compares Refract prompts (k demonstrations + zero-shot predictions appended to each + repeated challenging demonstrations) against a baseline containing only k demonstrations, never matching the added token count. Since long-context performance can improve simply from adding more demonstrations (Figure 1), any gain could reflect extra context rather than the proposed selection/repetition. Table 3, the only ablation, removes repetition but keeps the error signals and shows tiny differences (e.g., COUNTFACT 0.77 vs 0.77; DDI13 0.48 vs 0.48; most other datasets <=0.03), so it does not establish repetition as the driver. There is no ablation that removes the appended zero-shot predictions while keeping repetition, despite the conclusion attributing the effect to both components. Without a token-matched control and an error-signal ablation, the mechanism claim is underdetermined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies in-context learning (ICL) at large demonstration counts in long-context LLMs. Section 2 compares random, TF-IDF, T5x-based, and multi-task retrievers across 11 datasets and three model families (Flan-PaLM 2, Gemini, Gemini 1.5 Pro) with up to 2000 demonstrations, reporting that TF-IDF remains competitive and that scaling k alone does not guarantee gains. Section 3 introduces Refract ICL, which appends the model's zero-shot prediction to each demonstration and repeats demonstrations on which the zero-shot prediction is incorrect, and reports improvements on Gemini 1.5 Pro, especially for few-class classification tasks. The paper concludes that both repetition of challenging examples and integration of error signals drive the gains.","tokens_in":10270,"tokens_out":5093,"duration_ms":59796,"significance":"If the results hold, the paper makes two useful contributions: a broad empirical map of retrieval-based ICL at long context, and a simple, falsifiable prompt-construction trick that may improve long-context ICL on few-class tasks. The Section 2 comparison is valuable and largely convincing. The Refract ICL idea is clearly described and does not involve hidden fitting or circularity in the formal sense, since the target labels are not embedded in the input. However, the central mechanism claims for Refract ICL are currently underdetermined: the experiments do not control for added tokens, do not include the missing error-signal ablation, and do not report variance or significance. These are fixable with additional experiments, so the contribution is conditional rather than established.","major_comments":[{"comment":"The Refract ICL condition is not token-matched to the baseline. The baseline rows 'TF-IDF bal' and 'T5x bal' contain k demonstrations, whereas Refract ICL appends a zero-shot prediction to every demonstration and repeats 'challenging' demonstrations at the end of the context. Since Section 2 and Figure 1 show that simply increasing the number of demonstrations (and hence the number of tokens) improves Gemini 1.5 Pro on several datasets, the deltas in Table 2 could be explained by extra context alone rather than by the proposed repetition/error-signal mechanism. The paper needs a control that adds the same number of tokens without the proposed content (e.g., repeated non-challenging demonstrations, or appended neutral text) and should report matched-token comparisons.","section":"Section 3.1, Table 2"},{"comment":"The only ablation removes repetition but retains the appended zero-shot predictions, so it cannot isolate the contribution of repetition; there is no ablation that removes the zero-shot predictions while keeping repetition, despite the conclusion in Section 4 attributing the gains to both components. Moreover, the differences in Table 3 are mostly 0.00-0.03 (e.g., COUNTFACT 0.77 vs 0.77, DDI13 0.48 vs 0.48, MTOP-intent 0.97 vs 0.97), which does not support the stated 'noticeable performance decrease' without error bars or significance tests. Please add a 2x2 ablation crossing repetition on/off with error-signal on/off.","section":"Section 3, Table 3"},{"comment":"No error bars, confidence intervals, significance tests, or number of evaluation examples are reported. Many of the claimed gains in Table 2 are 0.01-0.05 on a 0-1 scale (e.g., EDOS-A +0.05 at k=50, COUNTFACT +0.02 to +0.05, DDI13 +0.02 to +0.06), and Table 3 differences fall in the same range. Without variance estimates or repeated runs, the headline claim that Refract ICL 'significantly' improves performance is not supported. Please report test-set sizes, seeds, and interval estimates for the main comparisons.","section":"Section 3.1, Tables 2-3"}],"minor_comments":[{"comment":"The method description does not specify the number of repetitions of challenging demonstrations, the threshold used to define 'challenging', or whether D' is selected per test input or globally. Please provide these details together with a prompt template in an appendix for reproducibility.","section":"Section 3"},{"comment":"Table 2 contains malformed entries such as '+−0.00' in the MTOP-intent rows, and Table 3's header 'w/ repeat w/o repeat' is unclear. Please clean up the formatting and clarify the metric used for ATIS-intent (95.8 vs 0.97 elsewhere suggests inconsistent scaling).","section":"Tables 2-3"},{"comment":"The bibliography contains stray spaces in 'Proceedings of the F ourth V olume' and a few other entries; please proofread the references.","section":"References"},{"comment":"Figure 1's y-axis is not labeled and the caption does not define the plotted metric or the aggregation across datasets. Please add axis labels and explicitly state the metric.","section":"Figures 1-2"},{"comment":"The future-work text in Section 3.1 is repeated nearly verbatim in Section 5 (Limitations); consider consolidating these passages.","section":"Sections 3.1 and 5"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of a CL conference and I see no formal circularity in the Refract ICL construction. The main risk is that the positive results in Section 3 are small and not controlled; if the authors cannot supply token-matched controls and the missing error-signal ablation, the Refract ICL contribution would be much weaker. The Section 2 empirical comparison is a solid basis for a revised paper. I recommend major revision rather than rejection because the required controls are concrete and within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The main thing you should know: this paper's qualitative message about ICL in long-context models is solid, but its proposed method, Refract ICL, is not yet established as the cause of the reported gains.\n\nThe scaling study is genuinely useful. Showing that randomly increasing k stops helping and that simple TF-IDF beats fine-tuned retrievers on 2000-shot prompts with Gemini 1.5 Pro is a real datapoint for anyone building long-context prompts. That part holds up, and it is the first evaluation I know of at this scale.\n\nThe soft spot is Section 3. Refract ICL adds repeated challenging demonstrations and appends zero-shot predictions to every demonstration. The baseline in Table 2 is the single-copy, no-prediction prompt. That is not token-matched, so the gains could simply come from having more text, which Figure 1 shows can matter. The ablation in Table 3 removes repetition but keeps the error signals, and the differences are tiny: COUNTFACT 0.77 vs 0.77, DDI13 0.48 vs 0.48, most other datasets within 0.01–0.02. No ablation removes the error signals while keeping repetition, so the paper's conclusion that both components contribute is not actually tested. Also, there are no error bars or significance tests, and many deltas are 0.01–0.05, which is noise-level for these models. The model comparison also confounds context length with model identity, but that is a minor point compared to the missing control.\n\nThat said, the paper is transparent about its limitations: it admits the method only helps on few-class tasks and says error-signal integration for complex tasks is future work. The citation pattern is fine, and I do not see circularity on the terms the reader defined. The building blocks are known, but the combination and the evaluation scale are new. This is a legitimate empirical contribution that needs stronger evidence before the mechanism claim is accepted.\n\nWho is this for? People working on ICL, prompt design, and long-context evaluation. It deserves a serious referee, but the referee should ask for a token-matched baseline and a clean ablation that isolates the error-signal component. I would read a revised version, and I would cite the scaling results, not the Refract method as such.","headline":"Useful scaling study for long-context ICL, but the Refract ICL mechanism claim is underdetermined by a missing token-matched control and an incomplete ablation.","tokens_in":10816,"tokens_out":1805,"would_cite":true,"duration_ms":21824,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Refract ICL claims that repeating a model's own zero-shot mistakes inside a long prompt improves in-context learning, especially on tasks with few output classes.","keywords":["in-context learning","long-context language models","demonstration selection","example repetition","error signals","zero-shot prompting","retrieval baselines","Gemini 1.5"],"falsifier":"Run Refract ICL against a length-matched control on the same benchmarks: replace the repeated challenging demonstrations and zero-shot predictions with the same number of tokens drawn from randomly chosen demonstrations or filler text. If the control matches Refract ICL's gains on EDOS-A or COUNTFACT, the proposed mechanism—repetition plus error signals—is not what drives the improvement; if it does not, the mechanism survives.","tokens_in":9891,"feed_emoji":"🧠","tokens_out":4588,"duration_ms":49192,"temperature":0.7,"pith_summary":"The paper argues that in the era of million-token context windows, in-context learning does not improve simply by packing more demonstrations into the prompt; choosing the right demonstrations still matters. It introduces Refract ICL, a prompt-construction method that identifies examples the model gets wrong under zero-shot prompting, repeats those challenging examples later in the context, and appends the model's zero-shot predictions as explicit error signals. On Gemini 1.5 Pro, this recipe produces consistent accuracy gains across many classification benchmarks, with the largest improvements on tasks that have a small number of output classes. The result matters because it shows that long-context models can be steered by cheap, training-free prompt manipulations rather than by ever-larger demonstration sets.","feed_headline":"Repeating hard examples lifts long-context ICL","feed_subtitle":"Refract ICL recycles the model's own wrong answers into the prompt—and the gains are biggest on few-class tasks.","key_machinery":"The load-bearing object is the Refract ICL context: a standard retrieved set of demonstrations d_1...d_n, followed by repeated copies d'_1... of the subset D' ⊂ D on which the model made wrong zero-shot predictions, with each demonstration and repetition followed by its zero-shot prediction z_i. This construction converts the model's own errors into training-like signals inside the prompt, and the physical repetition of hard examples is meant to break the recency bias of causal attention so that those examples get processed multiple times and interact with each other. The paper uses TF-IDF and T5x retrievers with class balancing as the base selection layer, and Refract ICL is applied on top.","core_discovery":"The central claim is that scaling the number of demonstrations k in long-context LLMs has diminishing and sometimes negative returns unless the demonstrations are chosen carefully, and that a specific selection strategy—Refract ICL—can recover large gains by focusing the model's attention on hard examples. Refract ICL first obtains zero-shot predictions for a pool of demonstrations, flags those the model answers incorrectly as 'challenging,' appends repeated copies of those challenging demonstrations to the end of the context, and then inserts the zero-shot predictions after each demonstration as error signals. The paper reports that this approach significantly improves Gemini 1.5 Pro's performance relative to strong retrieval baselines, especially on classification tasks with few output classes such as EDOS-A, COUNTFACT, and DDI13, while having weaker effects on tasks with many classes or sequence labeling. Ablations in the paper attribute part of the gain to the repetition itself, which it says counteracts the sequential bias of causal language models.","pith_inferences":["A length-matched control that repeats randomly chosen demonstrations, or inserts arbitrary filler tokens, would isolate whether the gain comes from repetition and error signals or merely from the extra prompt tokens that Refract ICL adds.","The same repetition-plus-error recipe could be tested on tasks with large label spaces by representing error signals differently, for example as the model's confidence or a corrected exemplar, rather than a single zero-shot label.","If repetition is what matters, Refract ICL should stack with any base retriever; the paper shows it on TF-IDF and T5x, which suggests it is orthogonal to retrieval quality.","The few-class advantage hints that zero-shot error signals act like a prior over a compact label set, so on high-cardinality tasks the error signal may need to carry more information than a wrong label."],"forward_implications":["With million-token models, prompt construction can matter as much as retrieval quality, so demonstration selection remains a live design problem even at scale.","Repetition inside the context is a cheap way to counteract the recency bias of causal attention, without retraining or fine-tuning the model.","Refract ICL's gains concentrate on tasks with a small number of output classes, suggesting the error signal acts as an effective prior in low-cardinality settings.","Scaling k alone is not sufficient: models with smaller context windows degrade past certain k values, and even Gemini 1.5 Pro does not benefit uniformly from more randomly chosen examples.","The method stacks on top of TF-IDF and T5x retrievers, indicating it can be combined with existing retrieval pipelines rather than replacing them."],"supporting_citations":[{"why":"Supplies the Gemini 1.5 Pro model with a 2 million token context, the extremely long-context setting the paper targets.","marker":"Reid et al., 2024"},{"why":"Introduces in-context learning with demonstrations, the paradigm this paper extends to long contexts.","marker":"Brown et al., 2020"},{"why":"Foundational result that demonstration similarity to the test input matters, the selection principle Refract ICL builds on.","marker":"Liu et al., 2022"},{"why":"The demonstration-replay idea for breaking sequential bias in causal LMs, which Refract ICL adapts via repetition.","marker":"Gong et al., 2023"},{"why":"Concurrent exploration of many-shot ICL with long-context models up to 80k tokens; this paper extends that regime to 2M tokens.","marker":"Bertsch et al., 2024"},{"why":"Provides the t5x-retrieval code base used to build the T5x retriever baseline.","marker":"Ni et al., 2022"},{"why":"The contrastive retrieval objective used to fine-tune mT5 for the retrieval baselines.","marker":"Izacard et al., 2021"}],"fun_headline_variants":["Repeating hard examples beats adding more for long-context ICL","Refract ICL: repeat hard examples flagged by zero-shot errors","Hard-example repetition with error signals improves few-class tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper never controls for the extra tokens that Refract ICL adds: the baseline has one copy of each demonstration with no appended text, while the method adds repeated demonstrations plus zero-shot predictions, so any gain attributed to repetition or error signals could in principle come from simply having more text or more copies in the prompt.","fun_headline_variants_meta":{"raw":{"variants":["Repeating hard examples beats adding more for long-context ICL","Refract ICL: repeat hard examples flagged by zero-shot errors","Hard-example repetition with error signals improves few-class tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000965,"raw_usage":{"total_tokens":4099,"prompt_tokens":930,"completion_tokens":3169,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":3112}},"tokens_in":546,"tokens_out":3169,"duration_ms":29761,"temperature":1.0,"reasoning_tokens":3112,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:52:12.721697+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Refract ICL against a length-matched control on the same benchmarks: replace the repeated challenging demonstrations and zero-shot predictions with the same number of tokens drawn from randomly chosen demonstrations or filler text. If the control matches Refract ICL's gains on EDOS-A or COUNTFACT, the proposed mechanism—repetition plus error signals—is not what drives the improvement; if it does not, the mechanism survives.","supporting_citations":[{"cited_title":"Improving Input-label Mapping with Demonstration Replay for In-context Learning","cited_arxiv_id":"2310.19572","evidence_quote":"The demonstration-replay idea for breaking sequential bias in causal LMs, which Refract ICL adapts via repetition."}],"review_version":1}