REVIEW 3 major objections 3 minor 1 cited by
This paper argues that diversity in the initial retrieval of in-context demonstrations—not ranking-stage optimization—is the dominant factor for biomedical few-shot LLM performance, and that a two-stage submodular selector achieves up to 5%
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
A diversity-first, submodular two-stage demonstration selector beats representative-only baselines for biomedical in-context learning, with 3 to 5 examples optimal.
T0 review reviewed 2026-08-05 challenge →
load-bearing objection A plausible and potentially useful method paper whose central empirical claims are unverified from the abstract alone; the stage-1 coverage worry is real and should be addressed in full-text review. the 3 major comments →
Data-Efficient Biomedical In-Context Learning: A Diversity-Enhanced Submodular Perspective
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
On the paper's own terms, the central claim is that a diversity-enhanced submodular objective at the candidate-retrieval stage dominates downstream ranking optimization in demonstration selection for biomedical in-context learning. Dual-Div's stage one chooses a small candidate set from the corpus by jointly maximizing representativeness and diversity (with the option of annotation to handle unlabeled data); stage two then ranks only those candidates against each test query. The reported result is that this pipeline outperforms baselines that emphasize representativeness at both stages, with up to 5% higher macro-F1, and that resistance to prompt permutations and class imbalance follows from
What carries the argument
The central object is the submodular objective in Dual-Div's two-stage pipeline. In stage one, a submodular function combines representativeness (coverage of the corpus) with diversity (penalizing redundancy among selected examples), yielding a limited candidate pool; optionally, annotation is used to incorporate unlabeled data. In stage two, those candidates are ranked against the test query to pick the most relevant and non-redundant few-shot demonstrations. The claim that diversity at initial retrieval outweighs ranking-stage optimization is carried by the comparison between this two-stage design and representativeness-only baselines.
Load-bearing premise
The approach assumes that the stage-one candidate pool, built from corpus-wide representativeness and diversity before seeing any test query, always contains the most informative example for that query—if a useful demonstration is pruned in stage one, the stage-two ranker can never recover it.
What would settle it
Take a small biomedical corpus and, for a fixed test query, compute the best possible demonstration by exhaustive search over all corpus examples. If in a substantial fraction of queries the best example lies outside the Dual-Div stage-one candidate set, or if replacing Dual-Div's stage-one pool with random sampling of equal size yields comparable macro-F1, then the claim that diversity in initial retrieval is the critical factor is falsified.
If this is right
- If Dual-Div is correct, demonstration selection for biomedical ICL can be improved without extra inference cost, purely by changing how prompt examples are chosen.
- The finding that 3–5 demonstrations are optimal gives a practical heuristic: adding more examples beyond that range is unlikely to help and may hurt.
- The stage-one-dominance claim implies that retrieval-stage diversity should be a default consideration in ICL pipelines, not an afterthought in ranking.
- Stability under prompt permutations and class imbalance, if true, makes Dual-Div a safer choice in low-resource biomedical settings where label distributions are skewed.
- The optional annotation step for unlabeled corpora suggests a route to using large raw text collections without full labels.
Where Pith is reading between the lines
- A testable extension would be to measure whether the same diversity-first ordering holds for general-domain ICL, not just biomedical text, since the paper's mechanism is task-agnostic.
- The coverage assumption implies a concrete failure mode: on a query whose ideal example is pruned early, no stage-two reranking can restore it. Probing with adversarial queries would reveal how often that happens.
- The 3–5 example optimum may depend on the evaluated LLMs and retrievers; the paper does not control for model scale, so larger models could shift the optimum.
- The 'diversity is more critical' claim is comparative, not absolute; it does not mean representativeness is useless, only that at the retrieval stage it matters less. A follow-up could test whether the two stages can be reordered or merged without loss.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Dual-Div, a two-stage demonstration-selection method for in-context learning on biomedical NLP tasks. Stage 1 builds a limited candidate set from a corpus by optimizing both representativeness and diversity via submodular functions; stage 2 ranks these candidates against a test query to pick relevant, non-redundant demonstrations. The authors evaluate on three biomedical tasks (NER, RE, TC) with two LLMs (LLaMA 3.1, Qwen 2.5) and three retrievers (BGE-Large, BMRetriever, MedCPT), reporting up to 5% higher macro-F1 than baselines, robustness to prompt permutations and class imbalance, and two findings: diversity in initial retrieval is more critical than ranking-stage optimization, and 3-5 demonstrations maximize performance efficiency.
Significance. If the claims hold, the paper makes a practical contribution by showing that diversity-aware candidate selection improves few-shot biomedical ICL, an area where representativeness-only selection is common. The evaluation grid—three tasks, two LLMs, three retrievers, plus robustness checks on prompt permutations and class imbalance—is broader than is typical in the demonstration-selection literature. The two-stage formulation is also sensible for scaling to large corpora. However, the abstract-level evidence does not establish the central explanatory claim that diversity at stage 1 is more critical than ranking at stage 2, because the stage-1 coverage assumption is not directly tested. The reported performance gains also lack statistical quantification. These are load-bearing gaps that need to be addressed before the findings can be accepted as stated.
major comments (3)
- [Abstract (two-stage process)] The central claim that 'diversity in initial retrieval is more critical than ranking-stage optimization' is underdetermined by the architecture. Stage 2 ranks only candidates selected in stage 1, which is built from corpus statistics before the test query is seen. If the stage-1 pool omits the most informative example for a given query, stage 2 cannot recover it. Without an oracle-recall analysis (does the stage-1 pool contain the best demonstrations?) or an ablation over candidate-pool size, including a full-corpus ranker as an upper bound, the observed superiority of stage-1 diversity over stage-2 ranking could be an artifact of a bottlenecked candidate set rather than a property of the selector. This is a structural risk directly tied to the paper's headline finding.
- [Abstract (performance claims)] The abstract reports 'up to 5% higher macro-F1 scores' with no variance, confidence intervals, or significance tests. 'Consistently outperforms' and 'robustness to prompt permutations and class imbalance' are strong claims; without error bars or paired statistical tests across seeds, tasks, and models, the reader cannot assess whether the improvements are reliable or within noise. This is especially important for the downstream claim that 3-5 demonstrations are optimal, which requires comparing adjacent values of k.
- [Abstract (finding on demonstration count)] The statement that 'limiting demonstrations to 3-5 examples maximizes performance efficiency' appears to be a grid-search result. The abstract does not indicate whether this holds consistently across all task/model/retriever combinations, or whether differences between, say, 3 and 5 demonstrations are statistically meaningful. If the full paper includes per-cell results and significance testing, this comment is addressable in revision; as presented, the claim overspecifies the evidence.
minor comments (3)
- [Abstract] The phrase 'with optional annotation for unlabeled data' is vague. Does this mean the method can incorporate labeled or unlabeled data depending on availability, or that it has an annotation suggestion mode? Please define this in the abstract or introduction.
- [Abstract] The paper is about submodular optimization, but the abstract does not name the submodular function used or how representativeness and diversity are balanced. A sentence with the specific objective (e.g., facility location, graph cut) would help readers identify the contribution.
- [General] Because this is an abstract-only review, I could not verify whether the full text contains the ablations suggested above. If they are present, the abstract should be revised to summarize them; if not, they must be added.
Circularity Check
No circularity evident in the abstract-only review; claims are empirical and measured against external baselines.
full rationale
The paper is an empirical method-and-evaluation study. The abstract describes a two-stage retrieval-and-ranking framework (Dual-Div) and reports measured macro-F1 improvements over baselines on three biomedical NLP tasks. There is no derivation chain in which a prediction reduces by construction to a fitted parameter or a self-citation. The claim that 'diversity in initial retrieval is more critical than ranking-stage optimization' and the '3-5 examples optimal' finding are empirical outcomes of experiments, not consequences of how the method is defined. No equations are provided in the abstract, so there is no way to exhibit a specific reduction. The skeptic's concern about the stage-1 candidate pool bottleneck is a potential validity threat (coverage assumption), but it is not circularity: the candidate pool is not defined in terms of the final prediction, and the finding could in principle be false. The 'up to 5%' figure is a best-case performance claim, not a fitted value renamed as a prediction. There is also no evidence of self-citation serving as load-bearing justification. Therefore, no circularity can be identified from the available text.
Axiom & Free-Parameter Ledger
free parameters (3)
- number of demonstrations k =
3 to 5
- stage-1 candidate pool size
- diversity-representativeness balance weight
axioms (4)
- domain assumption The selected utility function over demonstration sets is submodular (diminishing returns), so submodular maximization or its surrogate applies.
- domain assumption Retriever embeddings (BGE-Large, BMRetriever, MedCPT) are a faithful similarity space for defining both representativeness and diversity in biomedical text.
- ad hoc to paper The stage-1 candidate pool, built without the test query, contains the demonstrations that the stage-2 ranker would need (coverage assumption).
- domain assumption LLMs (LLaMA 3.1, Qwen 2.5) exhibit stable in-context learning on the three biomedical tasks, so performance differences are attributable to demonstration selection rather than model instability.
Cite this review
Pith. "Pith review of Data-Efficient Biomedical In-Context Learning: A Diversity-Enhanced Submodular Perspective." pith.science (2026). https://pith.science/paper/MJ2RJBY5
@misc{pith2026250808140,
author = {Pith},
title = {Pith review of: Data-Efficient Biomedical In-Context Learning: A Diversity-Enhanced Submodular Perspective},
year = {2026},
howpublished = {\url{https://pith.science/paper/MJ2RJBY5}},
note = {Machine review of arXiv:2508.08140}
}
read the original abstract
Recent progress in large language models (LLMs) has leveraged their in-context learning (ICL) abilities to enable quick adaptation to unseen biomedical NLP tasks. By incorporating only a few input-output examples into prompts, LLMs can rapidly perform these new tasks. While the impact of these demonstrations on LLM performance has been extensively studied, most existing approaches prioritize representativeness over diversity when selecting examples from large corpora. To address this gap, we propose Dual-Div, a diversity-enhanced data-efficient framework for demonstration selection in biomedical ICL. Dual-Div employs a two-stage retrieval and ranking process: First, it identifies a limited set of candidate examples from a corpus by optimizing both representativeness and diversity (with optional annotation for unlabeled data). Second, it ranks these candidates against test queries to select the most relevant and non-redundant demonstrations. Evaluated on three biomedical NLP tasks (named entity recognition (NER), relation extraction (RE), and text classification (TC)) using LLaMA 3.1 and Qwen 2.5 for inference, along with three retrievers (BGE-Large, BMRetriever, MedCPT), Dual-Div consistently outperforms baselines-achieving up to 5% higher macro-F1 scores-while demonstrating robustness to prompt permutations and class imbalance. Our findings establish that diversity in initial retrieval is more critical than ranking-stage optimization, and limiting demonstrations to 3-5 examples maximizes performance efficiency.
Forward citations
Cited by 1 Pith paper
-
Quantized Large Language Models in Biomedical Natural Language Processing: Evaluation and Recommendation
Quantizing LLMs to 4 or 8 bits cuts GPU memory by up to 75% with generally small performance changes across eight biomedical NLP benchmarks.
This paper was first reviewed by deepseek-v4-flash on August 5, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.