{"id":"3013b843-f88c-4da5-8081-b333e35d32c5","arxiv_id":"2504.14508","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A max-coverage graph algorithm with an adaptive similarity threshold selects 10-30% of synthetic data that trains classifiers as well as or better than the full dataset on sentiment, relation extraction, and NER tasks.","lead":"The paper introduces Adaptive Coverage Sampling (ACS), a graph-based method that picks a small, diverse subset of LLM-generated training data. Training a BERT classifier on 10-30% of this curated subset matches or beats training on the full synthetic dataset on three NLP tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Coverage target c=0.9 is selected on the SST2 test set (Section 4.2) and then evaluated on that same test set (Section 5.1), making the reported 'less is more' gains potentially circular and the central empirical claim unestablished.","rationale":"The reader's verdict is CONDITIONAL and already lists test-set tuning among the issues, but identifies the monotonicity of greedy coverage as the weakest assumption. I agree that the monotonicity gap is a genuine transferability risk: Theorem 3.2 only guarantees monotonicity for exact max cover, the paper uses greedy approximation, and the empirical validation is limited to three datasets. However, the paper explicitly acknowledges this gap and provides empirical evidence on the evaluated datasets, so it does not invalidate the reported results for those datasets. The test-set tuning of c is more directly load-bearing because it affects the validity of the headline numbers themselves. Section 4.2 tunes c by evaluating on the human-annotated test set, and Section 5.1 reports F1 on that same test set using the resulting c=0.9. This circular evaluation means the reported superiority over full-data training and baselines could be an artifact of overfitting the coverage hyperparameter to the test set. A concrete validation-split experiment would settle whether the claim survives unbiased evaluation. The abstract also overstates the results as 'superior' when Section 5 shows 'comparable' for several regimes, but the test-set tuning is the more serious flaw. Since the concern is addressable by re-running with a validation split, the verdict remains CONDITIONAL rather than moving to REJECT, hence UNCHANGED.","tokens_in":14516,"tokens_out":9376,"duration_ms":83994,"concrete_test":"Re-run the SST2 experiment (Section 5.1) with coverage target c selected on a held-out validation split rather than the test set: split the 6,000 synthetic training samples into 80% train and 20% validation, tune c ∈ {0.5,0.6,0.7,0.8,0.9,1.0} on the validation F1, then report F1 on the original human-annotated test set for ACS at 10%, 20%, ..., 100% subset sizes, with mean and standard deviation over the 5 seeds. If the validation-tuned ACS at 10% no longer matches or exceeds full-data training, the central claim is an artifact of test-set tuning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is empirical: ACS-selected subsets 'achieve superior performance compared to training on the entire dataset.' The evidence for this claim rests on the fine-tuning results in Section 5.1, yet the key hyperparameter of ACS, the coverage target c, is chosen in Section 4.2 by varying c, fine-tuning on synthetic SST2 data, and evaluating accuracy 'on a human-annotated test set.' The appendix (Section A.1) then states that 'full coverage is non-optimal in most instances, further motivating our usages of coverage = 0.9 throughout the experimental results.' The same human-annotated test set is used to report the headline F1 scores in Figure 3. Therefore the coverage level is tuned directly on the test set, and the subsequent comparison on that test set is not an independent evaluation. This is a circular evaluation step: it can inflate ACS's apparent advantage, especially the small margins over full-data training at 10% subset size (0.8280 vs 0.8176 for SST2). Without selecting c on a held-out validation split, the reported 'less is more' result is not established by the paper's own experiments. The monotonicity gap identified by the reader is real but is explicitly acknowledged and empirically validated on all three datasets; in contrast, the test-set tuning flaw directly undermines the validity of the headline numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Adaptive Coverage Sampling (ACS), a graph-based data-selection method for synthetic text corpora. Synthetic samples are embedded, a similarity graph is built, a binary search sets an edge-weight threshold to hit a target coverage c, and a greedy maximum-coverage algorithm selects k nodes. The authors claim that training BERT on these ACS-selected subsets matches or outperforms training on the full synthetic dataset while using only 10–30% of the data, across SST2 sentiment classification, FewRel relation extraction, and CrossNER named entity recognition. They also provide a monotonicity theorem for the exact max-cover problem, an empirical monotonicity validation for the greedy variant, and an appendix proposing a scalable threshold-transfer procedure.","tokens_in":14731,"tokens_out":3435,"duration_ms":32190,"significance":"If the central claim were cleanly established, ACS would be a practically useful and inexpensive method for reducing synthetic-data fine-tuning cost, with an appealing connection to maximum coverage as a formalization of diversity. The paper has several strengths: it compares against a broad set of baselines (random, EL2N, forgetting, prototypicality, AlpaGasus), it reports both task performance and a SelfBLEU diversity measure, it provides code and hyperparameters in the supplementary material, and it explicitly acknowledges the gap between the exact max-cover monotonicity theorem and the greedy approximation. However, the headline empirical result is currently weakened by a circular choice of the key hyperparameter c on the SST2 test set, by the absence of error bars or significance tests despite the use of five seeds, and by an abstract that overstates performance at small subset sizes for two of the three benchmarks. These issues are fixable but affect the core empirical claim.","major_comments":[{"comment":"The coverage target c=0.9, which is the central hyperparameter of ACS, is selected by tuning on a 'human-annotated test set' for SST2 in Section 4.2. Appendix A.1 then states that 'full coverage is non-optimal in most instances, further motivating our usages of coverage = 0.9 throughout the experimental results.' The same human-annotated SST2 test set is used to report the headline F1 scores in Figure 3. This makes the 'less is more' finding for SST2 partly circular: c was chosen by optimizing accuracy on the very test set used for evaluation. Please re-run the SST2 evaluation with c selected on a held-out validation split (or report the full accuracy-vs-c curve and evaluate at all c values), and clearly state for each benchmark whether c was tuned on a validation set or fixed a priori.","section":"Section 4.2 and Section A.1"},{"comment":"The abstract's unqualified claim that ACS 'achieves superior performance compared to training on the entire dataset' is contradicted by the paper's own tables at small subset sizes. On FewRel at 10% subset size, ACS achieves 0.2642 F1 versus 0.3729 for full-data training (Figure 4), and on CrossNER at 10%, ACS achieves 0.2502 versus 0.3842 (Figure 5). Even the SST2 at 10% margin (0.8280 vs 0.8176) is small. The claims in Section 5.1 are more careful ('approximately 10%' for SST2, '30%' for FewRel, '20%' for CrossNER), but the abstract and Discussion should be aligned with the actual subset sizes and should not imply superior performance at all sizes.","section":"Abstract and Section 5.1 (Figures 4 and 5)"},{"comment":"The paper states that five random seeds were used for model initialization, but all reported results are single point estimates with no standard deviations, confidence intervals, or significance tests. Several of the claimed advantages over baselines and over full-data training are small (e.g., SST2 at 10%: 0.8280 vs 0.8176; SST2 at 80%: 0.8357 vs 0.8304), and without variance information it is impossible to know whether these differences are meaningful. Please report means and standard deviations, and ideally pairwise significance tests, for the main comparisons.","section":"Section 3.3 and Figures 3–5"},{"comment":"The binary-search procedure's theoretical grounding applies to the exact maximum-coverage solution, but the implementation uses the greedy approximation, which the paper explicitly concedes is not guaranteed to be monotone in the threshold. The empirical monotonicity validation on three datasets is useful, but it does not establish a general guarantee, and on a new dataset the binary search could fail to converge to the intended coverage. This is acknowledged in the text, but it should be stated more prominently as a limitation of the method's theoretical guarantees, not as part of the positive claim that ACS 'ensures theoretical rigor.'","section":"Section 3.2, Theorem 3.2, and Section 4.1"}],"minor_comments":[{"comment":"The text contains several typos and stylistic errors, including 'analgous', 'evalute', 'compliment', 'Hasing', 'threhsold', 'datasests', 'optimial', and 'subsample' used inconsistently. These should be corrected in a revision.","section":"General"},{"comment":"The captions of Figures 6 and 7 describe the right-hand plots as 'for the sentiment analysis tasks,' but the figures show FewRel and CrossNER, respectively. The captions should be corrected.","section":"Appendix A, Figures 6 and 7"},{"comment":"The proof of Proposition B.1 is not fully rigorous as written: S and S' are greedy selections that depend on the random subsample and on each other, so the Hoeffding bound over fixed indicator variables does not directly apply, and the step replacing the expected coverage on V' by the coverage on V needs a clearer argument. The empirical transfer results are suggestive, but the formal statement should be either repaired or removed.","section":"Appendix B, Proposition B.1"},{"comment":"The derivation of d_max via the 'extended pigeonhole principle' is only sketched; please provide the precise statement or a citation, since this constraint affects the graph construction and hence the coverage computation.","section":"Section 3.2"},{"comment":"Figure 3 reports F1 'averaging results over five random initializations' but the table does not indicate whether the displayed numbers are averaged over those seeds or a single run. Please clarify the reporting convention.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The circular use of the SST2 test set to choose c=0.9 is the main barrier to acceptance. The authors should be encouraged to rerun the key experiments with c chosen on a validation split, or at minimum to provide a sensitivity analysis over c that does not touch the test set. The lack of error bars is also a serious concern given the small magnitudes of some claimed gains. If these issues are addressed, the paper could be a useful contribution; the theoretical monotonicity result, while correct, is too weak to carry the method, so the empirical validation must be sound."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: ACS is a reasonable data-selection method, and the idea of using a binary search on the similarity threshold to hit a coverage target is neat. But the paper's central 'less is more' finding is weakened by a circular choice of the coverage parameter c, and the abstract overstates the results. With validation-based tuning and a couple of added benchmarks it could be a solid contribution.\n\nWhat's genuinely new: applying max-coverage coreset selection to synthetic LLM data, with an adaptive threshold via monotone binary search. The monotonicity theorem is correct (though simple: more edges can only increase coverage), and the paper is honest that the greedy variant isn't guaranteed monotone and checks it empirically. The empirical setup is fairly clean: Gecko embeddings, BERT fine-tuning, three tasks, five seeds. The scalability appendix (threshold transfer from a subsample) is a nice addition, and the intuition that diversity (lower SelfBLEU) correlates with accuracy is plausible.\n\nSoft spots in order of importance. First, the coverage target c=0.9 is chosen in Section 4.2 by evaluating accuracy on the SST2 test set, and the headline results in Figure 3 are on that same test set. That's circular in the sense that c is tuned to the test set, so the comparison to full-data training is not independent. Fixing c=0.9 a priori and comparing ACS to baselines at that fixed value is still a fair method comparison, but the claim 'matches full data at 10%' is partly built from test-set information. The authors should use a validation split for selecting c and then report test scores once.\n\nSecond, no error bars or variance estimates anywhere, despite five seeds being mentioned. At 10% on SST2, ACS 0.8280 vs full 0.8176 is a small margin; without variance it's hard to know if that's meaningful. Third, the abstract's 'superior performance' is false on the paper's own tables: at 10% on FewRel (0.2642 vs 0.3729) and CrossNER (0.2502 vs 0.3842), ACS is far below full-data training. The introduction's 'match or outperform in specific regimes' is the accurate phrasing. Fourth, the closest coverage-based baseline, [57], is cited but never compared. That seems like a miss, since it directly targets the same kind of method. Fifth, the scalability proposition (B.1) has loose notation and a somewhat hand-wavy expectation step; it's in the appendix and not load-bearing, but worth tightening.\n\nWho should read this: people doing synthetic data curation or data pruning for fine-tuning. It's a useful idea with a flawed demonstration. I'd send it to peer review, but the authors need to re-run with proper validation, add error bars, and tone down the abstract.","headline":"A sensible data-selection method whose headline claim is compromised by tuning the coverage parameter on the test set; fix that and this becomes a solid, citable result.","tokens_in":15349,"tokens_out":4540,"would_cite":false,"duration_ms":36537,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adaptive Coverage Sampling claims that training a classifier on a semantically diverse subset of synthetic data—often 10–30% of the corpus—matches or beats training on the full dataset.","keywords":["synthetic data","data selection","maximum coverage","coverage sampling","data pruning","LLM-generated data","text classification","named entity recognition"],"falsifier":"Take a synthetic corpus in a new domain, compute ACS's coverage curve over a grid of similarity thresholds, and fine-tune on both the ACS subset and a random subset of equal size: if the coverage curve ever decreases when the threshold is lowered, or if the ACS subset does not beat the random subset, the central claim fails for that setting.","tokens_in":14268,"feed_emoji":"🎯","tokens_out":7480,"duration_ms":63076,"temperature":0.7,"pith_summary":"The paper claims that the redundancy built into LLM-generated training corpora is not just waste—it actively hurts downstream classifiers. It introduces Adaptive Coverage Sampling (ACS), which embeds synthetic samples, builds a similarity graph, and selects a small subset whose neighborhoods cover the rest of the graph. Trained on that subset, a BERT classifier matches or beats the same model trained on the full synthetic corpus using roughly 10–30% of the data on three NLP tasks: SST2 sentiment, FewRel relation extraction, and CrossNER named entity recognition. If this holds, data curation becomes a cheap graph-optimization step rather than repeated training runs or LLM-based rating, and fine-tuning cost drops sharply.","feed_headline":"A 10% slice of synthetic data can match or beat full-data training","feed_subtitle":"A graph-coverage sampler matches full-corpus fine-tuning using 10–30% of the data on three NLP tasks.","key_machinery":"The load-bearing object is the coverage statistic on a similarity graph: a selected set H ⊆ V of k samples covers fraction c of the data when the union of the closed neighborhoods of H equals c|V|. ACS engineers this statistic with three components: a text-embedding step that turns sentences into vectors, a binary search over the cosine-similarity threshold that tunes how many edges survive pruning, and a greedy max-coverage routine that repeatedly picks the highest-degree uncovered node. A per-node degree cap d_max > cN/k keeps the graph sparse. The monotonicity theorem (coverage cannot shrink when the threshold is lowered) is what licenses the binary search; the greedy approximation is the practical stand-in whose empirical monotonicity is verified in Section 4.1.","core_discovery":"ACS treats data selection as maximum coverage on a weighted similarity graph: each sample is a node, edges connect semantically similar samples, and a subset of k nodes is chosen so that the union of their neighborhoods covers a target fraction c of the corpus. Because coverage is monotone in the edge threshold for an exact max-cover solution (Theorem 3.2), a binary search can find the loosest similarity threshold that still hits the target coverage; the paper then runs a greedy approximation in practice and verifies monotonicity empirically on all three datasets. The central empirical discovery is that full coverage, c = 1, is not optimal: accuracy peaks just below full coverage, and ACS-selected subsets at roughly 90% coverage match or exceed full-corpus training with 10% of SST2, 30% of FewRel, and 20% of CrossNER. The selected subsets also show higher diversity (lower SelfBLEU), which the paper identifies as the mechanism connecting coverage sampling to better generalization.","pith_inferences":["An implication the paper leaves implicit: the same coverage search applies to human-labeled data and unlabeled corpora, since redundancy is not specific to LLM output; target coverage would become a tunable hyperparameter there.","A caveat the paper only partially addresses: monotonicity is proven for exact max cover, so a practitioner should plot coverage versus threshold on a new embedding before trusting the binary search; a non-monotone curve would silently select the wrong threshold.","The appendix's threshold-transfer result implies a scalable recipe the paper does not push: tune the threshold on a small random subsample and reuse it on the full graph, which combined with approximate nearest-neighbor construction should scale ACS to millions of samples.","Since ACS optimizes breadth while difficulty-based filters optimize informativeness, the two criteria could be composed—first cover, then score within covered regions—rather than treated as competing baselines."],"forward_implications":["On SST2, ACS matches full-corpus fine-tuning using only about 10% of the synthetic data; on FewRel the comparable fraction is about 30%, and on CrossNER about 20%.","Across nearly all subset sizes on all three tasks, ACS beats random selection, EL2N, forgetting scores, prototypicality, and an LLM-rater baseline (AlpaGasus).","Subsets chosen by ACS have lower SelfBLEU scores, so the accuracy gain is tied to higher diversity rather than to hardness or proximity to class centroids.","Because selection never uses labels and requires no repeated model training, the same pipeline applies to unlabeled synthetic pools and can be dropped next to any embedding method.","Setting target coverage below 1.0 (the paper uses 0.9) is consistently better than full coverage, so pruning redundant samples is part of the benefit, not a compromise."],"supporting_citations":[{"why":"supplies the GPT-3.5 synthetic corpora for SST2 and FewRel and the prompt-generation setup.","marker":"[11]"},{"why":"provides the greedy max-coverage approximation that ACS runs in practice.","marker":"[24]"},{"why":"provides the Gecko embeddings used to build the similarity graph from samples.","marker":"[29]"},{"why":"defines the BERT base model fine-tuned in every downstream evaluation.","marker":"[9]"},{"why":"defines the SST2 sentiment benchmark that the synthetic corpus emulates and that supplies the human test set.","marker":"[46]"},{"why":"defines the FewRel relation-extraction benchmark and its 64-label test structure.","marker":"[20]"},{"why":"defines the CrossNER AI-domain NER benchmark used for the token-level evaluation.","marker":"[33]"},{"why":"supplies the SelfBLEU metric used to measure and explain the diversity of selected subsets.","marker":"[58]"},{"why":"is the AlpaGasus LLM-rater baseline that ACS is compared against as a filtering alternative.","marker":"[6]"},{"why":"is the EL2N difficulty-scoring baseline that ACS outperforms in the comparisons.","marker":"[38]"}],"fun_headline_variants":["Less data, better accuracy: coverage sampling wins on NLP","Adaptive coverage picks 10-30% of synthetic data, beats full set","Why less synthetic data can train better classifiers","Coverage-based sampling cuts training data, boosts accuracy","Trimming synthetic data to 10% improves NLP model fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The binary search assumes coverage increases monotonically as the similarity threshold is lowered; the paper proves this only for an exact max-cover solution and relies on three empirical datasets to show the greedy approximation behaves the same way, so a new corpus with a non-monotone coverage curve would break the threshold search.","fun_headline_variants_meta":{"raw":{"variants":["Less data, better accuracy: coverage sampling wins on NLP","Adaptive coverage picks 10-30% of synthetic data, beats full set","Why less synthetic data can train better classifiers","Coverage-based sampling cuts training data, boosts accuracy","Trimming synthetic data to 10% improves NLP model fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000347,"raw_usage":{"total_tokens":1871,"prompt_tokens":889,"completion_tokens":982,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":898}},"tokens_in":505,"tokens_out":982,"duration_ms":8169,"temperature":1.0,"reasoning_tokens":898,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:46:50.158704+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a synthetic corpus in a new domain, compute ACS's coverage curve over a grid of similarity thresholds, and fine-tune on both the ACS subset and a random subset of equal size: if the coverage curve ever decreases when the threshold is lowered, or if the ACS subset does not beat the random subset, the central claim fails for that setting.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the greedy max-coverage approximation that ACS runs in practice."},{"cited_title":"D., Ng, A","cited_arxiv_id":null,"evidence_quote":"defines the SST2 sentiment benchmark that the synthetic corpus emulates and that supplies the human test set."},{"cited_title":"Crossner: Evaluating cross-domain named entity recognition","cited_arxiv_id":null,"evidence_quote":"defines the CrossNER AI-domain NER benchmark used for the token-level evaluation."},{"cited_title":"Texygen: A benchmarking platform for text generation models","cited_arxiv_id":null,"evidence_quote":"supplies the SelfBLEU metric used to measure and explain the diversity of selected subsets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"is the EL2N difficulty-scoring baseline that ACS outperforms in the comparisons."}],"review_version":1}