{"id":"6ded5e53-ca8d-4623-a96f-9679f74df375","arxiv_id":"2506.07169","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A noise- and redundancy-aware instance selection framework reduces transformer training sets by 41% on average while maintaining classification effectiveness, according to this dissertation summary.","lead":"This dissertation summary reports that an instance selection framework, biO-IS, cuts training data for transformer text classifiers by 41% on average without hurting accuracy. It matters because fine-tuning large language models is expensive, and the results point to cheaper, more sustainable NLP.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that biO-IS 'maintains effectiveness in all datasets' is inferred from failing to reject a difference under low-power paired t-tests with Bonferroni; absence of evidence is not equivalence, and no per-dataset effect sizes or margins are given.","rationale":"The reader's weakest assumption focuses on whether a weak classifier's confidence and entropy are reliable proxies for redundancy and noise in transformer fine-tuning. That is a real risk, but it is ultimately validated or falsified by the measured end-to-end effectiveness on these datasets. The more immediately load-bearing point is whether the reported measurements actually support the positive claim of 'same effectiveness.' The described protocol, a paired t-test with Bonferroni and k=5/10 folds, can only detect large and consistent differences; after Bonferroni, failing to reject is weak evidence for equivalence. This is not a disagreement with consensus; it is a question of what the paper's own numbers license. The linked dissertation, TOIS paper, and GitHub repository are genuine supporting artifacts and may contain the per-dataset confidence intervals and equivalence tests that would resolve the issue. If they do, the conditional verdict stands; if they do not, the headline claim should be reworded as 'no statistically significant difference was detected' rather than 'maintains same levels.' Since the reader already returned CONDITIONAL and my proposed check is exactly the kind of condition that would settle it, I recommend no change to the verdict.","tokens_in":13608,"tokens_out":9443,"duration_ms":110731,"concrete_test":"Using the per-dataset fold-level outputs available in the linked GitHub repository or dissertation, compute for each of the 22 datasets the paired MacroF1 difference (biO-IS selected subset minus full training) and its 95% confidence interval. Pre-specify an equivalence margin, e.g., -1.0 MacroF1 point or the largest loss the authors would deem acceptable. Apply a two one-sided equivalence test (TOST) per dataset. If any dataset's confidence interval extends below -margin, the 'maintains effectiveness in all datasets' claim is not supported; report the count of datasets outside the bound. This check distinguishes a genuine equivalence result from a merely non-significant difference.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sections 5 and 7 ground the central claim on a statistical protocol that cannot support the positive assertion of 'same effectiveness.' Section 5 ('Metrics and Experimental Protocol') says comparisons use a paired t-test at 95% confidence with Bonferroni correction, and Section 7 states that biO-IS 'maintained the same levels of effectiveness in all of the considered datasets.' That phrase is supported only by failing to reject the null of equal means. With k=5 folds for large datasets (k=10 for small), the t-test has 4 or 9 degrees of freedom, and Bonferroni over 22 datasets lowers the per-test threshold to about 0.0023; real MacroF1 drops of 1-2 points can easily go undetected. No equivalence margin, confidence interval, or per-dataset effect size is reported in this text. The issue is compounded because the beta reduction rate is itself selected per dataset by a weak classifier on a validation set: the 41% average reduction is an optimized outcome, not a fixed-method property. If the weak model's safe-rate estimate is even slightly optimistic, the low-power test can conceal the resulting transformer degradation. Section 6's footnote 5 and Section 7 also defer key proxy validation to the dissertation ('we test and confirm this premise in our Dissertation'), which is relevant but does not by itself turn a non-significant difference into an equivalence result. The central claim therefore rests on an inference that the described protocol cannot establish.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript is a condensed PhD dissertation summary (CTDGSI) on instance selection (IS) for automatic text classification (ATC). It reports a systematic literature review and comparative evaluation of 13 traditional IS methods combined with transformer-based classifiers on 22 datasets, and proposes two frameworks: E2SC, a redundancy-oriented confidence-based instance selection method, and biO-IS, an extension that additionally removes noisy instances using an entropy-based criterion. The central claims are that biO-IS reduces training sets by about 41% on average (up to 60%), removes 66.6% of manually inserted noise, maintains the same effectiveness in all 22 datasets, and achieves average speedups of 1.67x (up to 2.46x).","tokens_in":13931,"tokens_out":3316,"duration_ms":36831,"significance":"If the reported results are reliable, this work is practically significant: it demonstrates that instance selection can substantially reduce the cost of fine-tuning transformer text classifiers without degrading MacroF1, and it extends the IS literature from small tabular data to large, high-dimensional text collections. The manuscript is grounded in several peer-reviewed publications, provides a broad benchmark (22 datasets) and a proposed taxonomy, and includes pointers to code and data for reproducibility. Its main weakness is that the statistical evidence in this text does not support the positive claim of 'maintaining the same effectiveness' in all datasets, and the per-dataset results needed for verification are not included here.","major_comments":[{"comment":"The claim that biO-IS 'maintained the same levels of effectiveness in all of the considered datasets' is supported only by failing to reject the null hypothesis in paired t-tests with Bonferroni correction. With k=5 folds for large datasets and k=10 for small ones, the tests have 4 or 9 degrees of freedom, and Bonferroni over 22 datasets lowers the per-test threshold to roughly 0.0023; real MacroF1 drops of 1-2 points are unlikely to be detected. To support a positive equivalence claim, the authors should report per-dataset effect sizes, confidence intervals, or an explicit equivalence margin (e.g., a TOST or confidence-interval-based test), rather than only non-significance.","section":"Section 5 ('Metrics and Experimental Protocol') and Section 7 ('Experimental Results')"},{"comment":"The manuscript reports only aggregate point estimates for biO-IS (average reduction 40.1%, speedup 1.67x, 66.6% noise removal) and does not provide a per-dataset table of MacroF1, reduction rates, or speedups. Since the reduction rate beta is selected per dataset by a weak classifier on a validation set, the 41% average reduction is an optimized outcome whose stability is unknown. The authors should provide the per-dataset breakdown and the distribution/variance of the selected beta values so that readers can assess whether the average reduction is a robust property of the method or an artifact of per-dataset tuning.","section":"Section 7 ('Experimental Results') and Section 6 ('Experimental Results')"},{"comment":"The core mechanisms of biO-IS rely on the assumption that a weak classifier's confidence and entropy are reliable proxies for redundancy and noise for transformer fine-tuning. The text states that this premise is 'tested and confirmed' in the dissertation, but the details are not provided in this manuscript. Since this assumption is load-bearing for the central effectiveness claim, the authors should include a concrete validation experiment in this text (for example, comparing the weak model's predicted safe reduction rate with the transformer's actual MacroF1 across a range of beta values) or explicitly mark this as an untested limitation with reference to the dissertation's evidence.","section":"Section 6 (footnote 5) and Section 7 (weak classifier description)"}],"minor_comments":[{"comment":"The reduction and speedup numbers are inconsistent across the text: the abstract and conclusion say 41% reduction, Section 7 says 40.1%; Section 6 says E2SC achieved 27% average reduction, while Section 8 says 30%; Section 8 reports 'speedups of up to 70%' while Section 7 reports 1.67x average and 2.46x maximum. These should be reconciled.","section":"Abstract, Section 7, Section 8"},{"comment":"The formula for reduction R is typeset incorrectly: 'R = P k i=0 |Ti|-|Si| / k' is malformed and should be written with a summation index, e.g., R = (1/k) * sum_i (|T_i|-|S_i|)/|T_i|.","section":"Section 5 ('Metrics and Experimental Protocol')"},{"comment":"The text lists six transformer models (RoBERTa, BERT, DistilBERT, BART, AlBERT, XLNet) but then refers to 'the best of seven deep learning text classification methods'; the count should be made consistent.","section":"Section 5 ('Text Classification Methods')"},{"comment":"Several Portuguese labels remain in an English manuscript, such as 'Figura 1' and 'Tabela 1', and the references heading is 'Referências'; these should be translated or unified.","section":"Throughout"},{"comment":"Minor language and typographical issues include 'inspection-ed' (Section 4), 'effectivly' (Section 8), and inconsistent use of italics for method names; a careful proofread is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"This manuscript is a thesis summary for a Brazilian computing competition rather than a full research paper, and it leans heavily on the authors' previously published papers for methodological and per-dataset details. The editor may wish to evaluate whether the condensed format is appropriate for the venue, but the central claims are plausible and do not appear internally inconsistent; the main issue is that the statistical protocol and reporting in this text are insufficient to support the strongest version of the effectiveness claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hey,\n\nQuick take: this arXiv paper is a dissertation summary (for a Brazilian thesis contest) of work the authors already published in SIGIR 2023, TOIS 2024, and ACM Computing Surveys. There are no new results here. But as a compact entry point to a solid line of work on instance selection for transformer fine-tuning, it does the job.\n\nWhat's genuinely good: the paper brings together a head-to-head comparison of 13 classical and recent instance-selection methods on 22 text classification datasets under transformer classifiers. That comparison and the extended taxonomy are useful, and they're backed by the CSUR survey. The two proposed frameworks, E2SC and biO-IS, reduce training sets by roughly 40% (up to 60%) while keeping Macro-F1 and giving speedups around 1.7x. Those numbers come from extensive experiments (4,000+ runs, 5,600 hours) and have already survived peer review at strong venues. So the core empirical results are believable.\n\nWhere it's soft: the stress-test note is right on target. The paper claims biO-IS \"maintained the same levels of effectiveness in all datasets,\" but the supporting evidence in this text is a paired t-test with 95% confidence and Bonferroni correction over 22 datasets. Failing to reject a difference is not the same as showing equivalence. With 5 or 10 folds, the tests have very few degrees of freedom, and a real 1–2 point Macro-F1 drop could easily go unnoticed. No effect sizes, confidence intervals, or equivalence margins are reported here. The reduction rate is also tuned per dataset on a validation set, so the average 41% reduction is an optimized outcome rather than a fixed-method property. The paper's own footnotes defer details to the dissertation, which is honest but means this text alone cannot independently verify the strongest claims. The reliance on a weak classifier's confidence and entropy as proxies for redundancy and noise is another assumption worth flagging, again deferred to the dissertation.\n\nWho is this for? Someone who wants a fast overview of the state of the art in instance selection for ATC, or someone looking for pointers to the underlying papers. It is not the place to cite for the methods—cite the SIGIR and TOIS papers instead.\n\nMy recommendation: if this crossed my desk as a submission, I'd send it to review, but the reviewer's job would be to check the summary against the prior work and to demand that the equivalence claims be rephrased or backed by proper equivalence testing. The underlying research is valuable; this summary is a passable, if slightly overconfident, advertisement for it.","headline":"Useful condensed summary of an already-published, genuinely interesting line of work; the 'same effectiveness' claim is not supported by the statistical protocol described here.","tokens_in":14445,"tokens_out":3961,"would_cite":false,"duration_ms":37507,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A bi-objective instance-selection framework prunes redundant and noisy training documents for transformer text classifiers, cutting training sets by 41% on average (up to 60%) while preserving effectiveness across all 22 datasets and…","keywords":["instance selection","text classification","transformer fine-tuning","data reduction","noise removal","redundancy","calibrated classifiers","training set pruning"],"falsifier":"Compare transformer fine-tuning on (a) biO-IS's selected subset, (b) a random subset of the same size, and (c) the discarded instances, across several datasets. If the random subset achieves the same MacroF1 as the selected subset within the statistical margin, the confidence/entropy signal is not actually carrying the redundancy/noise information. A second check: if the weak classifier's reduction-effectiveness curve on the validation set predicts no loss at a reduction rate where the transformer's accuracy drops significantly, the surrogate-assumption fails.","tokens_in":13427,"feed_emoji":"📉","tokens_out":5830,"duration_ms":50769,"temperature":0.7,"pith_summary":"The dissertation claims that instance selection—picking a subset of training documents before fine-tuning—can reduce transformer text-classification training sets by 41% on average (up to 60%) without changing model effectiveness across 22 datasets, while removing most injected noise and speeding up model construction by 1.67x on average (up to 2.46x). Two frameworks are introduced: an earlier redundancy-only selector, E2SC, and the extended biO-IS, which handles redundancy and noise together. The central message is that large transformer classifiers can be fine-tuned on considerably less data, and a cheap weak classifier can identify which documents to drop.","feed_headline":"Training sets shrink 41% with no accuracy loss","feed_subtitle":"A selection method prunes redundant and noisy documents before transformer fine-tuning, cutting time and cost.","key_machinery":"The load-bearing object is the confidence- and entropy-weighted removal probability. A calibrated weak classifier (logistic regression) assigns each instance a confidence; confident correct predictions are treated as redundant, and for mispredicted instances the inverse entropy of the posterior distribution estimates noise likelihood—low-entropy wrong answers are confidently wrong and thus more likely noise. These two signals define an α-weighting, and an iterative statistical comparison of weak-model effectiveness with and without reduction sets the β reduction rate. The final training set is a random sample of size (1−β) weighted by α, so removal is probabilistic and tuned to preserve the deep model's expected behavior.","core_discovery":"biO-IS is a bi-objective instance-selection framework that removes both redundant and noisy training instances prior to transformer fine-tuning. It uses logistic regression as a calibrated weak classifier: high-confidence correct predictions mark easy, redundant documents, while low-entropy incorrect predictions mark likely noise; an iterative validation procedure estimates a near-optimal reduction rate using the weak model's effectiveness as a surrogate for the transformer's. In experiments across 22 topic and sentiment datasets, biO-IS reduced training sets by 40.1% on average (29–60%), removed 66.6% of manually inserted noise, preserved MacroF1 on every dataset, and delivered mean speedups of 1.67x (up to 2.46x), outperforming its predecessor E2SC and all tested baselines on the reduction-efficiency-effectiveness trade-off.","pith_inferences":["If the confidence/entropy proxy generalizes, the same pruning could be applied to fine-tune larger language models, lowering the compute barrier for custom classifiers in low-resource settings.","The framework could be coupled with active learning: the low-confidence, high-entropy instances it retains are precisely the ones a human annotator would most usefully label.","A testable extension is moving from whole-document selection to passage- or aspect-level selection, since noisy or redundant content may live inside documents rather than in entire documents.","The reported noise-removal result comes from simulated injected noise; real-world label noise may follow different distributions, so the 66.6% figure should be tested on naturally noisy labels."],"forward_implications":["Training sets for transformer fine-tuning can be cut by roughly 40% without lowering MacroF1 on the tested benchmarks.","Model construction time drops by 1.67x on average, with larger gains on larger datasets, making frequent re-training more feasible.","Traditional IS methods rarely improve effectiveness, but biO-IS removes a large fraction of injected noise, a capability no tested baseline matched.","The results support the dissertation's hypothesis that smaller, well-chosen training data can replace large datasets for many text classification tasks."],"supporting_citations":[{"why":"Introduces the E2SC redundancy-selection framework that biO-IS extends, supplying the confidence-based selection and reduction-rate estimation mechanism.","marker":"[Cunha et al. 2023b]"},{"why":"The journal paper presenting biO-IS, the noise-oriented and redundancy-aware framework whose numbers are reported here.","marker":"[Cunha et al. 2024]"},{"why":"The comparative survey and taxonomy of 13 IS methods that provides the empirical baseline context for the tripod constraints.","marker":"[Cunha et al. 2023c]"},{"why":"The earlier instance-selection taxonomy that the paper extends, framing the traditional method families.","marker":"[Garcia et al. 2012]"},{"why":"Defines Condensed Nearest Neighbor, one of the baseline IS methods compared against.","marker":"[Hart 1968]"},{"why":"Defines Edited Nearest Neighbor, another baseline IS method compared against.","marker":"[Wilson 1972]"},{"why":"Defines Drop3 and reduction techniques used as additional baselines.","marker":"[Wilson and Martinez 2000]"},{"why":"Documents human labeling error rates (56–64%), motivating the noise-removal objective.","marker":"[Martins et al. 2021]"},{"why":"Supplies MacroF1, the effectiveness metric used throughout the experiments.","marker":"[Sokolova and Lapalme 2009]"}],"fun_headline_variants":["41% less training data, same classification accuracy","Shrink training sets 41% and keep MacroF1 intact","Data-pruning method cuts training size 41% without loss","biO-IS: train on 41% fewer examples, identical accuracy","Prune noisy and redundant text, cut training set 41%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that a cheap classifier's confidence and entropy faithfully identify which documents are redundant or noisy for a much larger transformer model, and that the weak model's behavior under reduction predicts the transformer's behavior; if these proxies misalign with what transformer fine-tuning needs, the selected subsets could hurt accuracy despite the reported results.","fun_headline_variants_meta":{"raw":{"variants":["41% less training data, same classification accuracy","Shrink training sets 41% and keep MacroF1 intact","Data-pruning method cuts training size 41% without loss","biO-IS: train on 41% fewer examples, identical accuracy","Prune noisy and redundant text, cut training set 41%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000806,"raw_usage":{"total_tokens":3563,"prompt_tokens":990,"completion_tokens":2573,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":2485}},"tokens_in":606,"tokens_out":2573,"duration_ms":17775,"temperature":1.0,"reasoning_tokens":2485,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:39:55.340847+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare transformer fine-tuning on (a) biO-IS's selected subset, (b) a random subset of the same size, and (c) the discarded instances, across several datasets. If the random subset achieves the same MacroF1 as the selected subset within the statistical margin, the confidence/entropy signal is not actually carrying the redundancy/noise information. A second check: if the weak classifier's reduction-effectiveness curve on the validation set predicts no loss at a reduction rate where the transformer's accuracy drops significantly, the surrogate-assumption fails.","supporting_citations":[{"cited_title":"A Noise-Oriented and Redundancy-Aware Instance Selection Framework","cited_arxiv_id":null,"evidence_quote":"The journal paper presenting biO-IS, the noise-oriented and redundancy-aware framework whose numbers are reported here."}],"review_version":1}