{"id":"27421a1f-bbc4-469e-a4ee-6393dd9f137e","arxiv_id":"2411.10896","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"RoSE, a validation-set-based filter that restricts active learning acquisition functions to estimated negative flips, improves accuracy and/or reduces negative flip rates on several image benchmarks.","lead":"This paper introduces RoSE, an active learning add-on that first guesses which unlabeled images the current model would get wrong, then asks the usual sampling rule to choose only from those candidates. It reports that this cut improves accuracy and reduces 'negative flips', cases where a new model breaks predictions the old model got right.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RoSE's subset size k=(1-acc_val)|D_pool| estimates total misprediction rate, not negative flip prevalence, so q_phi may not actually target negative flips; the claimed mechanism needs direct validation.","rationale":"The reader's weakest_assumption points to Equation (5) and the energy-ranking step as the load-bearing component, and my analysis agrees: the subset size heuristic equates validation misprediction rate with the number of negative flips, which is not conceptually justified. I additionally note that Equation (4) is internally inconsistent because both-wrong samples with different wrong predictions are counted as prediction switches, so the search-space reduction does not isolate SP ∪ SN as claimed. This does not by itself falsify the empirical results, but it means the paper's explanation of why RoSE works is not established. The proposed concrete test directly measures whether RoSE's selected set actually matches negative flips and whether the threshold size matters; this is a feasible experiment using labels that the authors already use in the idealized setup. The reader's CONDITIONAL verdict is appropriate: the central claim is plausible and partially supported by the idealized experiment, but the estimation mechanism needs verification. A REJECT would be premature because the empirical gains might still be real and reproducible, and the test could validate the mechanism. Therefore I recommend no change to the reader's verdict.","tokens_in":13589,"tokens_out":5982,"duration_ms":65782,"concrete_test":"On CIFAR100 with entropy sampling, use the training labels to compute the true negative flip set NF_t in each active learning round, as in the idealized experiment of Section IV-B. Record RoSE's selected set S_t of k samples from SPN. Report precision |S_t ∩ NF_t|/|S_t| and recall |S_t ∩ NF_t|/|NF_t| over rounds, and compare accuracy/NFR against (i) the baseline, (ii) RoSE with oracle k = |NF_t|, and (iii) random selection from SPN of size k. If precision/recall is low, or if oracle-k does not change the results, the threshold mechanism is not what drives the reported gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism of RoSE is that q_phi(D_pool) approximates the set of negative flips. Equation (5) sets k=(1-acc_val)|D_pool|, which is the number of samples the current model mispredicts on the validation set, scaled to the pool size. Negative flips are a different quantity: they are samples correctly classified by the previous model and misclassified by the current model. Their prevalence is not determined by validation accuracy, and the paper provides no argument that k matches the number of negative flips in SPN. Additionally, Equation (4) is not a correct partition: a 'both wrong' sample whose two wrong predictions differ has a prediction switch, so it falls into SPN even though it is not a negative flip. Selecting the top-k energy samples from SPN can therefore include many both-wrong samples and exclude true negative flips when k is miscalibrated. The empirical gains in Section VI could arise from selecting any misprediction-heavy switch subset, or from the extra validation labels, rather than from specifically targeting negative flips. The paper never measures the overlap between q_phi(D_pool) and the true negative flip set, so the stated mechanism is unsupported despite the favorable learning curves.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies negative flips in active learning: samples that were correctly predicted by the previous model and become mispredicted after retraining on a larger labeled set. The authors make two observations: (i) negative flip rate and accuracy are decoupled, and (ii) restricting the acquisition search pool to negative flips improves accuracy and/or reduces negative flips. They propose RoSE, a plug-in that first restricts the unlabeled pool to prediction-switch samples and then selects a subset of size k=(1-acc_val)|D_pool| using energy-based misprediction scores, where acc_val is the accuracy on a small labeled validation set. Experiments on CIFAR100, CINIC10, and TinyImageNet with seven acquisition functions and two architectures report that RoSE matches or improves the baseline in most rounds, in both accuracy and negative flip rate.","tokens_in":13767,"tokens_out":4422,"duration_ms":46051,"significance":"If the claimed results hold, RoSE would be a useful, model-agnostic plug-in that addresses an underexplored failure mode of active learning, and it would give validation sets a new role beyond early stopping and hyperparameter tuning. The oracle experiment in Section IV-B is a well-designed control that directly supports the premise that sampling from the true negative-flip subset can improve both accuracy and NFR. The paper also provides a broad empirical study across datasets, acquisition functions, and architectures, and the authors release code, which aids reproducibility. However, the significance is conditional on two unresolved issues: the experimental comparison does not hold the label budget equal between RoSE and the baselines, and the proposed estimator is not actually shown to recover negative flips rather than generic mispredictions. These issues directly affect the interpretation of the reported gains.","major_comments":[{"comment":"The label budget is not held equal between RoSE and the baselines. RoSE uses a labeled validation set (1% for CINIC10, 10% for CIFAR100, 5% for TinyImageNet) to compute acc_val and set the threshold k, while the baseline acquisition functions are given no access to these labels. Active learning performance is highly sensitive to the amount of labeled data, so the reported improvements in Table I, Table II, and the learning curves could be partly or entirely due to the extra validation labels rather than to the negative-flip restriction. The authors should either add the validation labels to the baseline training sets, or use the same validation set for all methods (e.g., for early stopping or hyperparameter selection) and explicitly account for its label cost in the comparison.","section":"Section VI-A, Tables I-II, Figures 6-8"},{"comment":"The proposed estimator does not specifically target negative flips. Equation (4) defines SPN as all samples whose predicted label changes between rounds, but this includes both-wrong samples whose two wrong predictions differ; such samples are not negative flips. Equation (5) then sets k=(1-acc_val)|D_pool|, which estimates the total number of mispredicted samples in the pool under the validation accuracy, not the number of negative flips. Negative flips are the subset of switches where the old model was correct and the new model is wrong, and their prevalence is not determined by validation accuracy. The paper never measures the overlap between q_phi(D_pool) and the true negative-flip set, so the central mechanism that RoSE 'restricts acquisition functions to negative flips' is unsupported by direct evidence. The authors should report, for at least one dataset and several rounds, the precision and recall of the estimated set against oracle-computed negative flips, and should include an oracle-k baseline that uses the true number of negative flips to separate the effects of subset ordering from subset size.","section":"Section V-B, Eq. (4)-(5)"},{"comment":"The key calibration assumption that the validation misprediction rate equals the pool misprediction rate is untested and load-bearing. Because k is set from acc_val, any distribution shift between the validation set and the unlabeled pool directly biases the subset size; this is a concrete risk for CINIC10, which combines samples from two distinct sources. In addition, the validation fractions (1%, 5%, 10%) are selected per dataset without sensitivity analysis. The authors should verify the calibration by reporting pool accuracy alongside validation accuracy across rounds, and should vary the validation fraction to demonstrate that the method's performance does not hinge on a particular hand-chosen value.","section":"Section V-B and Section VI-A, Eq. (5)"}],"minor_comments":[{"comment":"The formula for the outperformance ratio is stated as Ntotal/NRoSE, but the surrounding text says a value larger than 0.5 indicates RoSE outperforms in the majority of rounds. This only holds for the inverse ratio NRoSE/Ntotal; the formula or the explanation should be corrected.","section":"Section VI-C"},{"comment":"The set-builder notation in Equation (4) is garbled: 'si = 1 ˜yi new!= ˜yi old = 1' is not a well-formed predicate. The definition of si should be made explicit, for example si = 1 if the predicted label of the new model differs from that of the old model, and the set SPN should be written as {xi in Dpool : si = 1}.","section":"Section V-B, Eq. (4)"},{"comment":"The text states that 'the both correct subset has the lowest NFR' and later that the negative-flip subset 'consistently performs among the highest in terms of accuracy while still reducing the NFR.' It would be helpful to include the corresponding numeric values or a table, since the qualitative description alone makes it hard to judge the magnitude of the differences between the subsets.","section":"Section IV-B, Figure 5"},{"comment":"The abstract says 'integrating a validation set results in a significant performance boost,' but the validation set is only used to compute the scalar acc_val, not as an additional training signal. The wording could be sharpened to avoid implying that the validation labels are added to the training set.","section":"Abstract and Section V-B"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant and timely problem, and the oracle study is a solid motivation. However, the current experimental protocol gives RoSE extra labeled data relative to the baselines, and the estimation step is not validated as a negative-flip estimator. Both issues are fixable with additional experiments and appear within the scope of a major revision rather than a rejection. The authors should also consider benchmarking against baselines that use the same validation labels for other purposes (e.g., early stopping or threshold selection) to ensure a fair comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper is worth knowing about but the headline claim outruns the evidence. The genuinely new parts are the empirical decoupling of negative flip rate (NFR) from accuracy in active learning, and the idea of restricting acquisition to an estimated negative-flip subset. The idealized experiment in Section IV-B is clean evidence that sampling from true negative flips can help both metrics. And they test broadly: seven acquisition functions, three datasets, two architectures, with code published.\n\nThe soft spots are real. First, the comparison is not equal: RoSE gets a labeled validation set (1-10% of the training set) while the baselines get nothing. Some of the measured gains may just be extra label information. The paper should either use the same validation labels for baseline early stopping or report a budget-adjusted comparison. Second, the mechanism itself is underdetermined. Equation (5) sets k from validation accuracy, which estimates the number of current-model mispredictions in the pool, not the number of negative flips. And Equation (4) is not a clean partition: a both-wrong sample with a prediction switch lands in SPN. So the selected set q_phi is a mixture of negative flips and other mispredictions. The paper never measures the overlap between what RoSE selects and the true negative flip set. The observed gains could come from selecting any misprediction-heavy switch subset. They even hint at this in Section VII, where any subset restriction, including both-wrong, concentrates negative flips — which suggests the value may not be specific to negative flips. The \"overwhelmingly\" language also does not survive Table I; several ratios are below 0.5.\n\nNone of this makes the paper useless. The decoupling observation alone is a useful sanity check for active learning practice. The plug-in is simple and modular. But as it stands, the central claim is conditional: RoSE works, but we don't know whether it works because it targets negative flips or because it filters to high-misprediction switch samples while using extra labels.\n\nI'd send this to a serious referee. The right revision would add a direct overlap measurement between q_phi and the actual negative flip set, a fairer baseline, and sensitivity analysis on k and the validation fraction. That is a manageable path to a solid paper.","headline":"Useful empirical study with a plausible plug-in, but the negative-flip mechanism is unmeasured and the evaluation has a label-budget confound.","tokens_in":14364,"tokens_out":2515,"would_cite":false,"duration_ms":24693,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Restricting active learning to estimated negative-flip samples improves accuracy, negative flip rate, or both across benchmarks.","keywords":["negative flips","software regression","active learning","validation set","acquisition function","misprediction detection","energy score","subset selection"],"falsifier":"Take an active learning run where the unlabeled pool labels are available to the experimenter; after each round, compute precision and recall of the estimated negative-flip subset against the true negative flips. If precision and recall are near random, or if replacing the validation-derived $k$ with the true pool misprediction count changes the accuracy and NFR results, the estimation step is not what drives the gains.","tokens_in":1456,"feed_emoji":"🎯","tokens_out":2262,"duration_ms":60850,"temperature":0.7,"pith_summary":"This paper argues that active learning should optimize not only accuracy but also negative flips, meaning samples the model predicted correctly before a training update and incorrectly after it. It shows first that negative flip rate and error rate move independently across active learning rounds, so reducing error does not automatically reduce regression. The authors then propose RoSE, a plug-in that restricts any acquisition function to the estimated negative-flip subset of the unlabeled pool. On CIFAR100, CINIC10, and TinyImageNet with seven acquisition functions and two architectures, RoSE matches or improves accuracy, NFR, or both in most rounds. Because validation sets already exist in active learning pipelines, the method adds only a small computational step to a standard workflow.","feed_headline":"Restrict active learning to negative-flip samples","feed_subtitle":"A plug-in that limits any acquisition function to estimated regression-prone samples lifts accuracy and cuts negative flips.","key_machinery":"RoSE is a two-stage subset estimator followed by an unmodified acquisition step. First, prediction switches between the previous and current model partition the unlabeled pool into the union of positive and negative flips $\\mathcal{S}_{PN}$ and the union of both-correct and both-wrong samples $\\mathcal{S}_{CW}$, using the identity in Equation 4. Second, an energy-based misprediction detector assigns scores to $\\mathcal{S}_{PN}$, and the $k$ samples with the highest scores form the estimated negative-flip subset, with $k = (1-\\text{acc}_{\\text{val}})|\\mathcal{D}_{\\text{pool}}|$ read off the validation accuracy. This turns a small labeled validation set into an estimate of the pool's misprediction count and lets an arbitrary acquisition function operate on a smaller, regression-relevant search space.","core_discovery":"The central claim is that negative flips are a distinct, addressable failure mode in active learning and that sampling from them can be a win-win. Empirically, restricting an acquisition function to samples that switched prediction between rounds and are then scored as mispredictions improves accuracy in many rounds, for example by about two percent for entropy sampling on TinyImageNet, while also lowering the negative flip rate. The method treats the negative flip subset as a smaller search space for any acquisition function, so the same plug-in works with uncertainty-based, representation-based, and Bayesian acquisition functions. The authors also report that when RoSE lowers accuracy in a particular round, it usually lowers NFR as well, providing a trade-off rather than a pure loss.","pith_inferences":["A direct test the paper does not run is to measure precision and recall of the estimated negative-flip subset against true negative flips; this would separate the contribution of the estimation step from the contribution of the prediction-switch partition alone.","The same negative-flip targeting could transfer to other iterative settings such as continual learning or federated learning, where models are updated repeatedly and regression is a known problem; the main obstacle is obtaining a validation set representative of the pool.","The method implicitly assumes the validation distribution matches the unlabeled pool; in label-shift or domain-shift scenarios, recalibrating the threshold k with importance weighting would be a natural extension.","Combining RoSE with training-time regression-reduction methods such as positive-congruent training could attack negative flips simultaneously through data selection and model optimization, a combination the paper does not explore."],"forward_implications":["Wrapping any acquisition function with RoSE yields accuracy and/or NFR gains in most rounds without changing the architecture or training procedure.","RoSE's gains grow in later active learning rounds, when the model is more stable; in early rounds it tends to match the baseline.","When RoSE does reduce accuracy, as with margin sampling on CINIC10, it simultaneously reduces NFR, allowing users to trade one metric for the other.","The gains hold across datasets with different complexity and domain mix, and across both CNN and convmixer architectures."],"supporting_citations":[{"why":"Defines negative flip rate and positive-congruent training; supplies the regression metric that RoSE targets.","marker":"[1]"},{"why":"Energy-based out-of-distribution detection; provides the misprediction score used to rank the negative flip subset.","marker":"[52]"},{"why":"Establishes the use of validation sets in active learning for early stopping; motivates reusing them for regression.","marker":"[13]"},{"why":"Entropy and least confidence acquisition functions; baselines that RoSE wraps and improves in most rounds.","marker":"[18]"},{"why":"BADGE acquisition function; a baseline that RoSE is tested against and improves in most rounds.","marker":"[29]"},{"why":"BALD acquisition function; a Bayesian baseline that RoSE is compared against.","marker":"[25]"},{"why":"BalEntAcq balanced entropy acquisition function; baseline showing RoSE gains on Bayesian methods.","marker":"[31]"},{"why":"PowerBALD; baseline showing cases where RoSE helps NFR or accuracy only partially.","marker":"[33]"}],"fun_headline_variants":["Curb negative flips with validation-driven acquisition","Plug-in steers active learning away from forgetting","Validation set targets regression-prone samples","Reduce negative flips while boosting accuracy","Active learning without forgetting correct predictions"],"cache_read_input_tokens":16384,"weakest_assumption_plain":"The whole method rests on the assumption that the small labeled validation set has the same misprediction rate as the unlabeled pool, so $k=(1-\\text{acc}_{\\text{val}})|\\mathcal{D}_{\\text{pool}}|$ counts the pool's true negative flips, and that the energy score ranks negative flips above positive flips within the prediction-switch set.","fun_headline_variants_meta":{"raw":{"variants":["Curb negative flips with validation-driven acquisition","Plug-in steers active learning away from forgetting","Validation set targets regression-prone samples","Reduce negative flips while boosting accuracy","Active learning without forgetting correct predictions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000464,"raw_usage":{"total_tokens":2312,"prompt_tokens":933,"completion_tokens":1379,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":1316}},"tokens_in":549,"tokens_out":1379,"duration_ms":11968,"temperature":1.0,"reasoning_tokens":1316,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:10:19.813820+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take an active learning run where the unlabeled pool labels are available to the experimenter; after each round, compute precision and recall of the estimated negative-flip subset against the true negative flips. If precision and recall are near random, or if replacing the validation-derived $k$ with the true pool misprediction count changes the accuracy and NFR results, the estimation step is not what drives the gains.","supporting_citations":[{"cited_title":"Positive-Congruent Training: Towards Regression-Free Model Updates","cited_arxiv_id":"2011.09161","evidence_quote":"Defines negative flip rate and positive-congruent training; supplies the regression metric that RoSE targets."},{"cited_title":"Generalization and parameter estimation in feedforward nets: Some experiments,","cited_arxiv_id":null,"evidence_quote":"Establishes the use of validation sets in active learning for early stopping; motivates reusing them for regression."},{"cited_title":"Active Learning in Bayesian Neural Networks with Balanced Entropy Learning Principle","cited_arxiv_id":"2105.14559","evidence_quote":"BalEntAcq balanced entropy acquisition function; baseline showing RoSE gains on Bayesian methods."}],"review_version":1}