{"id":"5e411661-a340-4b0c-9713-a319b032a5ce","arxiv_id":"2501.00818","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"SPARNet combines sample partitioning, generalized entropy minimization, mean-teacher consistency, and an anti-forgetting regularizer to improve continual test-time adaptation on CIFAR10-C, CIFAR100-C, and ImageNet-C.","lead":"SPARNet is a method for adapting a pre-trained image classifier on the fly as it encounters a sequence of corrupted test domains, without labels. It splits test samples into reliable and unreliable groups, then combines entropy-based training, a mean-teacher consistency loss, and a forgetting-prevention regularizer to keep the model accurate over long adaptation runs. The reported gains are a few percentage points over existing continual TTA methods.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The anti-forgetting regularizer, which the ablation shows is the largest contributor, requires source-domain samples Q in Eq. (9), contradicting the paper's stated source-free privacy setting and making the comparison against source-free baselines unfair.","rationale":"The reader's weakest_assumption correctly identifies the source-data contradiction in Eq. (9). This is the single most load-bearing concern because the paper's central claim is that SPARNet achieves strong continual TTA performance in a source-free, privacy-preserving setting, and the ablation attributes the largest part of that performance to the anti-forgetting regularizer, which is computed from source-domain samples Q. The paper is otherwise clearly written, with a complete method description, standard benchmarks, and a plausible sample-partitioning mechanism. However, the contradiction is internal, not merely a disagreement with community consensus: the method definition in Section III.A and the abstract rule out using source data, while Eq. (9) and the implementation details require it. This affects the fairness of every headline comparison, not just a secondary number. The proposed concrete test would directly determine whether the regularizer can be replaced by a target-data estimate without losing the reported gain; if it cannot, the paper must either reposition itself as a source-data-assisted method or the comparison must be redone against baselines that also use source data. I therefore agree with the reader's REJECT verdict and recommend no change to that verdict.","tokens_in":11516,"tokens_out":3695,"duration_ms":36993,"concrete_test":"Re-run the CIFAR10-C continual TTA experiment with the importance Omega_i in Eq. (9) computed from 2000 unlabeled samples drawn from the first target corruption batch instead of from the source-domain set Q, and recompute the Table IV row with GEM+Reg as well as the full method. If the average error moves substantially from 16.0 toward the 27.3 level of GEM+SCE, the reported gain is contingent on source-data access; if the error stays near 16.0, the contradiction is less damaging because the regularizer can be computed without source data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's stated setting is source-free continual TTA: the abstract and introduction say 'the source data is unavailable due to privacy concerns,' and Section III.A defines the problem without using source data during test time. Yet Eq. (9) computes the MAS parameter importance as Omega_i = (1/|Q|) * sum_{q in Q} || d[f_0(x)]^2 / d theta_i ||_2, where Q is explicitly 'the source domain data set.' Section IV.A then reports using 2000 samples to compute Omega. This is not a harmless one-time computation from the deployed source model; it requires direct access to source-distribution samples before deployment. The ablation (Table IV) shows this regularizer is the dominant component: with GEM alone the error is 30.6, with GEM+SCE 27.3, with GEM+Reg (without SCE) 16.2, and with all components 16.0. Thus, when the source-data-dependent Reg term is removed, the remaining proposed mechanism improves the CIFAR10-C error only from 30.6 to 27.3, far from the headline 16.0. Because CoTTA, RoTTA, and AR-TTA do not access source samples, the reported comparisons in Tables I-III are not apples-to-apples. The central claim as written therefore does not hold for a source-free continual TTA setting.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SPARNet, a continual test-time adaptation method that splits incoming test samples into reliable and unreliable groups by prediction entropy. Reliable samples are trained with a generalized entropy-minimization loss, while unreliable samples are used for mean-teacher consistency learning with augmentation-averaged pseudo-labels and symmetric cross-entropy. To prevent catastrophic forgetting, the method adds a Memory-Aware-Synapses-style regularizer that penalizes changes to parameters deemed important for the source task. Experiments on CIFAR10-C, CIFAR100-C, and ImageNet-C report average error rates of 16.0%, 30.6%, and 67.3%, respectively, which the paper claims outperform CoTTA, RoTTA, and AR-TTA. An ablation on CIFAR10-C attributes the largest error reduction to the anti-forgetting regularizer.","tokens_in":11810,"tokens_out":3459,"duration_ms":34616,"significance":"If the reported results were obtained under the stated source-free continual test-time adaptation setting and the comparisons were fair, SPARNet would be a useful contribution: the sample-partitioning idea is simple, the combination of GEM and mean-teacher consistency is reasonable, and the ablation study is informative. The paper also addresses a real problem, namely catastrophic forgetting during long-term adaptation. However, the central claim is undermined by the fact that the dominant component, the anti-forgetting regularizer, requires access to source-domain samples to compute parameter importance, which contradicts the paper's stated privacy motivation and makes the comparison with source-free baselines not apples-to-apples. In addition, the reported gains are not accompanied by error bars, and key hyperparameters are tuned per dataset on the same evaluation benchmark, so the quantitative claims are not yet established at the level expected for this venue.","major_comments":[{"comment":"The anti-forgetting regularizer is computed from Omega_i = (1/|Q|) * sum_{q in Q} || d[f_0(q)]^2 / d theta_i ||_2, where Q is explicitly the source domain data set, and Section IV.A reports using 2000 source samples to calculate Omega. This directly contradicts the abstract and introduction, which state that 'the source data is unavailable due to privacy concerns,' and Section III.A, which defines the setting as 'without using source domain data during test-time.' Computing Omega once before deployment is still access to source-domain samples, and the paper itself acknowledges that 'regularization-based methods inevitably use the original distribution samples.' More importantly, Table IV shows that this regularizer is the dominant contributor to the reported gain: GEM alone gives 30.6, GEM+SCE gives 27.3, but adding Reg drops the error to 16.2, and the full method gives 16.0. Since CoTTA, RoTTA, and AR-TTA do not use source-domain samples, the comparisons in Tables I-III are not on equal footing. As written, the paper's headline claim for a source-free continual TTA setting is not supported; the main experimental gain comes from a component that violates the stated setting.","section":"Section III.E, Eq. (9); Section IV.A; Table IV"},{"comment":"The quantitative claim of state-of-the-art performance is fragile. No error bars or multiple-seed results are reported, and the hyperparameters lambda and beta are set separately for each dataset (1.8, 1.8, 0.3 and 1, 1, 0.005, respectively) on the same evaluation benchmark used for the final tables. The sensitivity analysis in Section IV.C is only shown for CIFAR10-C, so the reader cannot assess how the selected values affect CIFAR100-C and ImageNet-C. Moreover, the margin over CoTTA on CIFAR10-C is 0.2 percentage points (16.0 vs 16.2), which is likely within run-to-run variability for this type of experiment. The central claim that SPARNet outperforms prior work therefore needs stronger empirical support, preferably with error bars and with hyperparameters chosen on a validation split or shown to be stable across datasets.","section":"Section IV.A and IV.C; Tables I-III"}],"minor_comments":[{"comment":"For ImageNet-C, AR-TTA is reported only as a mean value (68.0) with no per-corruption entries; the reader cannot verify whether the comparison is on exactly the same corruption sequence and severity as the other rows.","section":"Section IV.B, Table III"},{"comment":"The implementation details contain several grammatical errors, e.g., 'we using the Adam optimizer' and 'we using the SGD optimizer,' which should be corrected.","section":"Section IV.A"},{"comment":"The sensitivity figures for lambda and beta are described only by their captions; the axes and exact metric are not described in the text, making the sensitivity claim hard to evaluate from the figures alone.","section":"Section IV.C, Figs. 2 and 3"},{"comment":"The notation for augmentation-averaged teacher predictions in Eq. (4) is somewhat unclear: the sum over i of aug_i(x) is written without an explicit average normalization, and the teacher predictions y_tu should be defined as the normalized average, not the unnormalized sum.","section":"Section III.D, Eq. (4)"}],"recommendation":"reject","confidential_remarks":"The source-data dependency in Eq. (9) is not just a presentation issue: the ablation shows it is the main source of the reported improvement, so the central claim does not hold in the stated source-free setting. The paper would need to be substantially reworked, either by removing the source-data-dependent regularizer and re-evaluating, or by honestly changing the setting and comparing with methods that also have access to source data. As it stands, I do not see a path to acceptance within the current scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know about SPARNet. Its reported advantage over CoTTA comes almost entirely from a regularization term that needs source-domain samples, which contradicts the paper's source-free privacy premise. And on its own, the proposed partitioning plus consistency module is not competitive: in the ablation, GEM+SCE gives 27.3% error on CIFAR10-C while CoTTA gives 16.2%. The headline 16.0% is basically the MAS regularizer doing the work.\n\nWhat's good: the paper is clearly written, uses the standard continuous TTA benchmarks, reports per-corruption results, and the ablation table is actually informative. The particular package — entropy-based sample partition with GEM on reliable samples, mean-teacher SCE on unreliable ones, plus a parameter-importance regularizer — is new as a combination even though each piece is prior art. I don't see evidence of fabricated numbers.\n\nThe soft spots are real. Eq. (9) computes Omega_i using Q, defined as the source domain dataset, and Section IV.A says 2000 source samples were used. The authors note you compute Omega once before deployment, but that still requires access to the source distribution. CoTTA, RoTTA, and AR-TTA never touch source data, so the comparison is not apples-to-apples. This is load-bearing because the ablation shows the regularizer is the single largest contributor. Remove it and the new modules actually underperform CoTTA by a wide margin on CIFAR10-C.\n\nSeparately, the margins over CoTTA are thin: 0.2 points on CIFAR10-C, 1.9 on CIFAR100-C, 2.0 on ImageNet-C. There are no error bars, lambda and beta are chosen per dataset on the same benchmark, and no code is provided. Those issues would be minor on their own, but together they make the exact numbers hard to trust.\n\nOne nuance: the authors do disclose the one-time source use in Section III.E, so this isn't a hidden assumption. But the abstract and intro explicitly say source data is unavailable due to privacy, which makes the framing wrong, not just under-specified.\n\nWho is this for? Someone assembling a TTA benchmark might use the tables, but as a scientific claim about source-free continual TTA, it needs rewriting. I'd still send it to peer review — the empirical work is usable and the issue is fixable — but as a reviewer I'd require either a genuinely source-free regularizer or an honest reframing that allows a pre-deployment source pass, plus error bars and code. Without those, I wouldn't accept it.","headline":"SPARNet's headline gain over CoTTA is driven almost entirely by a source-data-requiring regularizer that contradicts its own source-free framing; the proposed modules alone are not competitive.","tokens_in":12383,"tokens_out":5602,"would_cite":false,"duration_ms":48504,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SPARNet claims that splitting test samples by confidence and constraining important weights lets a model adapt across a stream of corruptions without forgetting, reaching average errors of 16.0% on CIFAR10-C, 30.6% on CIFAR100-C, and…","keywords":["continual test-time adaptation","sample partitioning","generalized entropy minimization","mean teacher consistency","anti-forgetting regularization","catastrophic forgetting","CIFAR-10-C","ImageNet-C"],"falsifier":"Train a separate run of SPARNet in which the parameter importances $\\Omega_i$ are computed from unlabeled target-domain data (the first available batches) or from random initialization instead of the source-domain set $Q$, keeping everything else fixed. If the average CIFAR10-C error rises back toward the 27.3% of the no-regularizer ablation, the method's stated source-free premise is falsified in practice.","tokens_in":11307,"feed_emoji":"🎯","tokens_out":4372,"duration_ms":35996,"temperature":0.7,"pith_summary":"The paper tries to show that a deployed classifier can keep improving through a long, unlabeled stream of corrupted test images, even though the corruption type changes without warning, if the adaptation is guided by a confidence split and a weight-protection regularizer. The proposed SPARNet framework separates each batch into low-entropy (reliable) samples that are trained with a softened entropy loss and high-entropy (unreliable) samples that are used only for mean-teacher consistency between augmented views. On top of both, an anti-forgetting term penalizes changes to parameters that the original model depends on most. On the continual CIFAR10-C, CIFAR100-C, and ImageNet-C benchmarks, the authors report average error rates that beat the continual TTA baselines they compare against, with the ablation on CIFAR10-C showing the anti-forgetting term as the single largest source of gain. If true, this points to a practical recipe for keeping models stable under long-term distribution shift without labels.","feed_headline":"SPARNet cuts continual TTA error to 16.0% on CIFAR10-C","feed_subtitle":"Partitioning samples by confidence and guarding important weights keeps the model adapting across 15 corruptions without forgetting.","key_machinery":"The load-bearing object is the entropy-based sample partition plus the parameter-importance vector. A threshold $E_0$ on prediction entropy splits each batch into reliable and unreliable groups; the reliable group drives adaptation through generalized entropy minimization with a temperature-scaled softmax, while the unreliable group drives representation learning through symmetric cross-entropy consistency between the student and a mean teacher whose weights are an exponential moving average of the student. The anti-forgetting term uses the sensitivity of the network's output function to each parameter, computed over source-domain samples before deployment, to build an importance weight $\\Omega_i$; the regularizer then penalizes the squared change of each parameter scaled by that weight. This design is what lets the model adapt long-term without drifting.","core_discovery":"SPARNet's central claim is that continual test-time adaptation is best handled by treating the confidence of the model's own predictions as a switch: reliable samples receive a generalized entropy-minimization loss with softened logits, unreliable samples receive a symmetric cross-entropy consistency loss between a student and a mean-teacher model, and a memory-aware-synapses style regularizer computed once before deployment keeps the most important parameters close to their source values. The authors report that this combination yields average error rates of 16.0% on CIFAR10-C, 30.6% on CIFAR100-C, and 67.3% on ImageNet-C in the continual setting, outperforming CoTTA, RoTTA, and AR-TTA in their runs. The ablation in Table IV attributes the largest single improvement on CIFAR10-C to the anti-forgetting regularizer: adding it to GEM plus SCE drops the average error from 27.3% to 16.2%.","pith_inferences":["A natural next test is whether the importance vector could be estimated from a tiny held-out source set or synthesized prototypes, which would keep the anti-forgetting gain while respecting the source-privacy constraint; the paper itself does not explore this.","Because the partition threshold and the regularization strength are dataset-specific hyperparameters, the method's practical advantage may depend on how much tuning is allowed; a fair benchmark would fix them across all corruption types.","The same recipe could apply to semantic segmentation or object detection under continuous domain shift, where pseudo-label noise is more severe and per-pixel confidence partitioning would be the analogue of the sample split.","One could test whether the anti-forgetting regularizer alone, without sample partitioning, explains most of the improvement; the Table IV ablation suggests yes for CIFAR10-C, though the paper reports only one combination order."],"forward_implications":["If SPARNet's results hold, continual TTA systems can stop discarding low-confidence test samples and instead use them as consistency-training data without contaminating the model.","The large ablation gain of the anti-forgetting term suggests that preventing drift of important parameters matters more than the choice of pseudo-labeling loss for long adaptation streams.","A fixed entropy threshold $E_0 = 0.4\\ln C$ and precomputed importance weights make the method attractive for edge deployment, since the per-step updates are standard backpropagation.","The reported margins over CoTTA, RoTTA, and AR-TTA indicate that a simple two-branch loss plus weight protection is sufficient to beat more complex memory-buffer or robust-optimization machinery."],"supporting_citations":[{"why":"Supplies the continual TTA benchmark protocol and the augmentation-averaged pseudo-label mean-teacher idea that SPARNet adapts.","marker":"[1]"},{"why":"Provides the entropy threshold formula $E_0 = 0.4\\ln C$ and the reliability filter that SPARNet extends.","marker":"[5]"},{"why":"Memory-aware synapses method, the source of the anti-forgetting importance computation in Eq. (9).","marker":"[31]"},{"why":"Generalized entropy minimization loss on softened logits used for reliable samples.","marker":"[32]"},{"why":"Symmetric cross-entropy loss used for the unreliable-sample consistency term.","marker":"[39]"},{"why":"Mean-teacher weight averaging that underlies the teacher update in Eq. (6).","marker":"[35]"},{"why":"Spherical knowledge distillation scaling that motivates the dynamic temperature in Eq. (3).","marker":"[33]"},{"why":"AR-TTA, a baseline that SPARNet claims to outperform on ImageNet-C.","marker":"[24]"},{"why":"RoTTA, a temporally-aware baseline that SPARNet claims to outperform.","marker":"[25]"}],"fun_headline_variants":["SPARNet: sample partitioning and anti-forgetting cut CIFAR10-C error to 16%","Trustworthy samples guide long-term TTA: SPARNet hits 16% on CIFAR10-C","Anti-forgetting regularizer is key: SPARNet drops error to 16%","For continual TTA, SPARNet splits samples and guards weights","SPARNet's dual strategy: reliable samples, reliable weights = 16% error"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The anti-forgetting regularizer requires access to source-domain samples to compute parameter importances before deployment, which sits uneasily with the paper's claim that source data is unavailable for privacy reasons; without those samples, the largest single component of the reported gain cannot be computed.","fun_headline_variants_meta":{"raw":{"variants":["SPARNet: sample partitioning and anti-forgetting cut CIFAR10-C error to 16%","Trustworthy samples guide long-term TTA: SPARNet hits 16% on CIFAR10-C","Anti-forgetting regularizer is key: SPARNet drops error to 16%","For continual TTA, SPARNet splits samples and guards weights","SPARNet's dual strategy: reliable samples, reliable weights = 16% error"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000511,"raw_usage":{"total_tokens":2518,"prompt_tokens":1007,"completion_tokens":1511,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":623,"completion_tokens_details":{"reasoning_tokens":1395}},"tokens_in":623,"tokens_out":1511,"duration_ms":11709,"temperature":1.0,"reasoning_tokens":1395,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:42:13.803647+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a separate run of SPARNet in which the parameter importances $\\Omega_i$ are computed from unlabeled target-domain data (the first available batches) or from random initialization instead of the source-domain set $Q$, keeping everything else fixed. If the average CIFAR10-C error rises back toward the 27.3% of the no-regularizer ablation, the method's stated source-free premise is falsified in practice.","supporting_citations":[{"cited_title":"Robust test -time adaptation in dynamic scenarios,","cited_arxiv_id":null,"evidence_quote":"RoTTA, a temporally-aware baseline that SPARNet claims to outperform."},{"cited_title":"Ar -tta: A simple method for real -world continual test -time adap tation,","cited_arxiv_id":null,"evidence_quote":"AR-TTA, a baseline that SPARNet claims to outperform on ImageNet-C."},{"cited_title":"Continual test -time domain adaptation,","cited_arxiv_id":null,"evidence_quote":"Supplies the continual TTA benchmark protocol and the augmentation-averaged pseudo-label mean-teacher idea that SPARNet adapts."},{"cited_title":"Efficient test -time model adaptation without forgetting,","cited_arxiv_id":null,"evidence_quote":"Provides the entropy threshold formula $E_0 = 0.4\\ln C$ and the reliability filter that SPARNet extends."},{"cited_title":"Memory aware synapses: Learning what (not) to forget,","cited_arxiv_id":null,"evidence_quote":"Memory-aware synapses method, the source of the anti-forgetting importance computation in Eq. (9)."},{"cited_title":"Domainadaptor: A novel approach to test-time adaptation,","cited_arxiv_id":null,"evidence_quote":"Generalized entropy minimization loss on softened logits used for reliable samples."},{"cited_title":"Symmetric cross entropy for robust learning with noisy labels,","cited_arxiv_id":null,"evidence_quote":"Symmetric cross-entropy loss used for the unreliable-sample consistency term."},{"cited_title":"Mean teachers are better role models: Weight -averaged consistency targets improve semi-supervised deep learning results,","cited_arxiv_id":null,"evidence_quote":"Mean-teacher weight averaging that underlies the teacher update in Eq. (6)."}],"review_version":1}