{"id":"362ccb5f-f938-4d6f-a400-4d8df3672453","arxiv_id":"2412.15511","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"RESQUE is a single index, computed from representation angles or cluster-label agreement, that correlates with measured retraining cost, energy, and carbon emissions across several vision models and datasets.","lead":"This paper introduces RESQUE, a metric that estimates how expensive it will be to retrain an existing image model for a new task or a changed data distribution. It is designed to help practitioners choose between reusing a model and training from scratch, with an eye toward saving energy and carbon emissions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RESQUEtask's one-epoch ARI is not shown to be stable or monotone in full retraining cost; it may inherit optimization artifacts rather than intrinsic task difficulty.","rationale":"The reader's verdict is CONDITIONAL, and I retain it. The paper offers a plausible, architecture-agnostic estimator with released code and consistent correlations on several vision benchmarks, so rejection is not warranted. My concern targets the one specific step where the argument is least secure: the one-epoch fine-tune that produces the representations used for RESQUEtask. The estimator's value is not shown to be invariant to that step's hyperparameters, and its distinctive mechanism (KMeans ARI) is not compared with a simpler linear probe. These are empirical gaps, not mathematical contradictions, so the appropriate verdict remains CONDITIONAL. My concrete test would run the same task-change setup with a small grid of one-epoch learning rates and seed choices, and add a linear-probe baseline. If the correlations are stable and RESQUE beats the linear probe, the central claim survives; otherwise the paper needs revision. This is also a practical test because the authors release code, so the experiment is feasible.","tokens_in":19346,"tokens_out":10039,"duration_ms":91383,"concrete_test":"For a fixed original model (e.g., ResNet18 trained on CIFAR10) and a fixed set of target tasks (CIFAR100, GTSRB, SVHN, FashionMNIST), compute RESQUEtask while varying the one-epoch fine-tuning learning rate over {1e-4, 3e-4, 1e-3, 3e-3} and over at least 5 KMeans seeds. Keep the full-retraining protocol fixed. For each setting, record the Spearman correlation between RESQUEtask and each retraining measure (epochs, gradient norm, parameter change). Also compute a linear-probe accuracy on the same post-one-epoch representations and correlate that with the same measures. If the RESQUE correlations vary by more than roughly 0.1 across the learning-rate grid, or if the linear probe achieves equal or higher correlations, the proposed index is not a stable or distinctive estimator of retraining cost, and the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"RESQUEtask (Eq. 4) is 1 minus the Adjusted Rand Index between KMeans cluster labels on representations obtained after a single epoch of fine-tuning on the new task and the true labels. The central claim that a lower RESQUE predicts lower retraining cost requires this one-epoch agreement to be monotonically related to the total cost of full retraining. That assumption is the least secure part of the paper. First, the one-epoch representation is not a task-intrinsic quantity: it depends on the optimizer, learning rate, head initialization, data order, and random seed, none of which are varied or analyzed. If the chosen one-epoch schedule helps some tasks and hurts others, the index measures optimization artifacts rather than task difficulty. Second, KMeans assumes spherical clusters, while learned representation manifolds are rarely spherical; two classes can be linearly separable yet badly clustered by KMeans, giving a high RESQUEtask and inverting the intended ordering. A linear probe on the same representations would be a cheaper and more direct measure of class separation, but no comparison is made. Third, the abstract claims RESQUE is obtained 'before any computation,' yet RESQUEtask requires a full backward pass and parameter update for one epoch, contradicting that claim. The reported correlations (e.g., Spearman 0.97 in Table 1) have no confidence intervals and no baselines, so they do not establish that the ARI mechanism, rather than the shared one-epoch warm-up or noise-level monotonicity, drives the relationship.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RESQUE, a pair of indices intended to estimate the cost of retraining a pretrained model before the retraining is carried out. For distribution shifts, RESQUEdist is the average angular distance between class-wise L2-normalized sums of representation vectors from a single forward pass on the original and shifted data (Eqs. (1)-(3)). For task changes, RESQUEtask is the complement of the Adjusted Rand Index between KMeans cluster labels (obtained after one epoch of fine-tuning on the new task) and the true target labels (Eq. (4)). The authors report Pearson and Spearman correlations between these indices and retraining measures (epochs, total gradient norm, normalized parameter change, energy, and carbon emissions) on CIFAR10/CIFAR100/SVHN with three synthetic noise types, and across eight source datasets and several target tasks with ResNet18, VGG16, and a ViT. They conclude that lower RESQUE predicts lower retraining cost and enables sustainable model reuse decisions.","tokens_in":19578,"tokens_out":11647,"duration_ms":93738,"significance":"If the claimed correlations are robust, RESQUE would be a practically useful and inexpensive tool for model-reuse decisions, complementing the sustainability literature by linking representation geometry to resource expenditures. The paper releases code, uses multiple architectures and datasets, and reports both Pearson and Spearman coefficients with p-values; it also measures energy and carbon directly, which are assets. There is no circularity in the definition of RESQUE, since no parameter is fitted to the retraining outcomes. However, the evidence is mainly correlational over small sample sizes, the task estimator depends on a one-epoch fine-tuning whose monotonicity is not established, and no baseline comparison is provided. With additional validation and more careful claims, the result could be a useful practical tool.","major_comments":[{"comment":"The abstract and introduction state that RESQUE predicts retraining cost 'before any computation is performed' and that RESQUEdist requires 'no backward propagation or computation.' This is contradicted by the 'Change of Task' section, where RESQUEtask is computed only after one full epoch of fine-tuning on the new task, including backward passes and parameter updates, followed by KMeans clustering (Eq. (4), Algorithm 1). Because the 'before computation' claim is a central selling point for practical use, it must be either implemented (e.g., by an estimator that does not retrain) or carefully scoped to RESQUEdist only.","section":"Abstract / Introduction; Section 'Change of Task'"},{"comment":"The central assumption for RESQUEtask is that the agreement between KMeans cluster labels on representations after a single epoch of fine-tuning and the true class labels, measured by 1 - ARI, is monotonically related to the total cost of full retraining. This assumption is not justified or tested. The one-epoch representation is sensitive to optimizer, learning rate, head initialization, data order, and random seed, and KMeans assumes spherical clusters that may not match learned representation geometry. The paper reports no variance across seeds or hyperparameters, and it does not compare with a simpler and more direct class-separability measure such as a linear probe. I request a sensitivity analysis of the one-epoch budget and a demonstration that the relation is not an artifact of the chosen optimizer or seed; otherwise RESQUEtask may be measuring optimization dynamics rather than intrinsic task difficulty.","section":"Eq. (4); 'Adjusted Rand Index to Quantify Class Separation'"},{"comment":"The validation of both estimators relies on correlations over a small number of points: for RESQUEdist, 10 noise-intensity levels per noise type, and for RESQUEtask, 8 original tasks per target task in Table 2. No confidence intervals, baselines, or permutation tests are provided, and several reported p-values are not significant at the 0.05 level (e.g., Table 2, ResNet18 GTSRB: Spearman epochs p=0.052, parameter change p=0.119), yet the text in 'Experiments Across Different Original Tasks' calls the relationship 'strong and statistically significant' without qualification. I ask for confidence intervals, per-seed error bars, a null/baseline comparison (e.g., random representations or a trivial monotone-in-noise-level predictor), and a more careful wording of statistical significance. Without these, the central claim that RESQUE has strong correlation with retraining measures is not yet supported.","section":"Tables 1-2; 'Experiments Across Different Original Tasks'"},{"comment":"The introduction promises that 'through extensive experiments and by theoretical reasoning' the paper shows lower RESQUE correlates with lower retraining cost, but no theoretical argument, proposition, or proof appears anywhere in the manuscript. The support is entirely empirical correlation, and the experiments are conducted on the same datasets used to motivate the design. Either provide a formal statement of the conditions under which the correlation should hold, or remove the phrase and explicitly describe the result as an empirical finding.","section":"Introduction"}],"minor_comments":[{"comment":"The notation 'arccos(V^O_i,norm, V^S_i,norm)' is ambiguous; define it as the arccosine of the inner product of the two normalized vectors.","section":"Eq. (3)"},{"comment":"In the provided manuscript, several axis labels and legends are rendered as unicode placeholders (e.g., '/uni00000031...'), making some plots unreadable; please replace them with proper typeset labels.","section":"Figures 1-5"},{"comment":"The rows of Table 2 are labeled only by the target task; please clarify in the caption that the correlations are computed across the eight original tasks for each target task, since the surrounding text sometimes refers to both original and target tasks.","section":"Table 2"},{"comment":"The early-stopping thresholds of 25 or 50 epochs with 0.5% or 1% accuracy margins are not described as a design choice; please state how these thresholds were selected and whether the reported epoch counts are sensitive to them.","section":"Appendix; Hyperparameters"},{"comment":"The claim that RESQUE is 'model/architecture-agnostic' is based on one ViT configuration and two CNN backbone families; please temper this claim or add evidence from additional architectures.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"To the editor: this is a reasonable empirical contribution with a useful idea and released code, but it currently overclaims in several places. The main risks are that the RESQUEdist correlation may be trivially driven by noise level, and that the RESQUEtask one-epoch mechanism is not validated as a stable predictor of full retraining cost. The missing baseline comparisons and confidence intervals should be addressed before publication. I am not recommending rejection; the paper could become acceptable after substantial revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new piece is RESQUEtask: fine-tune the model on the new task for one epoch, k-means cluster the representations, then take the complement of the Adjusted Rand Index against the true labels. That is a reasonable, cheap proxy for task difficulty, and the paper shows consistent positive correlations between this index and epochs, gradient norms, parameter change, energy, and carbon across several vision datasets and two architectures. The code is public, which is good. RESQUEdist, the distribution-shift half, is essentially a re-expression of the authors' earlier angle-based representation-shift measures; I do not see much new there.\n\nThe soft spots are real but not fatal. The biggest one: the abstract says RESQUE estimates cost 'before any computation is performed.' That is false for RESQUEtask, which requires a full epoch of fine-tuning with backward passes. You can argue it is still cheap relative to full retraining, but it is not zero-cost, and the phrase is misleading. Second, the intro promises 'theoretical reasoning' and the paper never delivers it; the argument is entirely empirical. Third, the validation is thinner than the numbers suggest. Table 2 correlations are over 8 original tasks; Table 1 over 10 noise levels per model, but there are no confidence intervals, no comparison to existing transferability measures like Task2Vec, and no baseline like a linear probe on the same representations. So the strong correlations could partly reflect a shared monotone trend (more noise means both higher RESQUE and higher cost) rather than something specific to the ARI mechanism. Fourth, the one-epoch representation is optimizer- and seed-dependent, and that dependence is not explored.\n\nNone of this changes my sense that the core idea is plausible. The problem is real, the estimator is simple, and the empirical direction is supportive. What is missing is calibration: baselines, confidence intervals, and an honest statement of what RESQUEtask actually costs.\n\nI would send it to review. A competent referee can ask for the missing baselines and the computation-cost clarification. It is not a desk reject; it is a conditional accept that needs work.","headline":"A plausible but overclaimed retraining-cost estimator: RESQUEtask is the only genuinely new piece, the zero-cost framing is wrong, and the validation needs baselines and confidence intervals.","tokens_in":20164,"tokens_out":2480,"would_cite":false,"duration_ms":23096,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One index predicts what a model retraining will cost","keywords":["model reusability","retraining cost estimation","distribution shift","task change","Adjusted Rand Index","sustainable AI","representation analysis","Green AI"],"falsifier":"Find two task changes or two shifted distributions with nearly identical RESQUE values but substantially different full-retraining epochs and energy use; if such a pair exists, the index does not carry the monotone cost information the paper claims.","tokens_in":19097,"feed_emoji":"♻️","tokens_out":6436,"duration_ms":54979,"temperature":0.7,"pith_summary":"RESQUE is a proposed single-number index for estimating how expensive it will be to retrain an already trained model, either because the input data has drifted or because the model is being pointed at a new task. For distribution shift, it compares class-level averaged representations of the original and shifted data through one forward pass. For a task change, it fine-tunes the model for a single epoch, clusters the new data's representations, and measures how little those clusters agree with true class labels. The paper argues that a lower RESQUE value tracks lower total retraining cost, and reports strong correlations with epochs, gradient norm, parameter change, energy consumption, and carbon emissions across several architectures and datasets. If the claim holds, practitioners could estimate sustainability costs before spending compute on fine-tuning.","feed_headline":"One index predicts what a model retraining will cost","feed_subtitle":"RESQUE links a cheap representation check to epochs, energy, and carbon, so reuse decisions can be made before compute is spent.","key_machinery":"The load-bearing object is the RESQUE index itself, computed from the model's internal representations rather than from gradients or backpropagation. For distribution shift, the machinery is the average inverse cosine angle between class-wise normalized embedding sums of the original and shifted datasets (Eq. (3)). For task change, the machinery is the complement of the Adjusted Rand Index (Eq. (4)), a standardized score comparing two clusterings where 0 means chance agreement and 1 means perfect agreement; it compares the true labels of the new task with KMeans cluster labels obtained from representation vectors after a single epoch of fine-tuning. The index is what connects a cheap forward pass to estimates of retraining epochs, gradient norms, parameter change, energy, and carbon emissions, and it is the thing whose correlations with those measures the experiments test.","core_discovery":"The central claim is that a model's representation space, inspected at almost no training cost, contains enough information to forecast the full cost of adapting that model, and that this forecast can be compressed into one index. RESQUE (REpresentation Shift QUantifying Estimator) has two forms: RESQUE_dist is the mean inverse cosine angle between per-class normalized summed embeddings of the original and shifted distributions, computed with no backward pass; RESQUE_task is the complement of the Adjusted Rand Index between true labels and KMeans cluster labels assigned to representations after a single epoch of fine-tuning on the new task. On CIFAR10, CIFAR100, SVHN, and other vision benchmarks, the paper reports that both variants correlate positively with retraining epochs, total gradient norm, normalized parameter change, and the energy and carbon emissions measured during retraining, with Pearson and Spearman coefficients mostly above 0.7 and low p-values. The conclusion drawn is that a low RESQUE indicates a cheap reuse and a high RESQUE a costly one, making the index a decision aid for sustainable model reuse.","pith_inferences":["Beyond the paper: the same representation-based logic should carry over to language and speech models when a fixed representation layer is available, making RESQUE a general reuse-cost index rather than a vision-specific one.","Beyond the paper: semantic or adversarial shifts may be a harder test than the noise corruptions used here, since representation angles can stay small while training difficulty grows; the paper does not establish monotonicity in that regime.","Beyond the paper: because RESQUE_task requires only one fine-tuning epoch, it could serve as a cheap tie-breaker when choosing among pretrained backbones before expensive fine-tuning is run."],"forward_implications":["If a model has a low RESQUE for a target task, a user can expect fewer retraining epochs and lower energy use than with high-RESQUE alternatives, and can rank candidate models accordingly.","Users can decide between retraining an existing model and training from scratch by comparing RESQUE estimates with known full-retraining costs, as the paper shows retraining generally uses less than from-scratch training.","RESQUE could be used before deployment to flag distribution shifts that will be expensive to adapt to, so that monitoring systems can alert operators early.","Because RESQUE is computed from a single forward pass, plus one epoch for task changes, it can be applied repeatedly over a model's lifetime to track how adaptation costs evolve."],"supporting_citations":[{"why":"Defines the Adjusted Rand Index that RESQUE_task complements to turn cluster-label agreement into a cost estimate.","marker":"Hubert and Arabie 1985"},{"why":"Supplies the KMeans++ initialization used to obtain representation cluster labels for the task-change estimator.","marker":"Arthur, Vassilvitskii et al. 2007"},{"why":"Defines the corruption types and severity scale used to generate the distribution-shift benchmarks for RESQUE_dist.","marker":"Hendrycks and Dietterich 2019"},{"why":"Provides the CodeCarbon tracking used to measure the energy and carbon-emission retraining measures RESQUE is compared against.","marker":"Schmidt et al. 2021"},{"why":"Provides the ResNet18 architecture used in the task-change and distribution-shift experiments.","marker":"He et al. 2016"},{"why":"Provides the Vision Transformer architecture used to show RESQUE is not limited to convolutional networks.","marker":"Dosovitskiy et al. 2021"},{"why":"Motivates the aggregated parameter-change measure used as one of the retraining-cost baselines.","marker":"Zhang, Bengio, and Singer 2022"}],"fun_headline_variants":["RESQUE: a single number predicts retraining cost","One index forecasts the price of model reuse","A 1-index shortcut to sustainable model reuse","Predict retraining cost before you spend a cent","Cheap index reveals expensive retraining"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that what is visible after one epoch of fine-tuning and a cluster analysis of new representations, or, for distribution shift, in the angles between class-average embeddings, reliably previews the total cost of full retraining; if that early signal misorders tasks or shifts, RESQUE's ranking fails.","fun_headline_variants_meta":{"raw":{"variants":["RESQUE: a single number predicts retraining cost","One index forecasts the price of model reuse","A 1-index shortcut to sustainable model reuse","Predict retraining cost before you spend a cent","Cheap index reveals expensive retraining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000182,"raw_usage":{"total_tokens":1322,"prompt_tokens":969,"completion_tokens":353,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":284}},"tokens_in":585,"tokens_out":353,"duration_ms":3819,"temperature":1.0,"reasoning_tokens":284,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:21:17.932605+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Find two task changes or two shifted distributions with nearly identical RESQUE values but substantially different full-retraining epochs and energy use; if such a pair exists, the index does not carry the monotone cost information the paper claims.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Adjusted Rand Index that RESQUE_task complements to turn cluster-label agreement into a cost estimate."}],"review_version":1}