{"id":"76bdf8ce-2cb7-437c-a88e-8948d672822a","arxiv_id":"2412.14809","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"ResoFilter keeps fine-tuning examples that produce small parameter updates in the last layers, matching full-data fine-tuning on GSM8k with 50% of the math data.","lead":"This paper introduces ResoFilter, a data filtering method that scores each fine-tuning example by how much it changes a language model's weights after a short individual fine-tuning step, then removes the examples that cause the largest changes. The authors report matching full-data fine-tuning on a math benchmark while using only half of the data, though results on other domains are mixed.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GSM8k configuration ablations make the 50%-matches-full-SFT result a test-set-selected optimum; a held-out selection test is required.","rationale":"After reading the paper in good faith, the central empirical claim is that ResoFilter at 50% MetaMath data matches full SFT. The weakest point in the evidence is not the per-sample scoring mechanism (which could be wrong yet still yield a working heuristic) but the circular use of GSM8k: Section 5.1's ablations select the very configuration that is then evaluated on GSM8k in Table 1. This is a classic selection-on-test-set problem and directly threatens the headline number. The reader's rationale already includes this as a ground for the conditional verdict, so my read does not change the verdict, but I would elevate it to the primary load-bearing concern. The concrete test is a held-out configuration selection. I did not find evidence of dishonesty; the issue is methodological. The paper has useful ablation coverage, but the missing validation split makes the central result unverifiable from the preprint.","tokens_in":16992,"tokens_out":11975,"duration_ms":83234,"concrete_test":"Hold out an independent math benchmark (e.g., MATH test set) or a random 20% split of GSM8k as a configuration-selection set. Use only that set to pick the module, layer count, and statistic (Wup, last-3, mean versus alternatives). Then run the 50% MetaMath filter with the chosen configuration and evaluate on the remaining GSM8k items. If the score does not reach full SFT within run-to-run noise (e.g., 0.01), the headline result is a test-set selection artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1 chooses the method's default configuration (Wup module, last-3-layer mean) by comparing GSM8k scores across modules (Table 9), statistics (Table 4), and layers (Figure 2). The same GSM8k benchmark is then used in Table 1 to report the headline result that 50% selected MetaMath data matches full SFT. This is selection on the test set: the reported 50% scores are the best of a menu of variants screened on GSM8k, so the advantage over random/full SFT may be inflated by noise-driven choices. For Llama2-13B the 50% score (0.6853) is already below full SFT (0.6935); without a held-out configuration selection there is no way to know whether the 'comparable' claim survives. The same GSM8k-tuned configuration is transferred to code/MMLU/BBH, where it underperforms random at several ratios, so the generalization claim is also unvalidated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces ResoFilter, a data-filtering method for fine-tuning large language models. For each training sample, the method fine-tunes a fresh copy of the base model on that single sample, computes the mean absolute parameter difference in the up-projection matrices of the last three transformer layers before and after this one-sample update, and uses that scalar as a 'diff' score. The final fine-tuning set keeps the samples with the lowest diff scores, preserving their original dataset order, and discards the highest-scoring samples, which the authors interpret as 'dirty' data. The authors evaluate ResoFilter on MetaMath with GSM8k accuracy across Gemma2-2B, Llama2-7B, and Llama2-13B, reporting that using only 50% of the data selected by the method matches or slightly exceeds full-data SFT in two of three models. They also report cross-domain results on code (HumanEval-CodeX) and general knowledge (MMLU, BBH), plus a cross-model transfer experiment where data filtered by a small model is used to fine-tune larger models. The paper includes ablations over weight modules, statistical aggregators, layer positions, and data ordering, all evaluated on GSM8k.","tokens_in":17170,"tokens_out":9450,"duration_ms":60432,"significance":"If the 50% result survives a held-out configuration test, ResoFilter would be a practical contribution: it is conceptually simple, interpretable, and its 'filter-once-apply-many' protocol reduces the cost of data selection for larger models. The feature analysis in Section 5.2, which ties low-diff samples to longer, more repetitive, and more self-similar text, is a useful qualitative insight. The paper also reports cross-model scaling to 9B and 70B parameters, which is a concrete strength, and Appendix C provides three-seed variance for one model, which is a welcome reproducibility step. However, the current evidence is weakened by test-set-selected configurations, missing per-sample training details, single-run comparisons, and a contradictory generalization claim; these issues are addressable in revision, so the result is not yet ready for acceptance.","major_comments":[{"comment":"The default configuration (Wup module, last-3 layers, mean statistic) is selected by comparing GSM8k scores across ablations, and the same GSM8k benchmark is then used in Table 1 to report the headline that 50% selected data matches full SFT. This constitutes selection on the test set: the reported 50% scores are the best among a menu of variants screened on GSM8k, so the advantage over random/full SFT may be inflated by noise-driven choices. For Llama2-13B, the 50% score (0.6853) is already below full SFT (0.6935), so the 'comparable' claim is not adversarially tested. I request a held-out configuration selection: choose the configuration on a separate validation set or a different task, then report Table 1 with that pre-registered configuration, or at least show that the conclusion is stable across the menu of configurations.","section":"Section 5.1, Tables 9/4, Figure 2, Table 1"},{"comment":"There is an inconsistency between p as the fraction of data retained and p as the filtering ratio. Section 4.1 states that filtering methods produce 25%, 50%, and 75% of the total data, so p=0.25 means retaining 25%. Algorithm 1 computes k = ⌊|D|×(1−p)⌋ and its comment says 'this effectively removes the top p% of samples,' which for p=0.25 retains 75% and removes 25%. Equation (4) is consistent with p as the retained fraction, but Algorithm 1 is not. Please align the pseudocode, the comments, and the experimental description, and state explicitly whether p denotes the retained fraction or the removed fraction throughout.","section":"Algorithm 1, line 14, and Section 4.1"},{"comment":"Table 8 reports three seeds only for Gemma2-2B and shows run-to-run variation of roughly ±0.003 to ±0.01 across ratios. The main comparisons in Tables 1, 3, 4, 9, and 10 are single runs, and many differences between methods are within this noise band (e.g., in Table 1 our method at 25% equals random at 25% for Gemma2-2B, both 0.6042). The claim that ResoFilter consistently outperforms other selection approaches is therefore not supported for those cells. Please report per-seed results or error bars for all headline tables, at least for one model per domain.","section":"Tables 1, 3, 4, 9, 10 and Appendix C"},{"comment":"The per-sample fine-tuning that produces the scores diff_i is not specified: the number of optimization steps, learning rate, optimizer, batch size, sequence length, and compute budget are all omitted. Without these details the scores are not reproducible, and the core assumption that a single-sample fine-tuned weight change predicts a sample's contribution in a multi-sample run is never tested. Please specify the per-sample training protocol and include a sensitivity analysis over its key hyperparameters.","section":"Section 3.3 and Appendix F, Algorithm 1"},{"comment":"The generalization claim that the method 'consistently outperforms random sampling across different domains' is contradicted by the lower half of Table 1: on MMLU at 50% our method (0.4893) is below random (0.5152), and on BBH at 50% our method (0.3620) is below random (0.3759). Please correct the claim, and if the method is not intended to improve on general knowledge benchmarks, state this clearly and provide a test of the hypothesis that the method does not harm beyond random.","section":"Section 4.2, Table 1"},{"comment":"The objective function E is presented as capturing an optimal trade-off, but Fcharacteristic(p) is the average of f(j,M) over the selected set Dp while f(j,M) is never defined independently of the selection rule. Since the algorithm selects exactly the samples with the smallest s(j,M), any ranking of 'characteristic intensity' that is monotone in s makes the claimed optimum in p true by construction. The parameters β and λ are not estimated or used. Please either show how Eq. (1) guides the algorithm or restate it clearly as a descriptive model rather than an optimization objective.","section":"Section 3.2, Eqs. (1)-(3)"}],"minor_comments":[{"comment":"There are several typos and formatting issues: 'Alabtion' in Section 5.1 should be 'Ablation'; 'Nuggest' in Table 1 should be 'Nuggets'; 'examned' in Section 5.2 should be 'examined'; 'MaxLegnth' in Table 2 should be 'MaxLength'; and 'V olume' appears in two references and should be 'Volume'.","section":"Throughout"},{"comment":"The description of the Loss and PPL baselines says the data are 'ranked from high to low based on loss and PPL, and then selected the top 25%, 50%, 75%,' but the ranking direction determines whether high-loss or low-loss samples are selected; please clarify whether the selected samples are those with the highest or lowest loss/PPL, and confirm this matches the code.","section":"Section 4.1, Baselines"},{"comment":"The statement that data points causing smaller differences are 'potentially more valuable' appears before the ablation evidence; it would be clearer to present the ablation results in Section 5.1 as the basis for this choice.","section":"Section 3.3"},{"comment":"The x-axis is described as 'the first layer to the 26th layer'; please specify the model and layer indexing (0-based or 1-based) and note the total number of layers in that model.","section":"Figure 2"},{"comment":"The abstract states that code and data will be released upon acceptance, while the paper also states that the source code is publicly available in a GitHub repository; please reconcile these statements and include a version or commit hash.","section":"Abstract and Appendix C"},{"comment":"The 'Base Score' column lists Gemma2-9B base 0.69 and Llama2-70B base 0.61, but the text says '3.2% vs. 3.4% average gain at p25'; please define how these percentage gains are computed from the table values.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of a machine learning or NLP journal, and the core idea is a reasonable empirical contribution. The main risk is overclaiming caused by selecting the method's configuration on the same GSM8k benchmark used for the headline result. I would encourage the editor to ask for a held-out configuration test, a corrected and fully specified algorithm, and multi-seed results for the central tables before considering acceptance. The issues are fixable within the scope of a major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about ResoFilter. It is a genuinely new heuristic: fine-tune a copy of the base model on each sample alone, measure the mean absolute change in the up-projection matrices of the last three layers, and keep the samples with the smallest changes. That is not in LESS, Superfiltering, Nuggets, or any loss/PPL filter I know, and it is cheap to run. The second thing is that the headline claim—50% of MetaMath data selected this way matches full SFT—holds across three models in Table 1, and for two of them it beats full SFT. So the core empirical result is believable and worth taking seriously.\n\nThe paper does a few things well. The feature analysis in Section 5.2, showing that low-diff data is longer, more repetitive, and more internally consistent, is a useful descriptive contribution even if it is post-hoc. The cross-model transfer to 9B and 70B in Table 3 is a nice practical result, though it is only one run per configuration.\n\nThe soft spots are serious but not fatal. The method's configuration—Wup module, last three layers, mean statistic—is chosen in Section 5.1 by looking at GSM8k ablations, and the headline result is also measured on GSM8k. That is selection on the test set. The advantage over random at 50% is 2-4 points, and some of it could be noise-driven choice of the best variant from a menu. Llama2-13B at 50% is already below full SFT. The cross-domain numbers are mixed: MMLU and BBH often underperform random, so the abstract's 'strong generalization across domains' is overclaimed.\n\nReproducibility is a real issue. Algorithm 1 says 'fine-tune M0 using d_i' but does not specify the number of steps, learning rate, optimizer, or epochs. Without that, the scores cannot be reproduced. Only Gemma2-2B has repeated runs (Table 8); the other tables have no error bars. The objective function in Section 3.2 is decorative—the algorithm never uses it—so the 'optimal trade-off' language is best ignored.\n\nThe per-sample weight-delta assumption, that a single-sample fine-tune delta predicts a sample's contribution in a full multi-sample run, is plausible but never tested. The paper does not compare against a cheaper proxy or a short multi-sample warmup.\n\nOverall: this is a solid empirical paper with a new heuristic and one believable central result. It deserves a serious referee, and a conditional acceptance that demands held-out configuration selection, repeated runs, and a precise per-sample fine-tuning recipe is the right outcome.","headline":"A genuinely new per-sample weight-delta filter that shows a believable 50%-data effect on math, but the headline number is selected on GSM8k and the per-sample fine-tuning budget is unspecified.","tokens_in":17737,"tokens_out":3231,"would_cite":true,"duration_ms":26109,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ResoFilter claims that a sample's value for fine-tuning is visible in how little it moves the model's deep weights, and that keeping only the low-motion half of a math dataset matches full fine-tuning.","keywords":["data selection","instruction tuning","synthetic data filtering","parameter difference","weight analysis","large language models","mathematical reasoning","fine-tuning"],"falsifier":"On a small MetaMath subset, train the base model on the full subset, then retrain with each of the lowest-scored samples removed one at a time and with each of the highest-scored samples removed one at a time; if removing low-scored samples does not consistently hurt more than removing randomly chosen samples, the per-sample weight-change ranking is not measuring marginal contribution.","tokens_in":16752,"feed_emoji":"🎯","tokens_out":8494,"duration_ms":68530,"temperature":0.7,"pith_summary":"ResoFilter proposes a fine-grained way to prune synthetic fine-tuning data for large language models: measure how much each training example moves the model's weights when the base model is fine-tuned on that example alone, then keep the examples that move the weights least. The paper argues that these low-disruption examples are the useful ones, while high-disruption examples are likely noise, outliers, or off-task content. On the MetaMath mathematics dataset, using the 50% of data selected this way matches full-dataset fine-tuning on GSM8k, and using 75% slightly beats full fine-tuning on two of three tested models. If this holds, data filtering becomes a property of the data-model pair rather than an external quality score, which would make synthetic data pipelines cheaper and more interpretable.","feed_headline":"Weight-shift filter halves fine-tuning data without accuracy loss","feed_subtitle":"By keeping samples that barely move a model's deep weights, ResoFilter selects the half of MetaMath that matters.","key_machinery":"The central object is the per-sample parameter-difference score, the Data-Parameter feature. It is computed by fine-tuning a fresh copy of the base model on a single sample $d_i$ to obtain $M_i$, taking $\\Delta W = M_i - M_0$, and averaging the mean of $\\Delta W$ over the up-projection modules of the last $n=3$ layers. This score ranks every data point, and the method keeps the bottom $k$ samples while restoring their original order for training. The score is the bridge between data and task: it says a useful sample is one that changes deep model weights little, while a disruptive sample is treated as low-quality.","core_discovery":"The central discovery is that the magnitude of the per-sample parameter change during single-example fine-tuning is a usable data-quality signal. For each sample, the method trains a fresh copy of the base model on that one sample, subtracts the original weights, and averages the mean absolute difference over the up-projection (Wup) matrices of the last three layers; this value is the sample's score. Samples with the smallest scores are retained and the rest discarded, preserving original order for training. On MetaMath, retaining the bottom 50% by this score gives GSM8k accuracy essentially equal to full SFT (for example, 0.6497 versus 0.649 for Gemma2-2B), and retaining 75% outperforms the full dataset on Gemma2-2B and Llama2-7B. Feature analysis characterizes low-score ('good') data as longer, more repetitive, internally similar samples and high-score ('dirty') data as short, rare-vocabulary, diverse samples.","pith_inferences":["A natural extension the authors do not state is that the score should behave like an expectedness measure, so a direct test is to compare ResoFilter rankings with base-model loss or perplexity to see how much overlap exists.","The paper does not specify the single-sample fine-tuning budget (number of steps, learning rate, optimizer); a stress test is whether rankings survive changes to that protocol, since the method depends on those scores being stable.","The authors' own limitation section notes untested regimes (models above 70B parameters, Mixture-of-Experts, and conversational systems); the filter-once-apply-many pattern would be on firmer ground if tested there.","If the mechanism is causal, the low-diff subset should also show up in training dynamics as flatter loss curves or less forgetting; the paper does not report such a test."],"forward_implications":["Retaining only 50% of MetaMath by ResoFilter matches full-dataset SFT on GSM8k across Gemma2-2B, Llama2-7B, and Llama2-13B; at 75% it slightly beats full SFT on Gemma2-2B and Llama2-7B.","The selection transfers across model scales: data filtered by Gemma2-2B improves Gemma2-9B, and data filtered by Llama2-7B improves Llama2-70B, with relative gains comparable to those seen on the smaller models.","The signal transfers across domains: on HumanEval-CodeX, 50% of filtered code data gives 0.425 versus 0.3549 for random sampling, while on MMLU and BBH the results are comparable or mixed with full fine-tuning often below the base model.","The retained low-diff subset has a consistent profile: longer sequences, common vocabulary, low unique-token ratio, and high internal query similarity, suggesting the score selects for representative, low-noise examples rather than rare or outlier content.","Because the filtering can be run once on a small model and reused on larger ones, the method offers a practical filter-once-apply-many workflow for synthetic data construction."],"supporting_citations":[{"why":"Supplies the MetaMath dataset whose 50% subset is the main testbed for the method's headline result.","marker":"(Yu et al., 2023)"},{"why":"Provides the GSM8k benchmark on which all mathematics fine-tuning results are evaluated.","marker":"(Cobbe et al., 2021)"},{"why":"Defines the evaluation protocol and benchmark choices (MMLU, BBH, HumanEval-CodeX) that the paper follows.","marker":"(Wang et al., 2023a)"},{"why":"Gradient-based data influence baseline and the conceptual contrast: the paper distinguishes its parameter-difference method from gradient similarity.","marker":"(Xia et al., 2024)"},{"why":"Superfiltering is a weak-to-strong baseline data filter compared in the main results.","marker":"(Li et al., 2024b)"},{"why":"Nuggets is a baseline data selection method based on downstream benefit, used as a comparison.","marker":"(Li et al., 2023b)"},{"why":"Supplies the interpretability premise that model weights, especially in intermediate and deep layers, store knowledge, which motivates reading data quality from weight changes.","marker":"(Dai et al., 2022)"}],"fun_headline_variants":["Weight-shift filter halves data, keeps accuracy","Keep samples that barely move model weights","ResoFilter picks data by weight shift, halves training data","Half the data, same accuracy via weight-change filter","Data-parameter resonance: filter by weight shift"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the weight change from fine-tuning a fresh copy of the base model on one example, averaged over the last three layers' up-projection matrices, predicts that example's contribution when thousands of examples are fine-tuned together; the paper does not test this premise directly.","fun_headline_variants_meta":{"raw":{"variants":["Weight-shift filter halves data, keeps accuracy","Keep samples that barely move model weights","ResoFilter picks data by weight shift, halves training data","Half the data, same accuracy via weight-change filter","Data-parameter resonance: filter by weight shift"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001283,"raw_usage":{"total_tokens":5226,"prompt_tokens":914,"completion_tokens":4312,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":4239}},"tokens_in":530,"tokens_out":4312,"duration_ms":24665,"temperature":1.0,"reasoning_tokens":4239,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:53:44.743993+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a small MetaMath subset, train the base model on the full subset, then retrain with each of the lowest-scored samples removed one at a time and with each of the highest-scored samples removed one at a time; if removing low-scored samples does not consistently hurt more than removing randomly chosen samples, the per-sample weight-change ranking is not measuring marginal contribution.","supporting_citations":[],"review_version":1}