{"id":"2db23373-4739-4768-a0d7-444eff4fd824","arxiv_id":"2412.20996","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A plug-and-play weighting scheme that upweights preference pairs on which a model repeatedly errs gives small, inconsistent accuracy gains on GSM8K and MATH500 across DPO, SimPO, DPOP, and IPO.","lead":"This paper proposes a training trick for language models: sample each math question several times, count how often the model answers correctly, then weight harder questions more heavily during preference optimization. The authors report modest gains on math benchmarks with several optimization methods, though the results are mixed and lack error bars.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains are not isolated to the weighting: the framework bundles self-sampled preference-pair construction with Eq. (2) reweighting, and no unweighted control on the same pairs is reported; Table 1 also contains multiple negative entries, so 'consistent improvements' is unsupported.","rationale":"The reader's weakest assumption is that 16 samples yield a stable difficulty signal; that is real, but I think the more immediate gap is that the experimental comparison never isolates the weight. Section 3.2 defines both a data-construction rule and a weight rule; Figure 8 shows the data-construction rule alone (self-answer vs golden answer) can shift accuracy by +3.2 to +3.8. Unless the 'DPO/SimPO/DPOP/IPO' rows in Tables 1-2 use the identical self-constructed pairs with uniform weights, the reported gains are compatible with the data-construction effect. The abstract's 'consistent improvements' is also contradicted by the table's negative entries, so the effect size is fragile. The paper deserves credit for a clear motivation (Figure 3) and for reporting the self-answer vs golden-answer ablation, but it omits the one control that would validate the plug-and-play weighting claim. No code or formal verification is provided, so the precise form of Eq. (2) (including the redundant max and the confined weight range) cannot be checked by the reader. A matched random-weight control is cheap and decisive. Because the paper's central mechanism is unverified but not yet disproven, the original CONDITIONAL verdict remains appropriate, with the condition being exactly this control experiment.","tokens_in":13566,"tokens_out":8305,"duration_ms":80593,"concrete_test":"Run a matched 3-seed ablation on Qwen2-7B-Instruct with the first 20k MetaMath-MATH samples: use the exact same self-sampled preference pairs and the same training recipe for (a) unweighted DPO, (b) DPO with w computed from Eq. (2), and (c) DPO with randomly permuted w (same multiset of weights). Report mean and std of GSM8K and MATH500 accuracy, with paired bootstrap confidence intervals on the (b)−(a) and (b)−(c) differences. If (b) does not beat both (a) and (c) by more than seed noise, the claimed benefit of the computed weights is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Abstract) is that plugging w from Eq. (2) into pairwise PO losses yields consistent improvements. Table 1 does not exhibit consistency: e.g., Qwen2-1.5B GSM8K DPO 64.59 vs 64.74 (-0.15), DPOP 64.36 vs 64.74 (-0.38); GLM4-9B SimPO 57.92 vs 58.75 (-0.83), IPO 58.68 vs 58.83 (-0.15); Table 2 IPO 54.28 vs 55.34 (-1.06). The load-bearing problem is attribution. Section 3.2 changes two factors simultaneously: it builds preference pairs from the model's own samples (chosen = a correct self-generated response, rejected = the most common wrong answer; gold answer only if no correct sample) and then multiplies the loss by w. The baselines in Table 1 are described only as 'DPO', 'SimPO', etc.; the paper never states that those baselines use the same self-constructed pairs with uniform weights. If they instead use different preference data, the gains may come from the data construction—Figure 8 already shows large gains from self-answer over golden-answer chosen responses—and not from Eq. (2). In addition, Eq. (2) as written is not fully reproducible: the second branch's max(1,·) is redundant because the expression is always >1, and with N=16, α=1 the weights lie in roughly [1,2], a weak rescaling. Without a random-weight or uniform-weight control on identical pairs, the claimed mechanism is untested.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a plug-and-play weighting framework for preference optimization in mathematical reasoning. For each training question, the base model is sampled N times; the responses are grouped by extracted numeric answer, the numbers of correct and incorrect samples are counted, and a per-question weight w is computed by Eq. (2). Preference pairs are then constructed from the model's own samples (a self-generated correct answer as chosen, the most frequent wrong answer as rejected), and the pairwise preference loss of DPO, SimPO, DPOP, or IPO is multiplied by w. Experiments on GSM8K and MATH500 across Qwen2-1.5B/7B, GLM4-9B, and ChatGLM3-6B are reported in Tables 1-2, and the Abstract claims consistent improvements.","tokens_in":13940,"tokens_out":4715,"duration_ms":44049,"significance":"If the central claim were established, a simple scalar reweighting of pairwise preference losses that focuses training on questions the model has not mastered would be a useful and cheap addition to existing preference optimization methods. The framework is simple, applies to several losses, and the authors honestly discuss the answer-equivalence limitation and include a failure-case analysis for LLaMA3 in Appendix C. However, the current evidence does not support the claimed consistency: Table 1 contains many negative weighted-vs-unweighted differences, the design does not isolate the weight w from the self-sampled data-construction change, and no variance or significance information is provided. The value of the contribution will be clear only after a controlled comparison on identical preference pairs with error bars.","major_comments":[{"comment":"The Abstract's claim of \"consistent improvements\" is contradicted by the reported numbers. Across the 24 weighted-vs-unweighted comparisons in Table 1, only 13 show a positive difference and several are negative, e.g., Qwen2-1.5B GSM8K DPO 64.59 vs 64.74 (-0.15), Qwen2-1.5B GSM8K DPOP 64.36 vs 64.74 (-0.38), GLM4-9B GSM8K SimPO 57.92 vs 58.75 (-0.83), and GLM4-9B GSM8K IPO 58.68 vs 58.83 (-0.15). Table 2 adds another negative entry (IPO 54.28 vs 55.34, -1.06). The text should be revised to report this mixed pattern accurately, and the consistency claim should be either supported with an appropriate aggregate statistic or removed.","section":"§4.5, Table 1"},{"comment":"The experimental design does not isolate the effect of the weight w from the data-construction changes. The treatment changes two factors simultaneously: (i) preference pairs are built from self-sampled responses, with the chosen response being a self-generated correct answer and the rejected response being the most common wrong answer, and (ii) the pairwise loss is multiplied by w. The baselines in Tables 1 and 2 are described only as \"DPO\", \"SimPO\", \"DPOP\", and \"IPO\"; the paper never states whether these baselines use the identical self-constructed preference pairs with uniform weights or a different preference dataset. Because Figure 8 shows large accuracy gains from using self-generated answers over golden answers as the chosen response, the observed improvements in Tables 1-2 could be attributable entirely to the data construction rather than to Eq. (2). The authors should add an unweighted control trained on the same self-constructed pairs.","section":"§3.2 and §4.5"},{"comment":"Equation (2) is not fully reproducible as written and its effect size is unclear. In the Pc>0 branch, the expression 1 + α·Pe/((Pc+ϵ)·N) is always greater than 1 for valid counts, making the max(1, ·) redundant; the placement of \"· 1/N\" is ambiguous and the first branch (1 + α·Pe/N) does not share the same scaling structure as the second branch. With N=16 and α=1, the weights lie approximately in [1, 1.94], which is a weak rescaling. The paper reports no sensitivity analysis for α, N, or sampling temperature, and no evidence that 16 samples at temperature 0.7 produce a stable estimate of per-question difficulty; if the counts are noisy, the weights add variance without a clear signal.","section":"§3.2, Eq. (2)"},{"comment":"All experimental results are single-run and no error bars, multiple seeds, or significance tests are reported. Many weighted-vs-unweighted differences are within ±1 point on test sets of 500 (MATH500) or 1,319 (GSM8K) examples, so these differences cannot be distinguished from sampling noise without variance estimates. The authors should report multiple seeds or bootstrap confidence intervals, and a significance test for any aggregate improvement across models and methods.","section":"§4.4, Tables 1 and 2"}],"minor_comments":[{"comment":"There are multiple typos and grammatical errors, including \"we assigns\" (Section 1), \"responds\" in the Figure 3 and Figure 5 axis labels, \"Bdfore\" in Figure 5, \"Comparision\" in the Figure 9 caption, and \"T ype\" in Figure 6. The manuscript should be proofread.","section":"Throughout"},{"comment":"The derivation from Eq. (3) to Eq. (4) multiplies the logit difference by w, but it is not stated whether this is a heuristic modification or a consequence of a weighted Bradley-Terry likelihood. If the latter, the likelihood should be defined explicitly.","section":"§3.3, Eqs. (3)-(4)"},{"comment":"The sentence \"we use ChatGLM3-6B for alignment with GSM data\" is unclear; it should specify which training subset is used and that GSM8K is the evaluation set. The model-selection criterion in the same paragraph (\"selecting models that maintain a certain error rate\") should be made more precise.","section":"§4.2"},{"comment":"The GLM4-9B prompt discussion reports an inability to reproduce official accuracy but does not state the reproduced numbers or explain how the resulting baseline differences affect the comparison. Please provide the actual reproduced accuracies and clarify that the prompts are held fixed across methods.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses a reasonable and potentially useful idea, but the reported evidence is not sufficient for the claimed \"consistent improvements.\" The key missing experiment is a uniform-weight control trained on the exact same self-constructed preference pairs; without it, the attribution to Eq. (2) is untestable. The mixed Table 1 results and the absence of error bars would likely also be raised by another reviewer, so a revision that adds the control, reports variance, and tempers the claims is necessary before the paper can be considered for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a classic case of a plug-and-play trick that sounds good on paper but whose evaluation doesn't separate the two things it changes at once. The core idea—sample each question N times, count distinct numeric answers, weight each preference pair inversely to how often the model gets it right—is clean and cheap. Integrating a scalar weight into DPO/SimPO/DPOP/IPO is straightforward, and the paper does that clearly. The data distribution plots (Figures 3, 5, 6) give an honest look at what repeated sampling reveals, and the limitation section is a real attempt to bound the method's scope.\n\nThe soft spots are mostly about attribution and statistics. Table 1 shows 6 of 12 MATH500 cells and 5 of 12 GSM8K cells improve; several weighted rows are worse than their unweighted counterparts. The abstract's 'consistent improvements' is simply not supported by the paper's own numbers. More importantly, the framework bundles two changes: it builds preference pairs from the model's own samples (chosen = correct self-generated response, rejected = most common wrong answer, gold only if no correct sample) and then multiplies the loss by w. The baselines are just called 'DPO', 'SimPO', etc. If those baselines use different preference data (e.g., golden-answer pairs), any gain could come from the data construction, not from w. Figure 8 already shows a large boost from self-answer over golden-answer chosen responses, which makes this confound concrete. The paper never states the baselines use identical pairs with uniform weights, and without that control, the claimed mechanism is untested.\n\nThere are smaller issues: no error bars or multiple seeds, so the mixed differences (±0.5–1 point) are within plausible noise; Equation (2) has a redundant max(1,·) and, as written with N=16 and α=1, produces weights in a narrow [1,2] range, which suggests the rescaling is weak; and the model selection (ChatGLM3 for GSM, Qwen for MATH) is justified but leaves the cross-generalization question open.\n\nWho is this for? Someone working on preference optimization for math reasoning might find the weighting formula worth a closer look. But as a referee, I'd want to see an identical-pairs baseline with uniform weights, a random-weights control, and a small seed analysis. The idea deserves that follow-up; the current paper doesn't support its headline. I'd accept it for peer review with major revision expected.","headline":"Reasonable reweighting idea, but the paper's own tables don't support 'consistent improvements' and the evaluation confounds weight with data construction.","tokens_in":14468,"tokens_out":3155,"would_cite":false,"duration_ms":29935,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A plug-and-play weighting scheme uses repeated sampling to focus preference optimization on hard math problems, lifting accuracy on GSM8K and MATH500 across DPO, SimPO, DPOP, and IPO.","keywords":["preference optimization","mathematical reasoning","sample weighting","repeated sampling","DPO","output distribution","training framework","LLM alignment"],"falsifier":"Shuffle the computed weights across training questions while keeping their distribution fixed, retrain with the same preference optimizer, and compare GSM8K and MATH500 accuracy against the proposed weighting; if shuffled weights match the gains, the per-question difficulty signal carries no explanatory power.","tokens_in":13345,"feed_emoji":"🧮","tokens_out":9259,"duration_ms":81441,"temperature":0.7,"pith_summary":"This paper argues that standard preference optimization methods like DPO waste training signal by treating every preference pair equally, which is especially damaging in mathematical reasoning where some problems are already mastered and others are systematically misunderstood. It proposes a plug-and-play weighting framework: sample each training question multiple times, count how often the model produces a correct answer, and derive a per-question weight that upweights problems the model repeatedly fails. Those weights are multiplied into the reward gap of any pairwise preference loss, and the paper reports that this improves GSM8K and MATH500 accuracy across DPO, SimPO, DPOP, and IPO with several model families. If the claim holds, any pairwise preference optimization method can be upgraded with little extra compute to focus training on the examples that matter most.","feed_headline":"Reweighting preference pairs by sampling lifts math accuracy","feed_subtitle":"Repeated-sampling weights focus DPO and three other methods on hard problems, improving GSM8K and MATH500 scores.","key_machinery":"The load-bearing object is the weight $w$ from Equation (2): $w = 1 + \\alpha P_e/N$ when $P_c = 0$, and $w = \\max(1,\\, 1 + \\alpha P_e/((P_c+\\epsilon)N))$ when $P_c > 0$, where $P_c$ and $P_e$ are the numbers of correct and incorrect answers among $N$ samples, and $\\alpha$ is a hyperparameter that controls the adjustment magnitude. This scalar is a direct estimate of per-question difficulty from the model's own output distribution, and it is inserted as a multiplicative factor on the reward gap in the pairwise preference loss. It carries the argument by turning an observed sampling statistic into a training signal that shifts optimization effort toward questions the model has not mastered.","core_discovery":"The paper's central claim is that a model's own repeated-sampling answer distribution provides a usable, model-specific difficulty measure, and that feeding this measure back as a scalar weight on each preference pair improves mathematical reasoning. For each question, the model samples N=16 responses at temperature 0.7; the count of correct and incorrect extractions (Pc and Pe) is turned into a weight w via Equation (2). This weight multiplies the reward difference in the preference objective, so pairs on which the model consistently fails get a larger gradient. The paper further shows that choosing the model's own correct response as the 'chosen' example, rather than the dataset's gold answer, helps training on mathematical tasks. The reported experiments show that this weighting generally outperforms the unweighted baselines across four preference optimization methods and several open-weight model families.","pith_inferences":["One testable extension the paper does not run: shuffle the computed weights across questions while keeping their distribution fixed. If shuffled weights reproduce the gains, the per-question difficulty signal is not what matters; the improvement would come merely from uneven weighting.","The difficulty estimate is a snapshot of the model before training. As the model improves, the weights become stale; recomputing them mid-training or using an online estimator could yield larger or more persistent gains.","The framework's dependence on answer equivalence classes, acknowledged in the paper's limitation, means the same recipe could be applied to open-ended tasks by replacing exact numeric matching with semantic-equivalence clustering.","Because the chosen answer is sampled from the model itself, the method may compound existing biases or style quirks when the model is weak; a stronger model or a higher sampling temperature would reduce the risk of locking in errors."],"forward_implications":["Any pairwise preference optimization method (DPO, SimPO, DPOP, IPO) can incorporate the weights with negligible added compute: one extra sampling pass over the training set and a scalar multiply in the loss.","Training emphasis shifts to systematic mistakes: questions where the model repeats the same wrong answer receive the largest weights, so the gradient concentrates on undoing those misconceptions.","Using the model's own correct response as the chosen example keeps the optimized policy close to the original model's distribution, reducing the style-mimicry failure mode that gold-answer training can trigger in math.","The gains should be most visible on harder, accuracy-critical benchmarks like MATH500 and on models with intermediate error rates, because models that are too strong or too weak yield less informative sampling distributions."],"supporting_citations":[{"why":"Supplies the DPO baseline and the reward representation that the weights modify.","marker":"Rafailov et al., 2023"},{"why":"Supplies the SimPO baseline, a reference-free pairwise method the framework is tested with.","marker":"Meng et al., 2024"},{"why":"Supplies the DPOP baseline and the observation that pairwise methods can overfit to minimal edits.","marker":"Pal et al., 2024"},{"why":"Supplies the IPO baseline and its generalization formulation.","marker":"Azar et al., 2024"},{"why":"Motivates repeated sampling for output distributions and provides the unbiased pass@k formula used in analysis.","marker":"Brown et al., 2024"},{"why":"Motivates multiple sampling as a way to understand model output patterns.","marker":"Snell et al., 2024"},{"why":"Provides the MetaMath training data (GSM and MATH subsets) used for training.","marker":"Yu et al., 2024"},{"why":"Supplies the GSM8K dataset and test set.","marker":"Cobbe et al., 2021"},{"why":"Supplies the MATH dataset.","marker":"Hendrycks et al., 2021"},{"why":"Supplies the MATH500 subset used for evaluation.","marker":"Lightman et al., 2024"}],"fun_headline_variants":["Sampling-based reweighting boosts math reasoning","Model's own answer distribution weights preference pairs","Adaptive weighting via sampling lifts math accuracy","Plug-and-play framework reweights for math gains","Self-sampled difficulty weights enhance preference training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a handful of sampled answers at one temperature reliably indicates how well the model has mastered each problem, so weighting by that count reflects true difficulty rather than sampling noise.","fun_headline_variants_meta":{"raw":{"variants":["Sampling-based reweighting boosts math reasoning","Model's own answer distribution weights preference pairs","Adaptive weighting via sampling lifts math accuracy","Plug-and-play framework reweights for math gains","Self-sampled difficulty weights enhance preference training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000183,"raw_usage":{"total_tokens":1255,"prompt_tokens":824,"completion_tokens":431,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":440,"completion_tokens_details":{"reasoning_tokens":362}},"tokens_in":440,"tokens_out":431,"duration_ms":4896,"temperature":1.0,"reasoning_tokens":362,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:04:47.881779+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Shuffle the computed weights across training questions while keeping their distribution fixed, retrain with the same preference optimizer, and compare GSM8K and MATH500 accuracy against the proposed weighting; if shuffled weights match the gains, the per-question difficulty signal carries no explanatory power.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the IPO baseline and its generalization formulation."}],"review_version":1}