{"id":"66f22a5d-3db8-45f3-93d5-91e00b11c439","arxiv_id":"2505.12754","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ProDS picks instruction-tuning data by matching training-sample gradients to preference gradients from DPO, achieving slight gains over prior selection methods on MMLU, TYDIQA, BBH, and Alpaca-style tests.","lead":"This paper proposes ProDS, a data selection method that scores instruction-tuning examples by how well they align with target-task preferences, using gradients from direct preference optimization. It reports small accuracy gains over existing baselines across several benchmarks while using only 5 to 10 percent of the training data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ProDS's core score compares SFT-loss gradients at θ_S to DPO-loss gradients at θ_D_p; with no shared loss or checkpoint this cosine-similarity influence estimate is undefined, and the linear annealing objective reduces Λ to a threshold rule.","rationale":"The paper proposes ProDS, which scores training samples by cosine similarity between projected SFT gradients and projected DPO gradients of preference pairs, then selects top-k. The reader's conditional verdict flags the proxy's reliability and missing code/variance. My stress-test sharpens this: the two gradient sets are not only different loss types but are evaluated at different checkpoints. Eq. (3) uses ∇L_S(x, θ_S); Eq. (5) uses ∇L_D(x', θ_D_p, θ_D_r). Since θ_D_p is the result of DPO warm-up on top of θ_S, the two vectors are gradients of different objectives at different points; cosine similarity between them is not a known influence estimator. LESS, which the paper builds on, uses the same loss at the same checkpoint for both sides. The paper provides no derivation or empirical validation for the heterogeneous comparison, making the central selection criterion an ungrounded heuristic. This is a correctness risk, not a mere omission: if the equations are literal, the method's scores may not reflect preference alignment. Additionally, Appendix A.1 reveals the DPO warm-up pairs are constructed using the same ground-truth responses as preferred, meaning the 'preference' signal partly re-encodes the original data; however, this is less central. The annealing synthesis is also vacuous: the energy E(Λ) in Eq. (7) is linear in each Λ_i, so simulated annealing merely finds a threshold rule, contradicting the claimed 'intelligent' bidirectional synthesis. A concrete, feasible test is to recompute G_train at θ_D_p (or use the same loss for both) and see if the top-k selection and Wizardlm win rate change. Because the manuscript does not include code or variance estimates, and the central equations are internally inconsistent, I cannot verify the claim; the appropriate verdict is UNVERDICTED rather than ACCEPT or CONDITIONAL. If the authors confirm the equations are typos and that both gradients are computed at the same checkpoint, the paper could move back to CONDITIONAL.","tokens_in":19028,"tokens_out":7340,"duration_ms":72545,"concrete_test":"Recompute the selection for the 10% Alpaca/Wizardlm configuration with G_train computed at θ_D_p using the DPO loss (or equivalently, compute both G_train and G_val with the same loss at the same checkpoint). If the resulting winning score on Wizardlm does not reproduce the reported 1.06, or falls below the IFD baseline, the reported gains hinge on the heterogeneous gradient comparison. As a cheaper sanity check, compute the cosine similarity between random projections of G_train and G_val under the paper's procedure and under the corrected procedure; if the corrected procedure yields materially different top-k samples, the Eq. (3)/Eq. (5) mismatch is consequential.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The method's central scoring mechanism (Sec. 3.2-3.3) is an influence estimate built on cosine similarity in projected gradient space. The training-sample influence G_train is computed as the projected gradient of the SFT cross-entropy loss at θ_S, the parameters after SFT warm-up (Eq. 3). The validation preference influence G_val is computed as the projected gradient of the DPO loss at θ_D_p, the parameters after the separate DPO warm-up (Eq. 5). These are gradients of two different loss functions evaluated at two different parameter settings. LESS, the cited basis, computes both training and validation gradients of the same loss at the same checkpoint. Here, a high cosine similarity between an SFT gradient and a DPO gradient has no known influence-theoretic meaning: it is not an estimate of how a training sample changes the DPO loss or any downstream metric. The paper offers no justification for this heterogeneous comparison. The issue is compounded by Eq. (5), which sets G_val=[G_app,G_app] (duplicating the positive block instead of concatenating G_awy), and by the annealing synthesis (Algorithm 1, Eq. 7): since E(Λ) is linear in each Λ_i, the global optimum is the threshold rule Λ_i=1 if Γ_app_i+Γ_awy_i>0 else 0, so simulated annealing provides no modeling capability beyond a binary sign decision. If the equations are taken literally, the core preference-alignment signal is not a principled influence estimate and the selection may be arbitrary; if the authors instead compute both gradient types at the same checkpoint and with the same loss, the paper's equations misstate the method.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"ProDS (Preference-oriented Data Selection) is a method for instruction-tuning data selection. The paper proposes scoring training samples by the cosine similarity between their projected SFT-loss gradients and projected DPO-loss gradients computed on validation preference pairs, thereby incorporating response-preference signals into selection. The method has three stages: (1) SFT and DPO warm-up on small subsets; (2) construction of positive and negative validation preference sets (with GPT-4 as judge) and computation of projected gradients; (3) scoring of training samples via cosine similarity, with positive and negative preference scores combined by an instance-level weight optimized by simulated annealing. The top-k samples are then used to fine-tune the target LLM. Experiments on Alpaca (evaluated on Vicuna, Koala, WizardLM, Self-Instruct, and LIMA via GPT-4 pairwise comparison) and on FLAN/COT/DOLLY/OpenAssistant (evaluated on MMLU, TYDIQA, and BBH with external metrics) compare against BM25, DSIR, RDS, LESS, and IFD.","tokens_in":19380,"tokens_out":9105,"duration_ms":88541,"significance":"If the proposed scoring mechanism were principled, ProDS would advance instruction-data selection by introducing preference information and would be supported by a broad set of experiments and ablations, including selection-model transfer and preference-direction analysis. However, the central mechanism is undermined by (a) an internal inconsistency in Equation (5), (b) the lack of any justification for comparing gradients of different losses at different checkpoints, and (c) the degenerate linear annealing objective that reduces to a per-sample threshold rule. The Alpaca evaluation is further confounded by using the same judge (GPT-4) for both preference construction and outcome evaluation, while the benchmark results show small margins over LESS and no variance estimates. The contribution is thus currently not at the level of a journal publication, though the underlying idea is worth pursuing.","major_comments":[{"comment":"Equation (5) defines G_val = [G_app_val, G_app_val], duplicating the positive-preference block instead of concatenating G_app_val with G_awy_val. As written, the negative preference direction is never used in the scoring, which directly contradicts the stated bidirectional contribution. This must be corrected and the subsequent formulas verified.","section":"Section 3.2, Eq. (5)"},{"comment":"The core score compares projected SFT-loss gradients at θ_S with projected DPO-loss gradients at θ_D_p, i.e., gradients of two different loss functions evaluated at two different parameter checkpoints. LESS, which is cited as the basis, compares gradients of the same loss at the same checkpoint, for which influence-theoretic interpretations apply. The paper provides no argument or experiment showing that the cosine similarity between a training SFT gradient and a validation DPO gradient estimates how much a training sample improves target-aligned behavior. Without such justification, the selection signal is not a principled influence estimate; an empirical validation (e.g., correlation with actual fine-tuning gains or leave-one-out influence) is required.","section":"Section 3.2-3.3, Eqs. (3)-(6)"},{"comment":"The energy function E(Λ) is linear in each Λ_i, so its global optimum over Λ_i ∈ [0,1] is the boundary point Λ_i = 1 if Γapp_i + Γawy_i > 0, otherwise Λ_i = 0. Simulated annealing therefore implements a per-sample sign threshold and provides no additional modeling beyond that rule. This overstates the contribution of the 'annealing-based integrating method.' Moreover, Algorithm 1 does not specify bounds on Λ, leaving the objective potentially unbounded; the comparison against the 'fixed' baseline in Table 3 is not an ablation of the synthesis mechanism against the true boundary optimum.","section":"Section 3.3, Eq. (7) and Algorithm 1"},{"comment":"For the Alpaca-related test sets, the same LLM judge (GPT-4) is used both to construct the validation preference pairs (by scoring responses from Mbase and Mcmp) and to compute the final pairwise win scores. Consequently, the reported improvements may reflect alignment with the judge's particular preferences rather than a generalizable improvement in instruction following. The MMLU, TYDIQA, and BBH results are not subject to this circularity, but their margins over LESS are small.","section":"Section 4.1.2-4.1.3, Fig. 4/5, Tables 3-4"},{"comment":"No repeated runs, error bars, or significance tests are reported. The differences over LESS are small (e.g., +0.6 MMLU, +0.4 TYDIQA, +1.2 or +1.6 BBH, depending on the baseline), and the claimed consistent advantage in Fig. 5 could be within run-to-run variance. The paper should report means and standard deviations over at least three seeds, and ideally a significance test, for the main comparisons.","section":"Section 4.2, Table 1 and Fig. 5"}],"minor_comments":[{"comment":"The word 'drowned' should be 'drawn' in the sentence describing the projection matrix.","section":"Section 3.2, Eq. (3)"},{"comment":"The statement 'Since M_D_r is fixed during the DPO warm-up process, θ_D_p is the same as θ_S' is incorrect; it is the reference model parameters θ_D_r that remain equal to θ_S after initialization, not the policy parameters θ_D_p.","section":"Section 3.1"},{"comment":"The '∆' column does not specify the baseline for the comparison, e.g., whether it is Ours(7B) minus LESS(7B) or another difference; please clarify.","section":"Table 1"},{"comment":"The caption of Figure 4 is ambiguous about what the three numbers in each row represent and for which comparison direction the wins/ties/losses are counted; please clarify.","section":"Section 4.2, Fig. 4"},{"comment":"The text says Γapp and Γawy are computed via a 'weighted summation using the L2 norm' but does not give an explicit formula; define these quantities precisely.","section":"Section 3.3, Eq. (6)"},{"comment":"The reference to Har-Peled and Kushal (2005) is a coresets paper, not a standard K-Means clustering reference; cite an appropriate clustering source.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be in a draft state with multiple typos and an incorrect equation. The two central methodological issues—the heterogeneous gradient comparison and the degenerate annealing objective—are load-bearing and will require substantial rework or additional empirical validation. The Alpaca evaluation should be re-run with a judge different from the one used to create preferences, and all headline results need variance estimates. The benchmark results are thin but promising; with these fixes the paper could become publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a reasonable incremental extension of LESS with a new signal—DPO preference gradients—and a two-sided positive/negative scoring scheme. The empirical story is mostly coherent: selected subsets at 5–10% of the data roughly match or slightly beat full-data fine-tuning on several benchmarks, and the cross-architecture transfer experiments are a nice touch. The paper is worth taking seriously, but the method section has real problems that need fixing before the claimed mechanism is credible.\n\nWhat is actually new: using DPO gradients, rather than SFT generation gradients, to represent target preferences for data selection, and explicitly splitting those preferences into positive and negative directions. The ablations in Table 3 support the value of the split, and Table 4 shows that a smaller selection model can transfer to larger target models. Those are genuine contributions.\n\nWhere it gets soft: the central scoring mechanism in Section 3.2 compares SFT-loss gradients on training samples at θ_S with DPO-loss gradients on validation pairs at θ_D_p. That is a cosine similarity between gradients of two different losses evaluated at two different checkpoints. LESS works because both sides are gradients of the same loss at the same checkpoint; here the influence-theoretic meaning is simply not established. No argument is given for why a high cosine between an SFT gradient and a DPO gradient should predict improvement in target-aligned behavior. This is load-bearing, not cosmetic.\n\nThe equations make things worse. Equation (5) concatenates G_app with itself instead of G_awy, and the text says θ_D_p is the same as θ_S right after describing a DPO warm-up that updates the policy. I suspect typos, but as written they obscure the method. Also, the annealing step is trivial: E(Λ) is linear in each Λ_i, so the optimum is a binary threshold. Simulated annealing adds nothing; the “intelligent synthesis” reduces to taking max(Γ_app, −Γ_awy) per sample.\n\nEmpirically, the gains over baselines are small, and there are no error bars or repeated seeds. On the Alpaca experiments, GPT-4 provides both the preference labels used for selection and the evaluation scores, so part of the reported improvement could reflect alignment with that judge. The MMLU, TYDIQA, and BBH results use external accuracy metrics and help ground things, but the gains there are modest. No code or data is released, which is a problem for a method whose exact behavior depends on details the equations do not pin down.\n\nNet: the paper has a plausible core idea and useful experiments, but the mechanism as written is not principled and the optimization claim is overstated. I would send it to peer review—an editor should not desk-reject it—but I would expect major revision: fix the equations, clarify the checkpoint and loss mixture in the gradient similarity, report variance, and release code. For my own work, I would not cite it as a reliable technique yet; I would mention it as related work if I needed to acknowledge the DPO-gradient direction.","headline":"A plausible incremental twist on LESS—DPO preference gradients for instruction data selection—but the core gradient-similarity step is not justified as written, one equation is wrong, and the annealing is just a threshold, so it needs major revision before the mechanism can be trusted.","tokens_in":19925,"tokens_out":3519,"would_cite":false,"duration_ms":42206,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Scoring instruction data by alignment with human-preference directions, as ProDS does, matches or exceeds full-data fine-tuning at 5–20% of the data.","keywords":["instruction tuning","data selection","direct preference optimization","gradient influence","preference alignment","large language models","targeted data selection"],"falsifier":"Rank the same training pool with the ProDS score and with a deliberately reversed score that swaps positive and negative preferences, train on the top subsets of each, and compare on the target benchmark; if the reversed ranking performs as well, the bidirectional preference signal is not what drives the gains. A cleaner test would train each candidate example alone and correlate its measured effect on target-preference accuracy with its ProDS score; a near-zero correlation would falsify the claim that the score ranks samples by usefulness.","tokens_in":18860,"feed_emoji":"🎯","tokens_out":8255,"duration_ms":81965,"temperature":0.7,"pith_summary":"The paper proposes a new rule for instruction data selection: choose training samples by how well they point a fine-tuned model toward the human-preference direction of the target task, rather than by how hard or accurate the response generation looks. ProDS first warms up a model with a small supervised fine-tuning set and a DPO step, then builds positive and negative preference pairs from target instructions whose responses are scored by a judge model. Each candidate sample is represented by a projected gradient of the supervised loss, and its score is the cosine similarity between that gradient and the DPO preference gradients, combined as a weighted difference between positive and negative consistency. The paper reports that subsets selected this way outperform existing task-agnostic and targeted data-selection baselines and, at 5–20% of the training data, match or exceed fine-tuning on the full dataset.","feed_headline":"Preference-aligned data selection can beat full instruction tuning","feed_subtitle":"Gradient alignment with human-preference directions picks subsets that match or beat full-data fine-tuning.","key_machinery":"The central object is the bidirectional preference gradient. DPO warm-up turns human preference into a vector direction: for each validation pair, the DPO loss gradient points from a dispreferred response toward a preferred one, and the reversed pair points the other way. Candidate training samples are represented by randomly projected supervised-fine-tuning gradients, so that cosine similarity between a training gradient and a preference gradient measures how much a sample pushes the model in the preferred direction. The scoring formula combines these similarities through an instance-level weight optimized by simulated annealing, which is the mechanism that synthesizes positive and negative preference signals into a single rank.","core_discovery":"Instruction data selection benefits from treating 'preferred response' as a direction in model-parameter space rather than as a fixed target answer. The paper's central claim is that training examples whose fine-tuning gradient aligns with the DPO gradient of positive preference pairs, and does not align with the gradient of negative pairs, are the examples that transfer preference-aligned behavior to the target task. The score is formed per sample as a weighted difference of cosine similarities to the two preference directions, with the instance-level weight tuned by simulated annealing. On benchmarks with open-ended or reasoning-rich responses, the selected subsets are reported to beat full-data fine-tuning; on multiple-choice tasks where preferences carry little information beyond the correct option, the advantage is smaller.","pith_inferences":["This suggests the preference signal could be obtained without a full DPO warm-up, for instance from an existing reward model's gradients or from a small set of human preference judgments, which would cut the method's extra training cost.","A natural extension is to use the bidirectional score actively, re-scoring newly generated candidate responses as preferences accumulate, rather than only ranking a fixed pool.","The paper's response-length analysis shows the method is not merely selecting short answers; isolating which semantic preferences drive selection would be a direct next experiment.","If the gradient-overlap proxy holds across domains, the same scoring could apply to other alignment signals, such as safety or style preferences, by swapping the objective that defines the preference direction."],"forward_implications":["A data pool can be scored once offline, and the same scores reused across different target tasks and target models, since the expensive gradient computation is a one-time pass.","Small or architecturally different selection models can choose data for larger target models; the paper reports consistent selection-model transfer across several model families, which lowers the cost of running the method.","Separating positive from negative preference directions is load-bearing: collapsing them into a single preference score degraded performance in the paper's ablation.","The selection rule should be most valuable where responses have rich preference structure, such as dialogue and chain-of-thought reasoning, and least valuable on tasks where the correct answer is a single option."],"supporting_citations":[{"why":"Supplies the DPO objective whose gradients define the preference directions used to score training samples.","marker":"Rafailov et al., 2023"},{"why":"Provides the gradient-based influence estimation, random projection, and targeted data-selection framework that ProDS builds on.","marker":"Xia et al., 2024"},{"why":"Defines the instruction-following difficulty scoring method that serves as the main target-agnostic baseline and the Alpaca experimental setup.","marker":"Li et al., 2024b"},{"why":"Supplies simulated annealing, the optimization procedure used to set the per-sample balance between positive and negative preference scores.","marker":"Kirkpatrick et al., 1983"},{"why":"Supports the random projection technique that compresses gradient vectors so the cosine-similarity scoring is memory-efficient.","marker":"Park et al., 2023"},{"why":"Supports using gradients to estimate sample influence for instruction data selection, the assumption underlying the scoring rule.","marker":"Zhang et al., 2024"}],"fun_headline_variants":["Preference-directed data selection beats full tuning","Data selection via preference gradients outperforms full sets","Aligning data with human preferences picks better subsets","Preference-aware subset selection matches or beats full data","ProDS: pick data by preference direction, beat full data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's usefulness rests on one premise: the direction in which a training example would push the model during supervised fine-tuning is a trustworthy guide to how much that example improves preference-aligned behavior on the target task, so examples whose directions match the preference direction are the ones worth keeping.","fun_headline_variants_meta":{"raw":{"variants":["Preference-directed data selection beats full tuning","Data selection via preference gradients outperforms full sets","Aligning data with human preferences picks better subsets","Preference-aware subset selection matches or beats full data","ProDS: pick data by preference direction, beat full data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000239,"raw_usage":{"total_tokens":1449,"prompt_tokens":816,"completion_tokens":633,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":432,"completion_tokens_details":{"reasoning_tokens":559}},"tokens_in":432,"tokens_out":633,"duration_ms":6737,"temperature":1.0,"reasoning_tokens":559,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:27:00.705159+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rank the same training pool with the ProDS score and with a deliberately reversed score that swaps positive and negative preferences, train on the top subsets of each, and compare on the target benchmark; if the reversed ranking performs as well, the bidirectional preference signal is not what drives the gains. A cleaner test would train each candidate example alone and correlate its measured effect on target-preference accuracy with its ProDS score; a near-zero correlation would falsify the claim that the score ranks samples by usefulness.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies simulated annealing, the optimization procedure used to set the per-sample balance between positive and negative preference scores."}],"review_version":1}