Pith. sign in

REVIEW 4 major objections 6 minor 10 references

Improving LLM Safety and Helpfulness using SFT and DPO: A Study on OPT-350M

T0 review · 4 major / 6 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that SFT followed by DPO gives the best overall safety-helpfulness balance for OPT-350M, with the highest combined alignment score and helpfulness rate among four training recipes.

desk verdict The paper's own Table 1 refutes its headline claim—SFT+DPO does not beat SFT on harmlessness—so it is not review-ready, though a credible weaker result and open code/data are there. read the letter →

arxiv 2509.09055 v1 pith:7P5RFP2A submitted 2025-09-10 cs.CL cs.AIcs.LG

Piyush Pant This is my paper
classification cs.CLcs.AIcs.LG
keywords LLMalignmentDirectPreferenceOptimizationSupervisedFine-TuningrewardmodelevaluationharmlessnesshelpfulnessOPT-350MHH-RLHFdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks how to make a small, affordable language model safer and more helpful: through supervised fine-tuning alone, direct preference optimization alone, or by doing SFT first and then DPO. Its central claim is that the sequential SFT+DPO pipeline delivers the best overall alignment on a reward-model-scored benchmark, with the highest combined alignment score and helpfulness rate, and that SFT and DPO are complementary rather than interchangeable. This matters because small models like OPT-350M are widely deployed in settings where larger, costlier alignment pipelines are infeasible, so knowing the right cheap recipe has direct practical value. The paper also identifies data noise, weak base-model responses, and unequal training budgets as factors that likely held DPO back in its head-to-head with SFT.

What carries the argument

The evaluation machinery is a reward-model threshold classifier: each prompt+response pair gets a scalar score from a DeBERTa-based preference reward model; scores below -3 count as harmful and scores above -2 count as helpful, yielding Harmlessness Rate and Helpfulness Rate whose mean is the Combined Alignment Score. The training machinery is the sequential pipeline: full SFT on chosen responses for two epochs, then LoRA-based DPO for one epoch.

What would settle it

Re-score or re-label the same 100 generated responses with human annotators or a different preference reward model. If the SFT+DPO model's Combined Alignment Score no longer tops SFT's, or if shifting the threshold by a small amount changes which model leads, the paper's ranking is not robust.

Watch

Extended reading notes

Core claim

On a fixed set of 100 prompts from the HH-RLHF test split, four OPT-350M variants—base, SFT, DPO, and SFT+DPO—were scored by a preference reward model and classified with thresholds. The paper's central claim is that the SFT+DPO model achieves the highest Combined Alignment Score (55%) and Helpfulness Rate (66%), demonstrating that SFT and DPO are complementary when applied sequentially. The reported tables additionally show SFT alone has the highest Harmlessness Rate (48%), while DPO alone scores below SFT on both HmR and HpR; the paper attributes DPO's underperformance to noisy preference labels, low-quality base-model responses, and DPO's shorter one-epoch LoRA training budget.

Load-bearing premise

The entire ranking rests on the assumption that the reward model's scalar scores, with hand-chosen thresholds of -3 for harm and -2 for help, correctly classify OPT-350M's outputs; if the reward model is biased or the thresholds are not meaningful, all three metrics and the final ranking collapse.

Editorial extensions

If this is right

  • Small-model alignment pipelines should put SFT before DPO: the reported Combined Alignment Score is 55% for SFT+DPO, versus 52% for SFT alone and 41% for DPO alone.
  • SFT alone is the strongest single-stage recipe for harmlessness (48% Harmlessness Rate), while DPO alone drops harmlessness below the base model (36% vs 42%).
  • Reward-model thresholding with HmR/HpR/CAS offers a cheap, reproducible evaluation template for models that cannot afford LLM-as-judge APIs.
  • DPO's value in this setup is as a helpfulness booster after SFT, not as a standalone alignment method.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The unequal training budgets (one LoRA epoch for DPO, two full epochs for SFT) mean the head-to-head 'SFT beats DPO' conclusion is an artifact-prone comparison rather than a clean property of the algorithms.
  • Because the reward model was trained on the same HH-RLHF preference distribution, the improved scores may partly reflect reward overfitting; a held-out human evaluation would test genuine alignment.
  • The thresholds sit near the bulk of scores, so a few responses crossing -3 or -2 could flip the ranking; the 3-point CAS gap between SFT+DPO and SFT is fragile.
  • The same SFT-to-DPO recipe could be tested on larger models; if helpfulness gains persist but harmlessness degrades further, the complementarity claim may not generalize.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies whether SFT, DPO, or SFT followed by DPO improves alignment of OPT-350M, using the Anthropic HH-RLHF dataset. Four models are evaluated on 100 prompts (50 harmful-selected, 50 helpful) using the OpenAssistant reward model. Three metrics are defined from reward scores with fixed thresholds: Harmlessness Rate (HmR), Helpfulness Rate (HpR), and Combined Alignment Score (CAS). The reported results are: Base (42/22/32), SFT (48/56/52), DPO (36/46/41), SFT+DPO (44/66/55). The abstract and conclusion claim that SFT+DPO outperforms all others across all metrics, but Table 1 shows SFT has higher HmR. The paper also reports average reward scores, with SFT+DPO best on helpfulness (-1.15) but not on harmlessness (-3.68 vs SFT -3.52). Limitations such as dataset noise, DPO trained for one LoRA epoch versus SFT for two full epochs, and low-quality base responses are acknowledged.

Significance. If the main claim were correct, the paper would provide a useful datapoint on aligning small models with limited compute, with a reproducible pipeline (open-source code/eval set). However, the central claim is internally contradicted by the paper's own results: SFT+DPO does not achieve the highest HmR. Moreover, threshold selection is post hoc and no uncertainty quantification is provided, so even the supported comparisons (HpR, CAS) are not established beyond sampling noise. The study's value is therefore as a preliminary engineering report rather than a validated scientific claim. Credit is due for using a reward model rather than an API, for releasing code, and for candidly listing limitations, but the evidence does not support the stated conclusion.

major comments (4)
  1. [§5.1, Table 1] The central claim—'SFT+DPO outperforms all others across all metrics'—is contradicted by the paper's own data. SFT+DPO HmR=44%, while SFT HmR=48% (Base 42%, DPO 36%). The average harmlessness score in §5.3 also favors SFT (-3.52) over SFT+DPO (-3.68). Thus the abstract, §6, and the 'across all evaluation metrics' statement must be revised. The data support only a weaker claim, e.g., best HpR and CAS.
  2. [§4.2.2] The thresholds defining harmlessness (< -3) and helpfulness (> -2) are chosen after manual inspection of the reward scores, not specified in advance or validated on an independent set. HmR and HpR are counts of thresholded scores, so the resulting ranking is sensitive to threshold placement. No sensitivity analysis, error bars, or significance tests are reported. Without these, 'outperforms' is not statistically supported even for the metrics where SFT+DPO is highest.
  3. [§4.1] The evaluation uses only 100 prompts: 50 harmlessness prompts obtained by filtering for 'kill, murder, or rape' and 50 randomly sampled helpful prompts. The harmful-prompt subset is narrow and may not reflect the distribution of the HH-RLHF harmless test set. With n=50 per condition, differences like 44% vs 48% are within sampling noise; the paper should provide confidence intervals or bootstrap estimates before ranking models.
  4. [§3.3–§3.5, §5.4] The comparison is confounded by unequal training budgets: SFT is trained for 2 full epochs, while DPO (standalone and after SFT) is trained for 1 epoch with LoRA. The paper itself identifies this as a limitation. Consequently, 'SFT outperforms DPO' is not a clean comparison of algorithms, and the improvement from SFT+DPO over SFT could reflect additional training rather than complementarity. The claims should be framed as a resource-constrained engineering comparison.
minor comments (6)
  1. [Abstract] Typo: 'The combined SFT+DPO model' has an uppercase 'The' after a comma; should be lowercase.
  2. [§2, Eq. (2)] The displayed DPO objective is not the standard DPO loss; the reference-model term and the sigmoid/logistic form are missing. Either correct the formula or clarify that this is a simplified exposition.
  3. [§5.3] Calling SFT+DPO's harmlessness score (-3.68) 'competitive' is misleading because SFT's -3.52 is better; the paper should state this directly.
  4. [§5.2] Figures 2 and 3 lack axis labels and detailed captions in the text, making the score distributions hard to interpret.
  5. [§3.4] Hyperparameters for LoRA (rank, alpha, learning rate) and DPO beta are not given in the text; they are essential for reproducibility even if code is linked.
  6. [References] References [4] and [6] are listed but not cited in the body; either cite them or remove them.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the central claim is an empirical summary that is internally contradicted by Table 1, but no derivation reduces to its inputs.

full rationale

The paper's claims are empirical comparisons of trained models, not first-principles derivations. Evaluation metrics HmR, HpR, and CAS are defined in Section 4.2.3 from thresholded reward-model scores; the thresholds in Section 4.2.2 are post hoc but transparently disclosed and do not constitute a fitted parameter being relabeled as a prediction. The reward model is an external OpenAssistant model, not a self-citation, and the test prompts come from the HH-RLHF test split rather than the training set. No uniqueness theorem, ansatz-by-citation, or definitional equivalence is invoked. The abstract's claim that SFT+DPO 'outperforms all others across all metrics' is contradicted by Table 1 (SFT HmR 48% vs. SFT+DPO 44%), but this is an internal factual inconsistency, not circularity. Limitations (noisy data, one-epoch DPO, low-quality base responses) are acknowledged in Sections 5.4 and 6 and affect generalizability, not the logical independence of the evaluation.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The ledger shows two fitted thresholds that directly determine the reported metrics, several unreported but influential training hyperparameters, and three domain assumptions about the reward model, dataset quality, and model representativeness. No new physical or theoretical entities are introduced.

free parameters (4)
  • harmless threshold = -3 reward score
    Chosen after manual inspection and experimentation with a wide range of samples to classify responses as harmful vs harmless (Section 4.2.2). Directly determines HmR.
  • helpful threshold = -2 reward score
    Chosen after manual inspection of reward model scores to classify responses as helpful vs unhelpful (Section 4.2.2). Directly determines HpR.
  • DPO beta
    The DPO objective uses a temperature parameter beta, but its value is not reported in the paper. It likely uses a TRL default, but it is a free hyperparameter affecting training.
  • LoRA rank and learning rate
    The paper states LoRA was used for DPO (Section 3.4) but does not report the rank, alpha, or learning rate, all of which affect the outcome.
assumptions (3)
  • domain assumption Reward scores from OpenAssistant/reward-model-deberta-v3-large-v2 accurately reflect human judgments of helpfulness and harmlessness.
    The entire evaluation pipeline relies on this reward model as ground truth (Section 4.2.1). If the reward model is biased or miscalibrated for OPT-350M's outputs, all reported metrics and the conclusion are invalid.
  • domain assumption The Anthropic HH-RLHF chosen/rejected labels are sufficiently reliable for alignment training despite acknowledged noise.
    The paper itself notes examples where the chosen response is as harmful as the rejected one (Section 5.4 and Appendix B). DPO training depends on the quality of these preferences.
  • domain assumption OPT-350M is a representative testbed for small language models.
    The paper generalizes from this single model to 'smaller LLMs' (Section 1). The similarity of OPT-350M to other small models is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving LLM Safety and Helpfulness using SFT and DPO: A Study on OPT-350M." pith.science (2026). https://pith.science/paper/7P5RFP2A

@misc{pith2026250909055,
  author       = {Pith},
  title        = {Pith review of: Improving LLM Safety and Helpfulness using SFT and DPO: A Study on OPT-350M},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7P5RFP2A}},
  note         = {Machine review of arXiv:2509.09055}
}
read the original abstract

This research investigates the effectiveness of alignment techniques, Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and a combined SFT+DPO approach on improving the safety and helpfulness of the OPT-350M language model. Utilizing the Anthropic Helpful-Harmless RLHF dataset, we train and evaluate four models: the base OPT350M, an SFT model, a DPO model, and a model trained with both SFT and DPO. We introduce three key evaluation metrics: Harmlessness Rate (HmR), Helpfulness Rate (HpR), and a Combined Alignment Score (CAS), all derived from reward model outputs. The results show that while SFT outperforms DPO, The combined SFT+DPO model outperforms all others across all metrics, demonstrating the complementary nature of these techniques. Our findings also highlight challenges posed by noisy data, limited GPU resources, and training constraints. This study offers a comprehensive view of how fine-tuning strategies affect model alignment and provides a foundation for more robust alignment pipelines in future work.

Figures

Figures reproduced from arXiv: 2509.09055 by the authors.

Figure 1
Figure 1. Comparison of HmR, HpR, and CAS across models [PITH_FULL_IMAGE:figures/full_fig_p011_1.png] view at source ↗
Figure 2
Figure 2. Distribution of Reward Scores for Harmlessness and Helpfulness [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Average Reward Scores for Harmlessness and Helpfulness across [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

10 extracted references · 10 linked inside Pith

  1. [1]

    D., Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv preprint arXiv:2305.18290

    arXiv 2023

  2. [2]

    V., Mihaylov, T., Ott, M., Shleifer, S., Shuster, K., Simig, D., Koura, P

    Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., Mihaylov, T., Ott, M., Shleifer, S., Shuster, K., Simig, D., Koura, P. S., Sridhar, A., Wang, T., Zettlemoyer, L. (2022). OPT: Open Pre-trained Transformer Language Models. arXiv preprint arXiv:2205.01068

    arXiv 2022

  3. [3]

    R., Christiano, P., Knight, M., Kaplan, J

    Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Hernandez, D., Hume, T., Kwon, M., Lee, A., Leike, J., Lightman, K., McKinnon, C., Mikulik, V., Miller, J., Mindermann, S., Nye, M., Olsson, C., Rauh, M., Ringer, S., Schiefer, N., Schlatter, J., Schulman, J., Smith, N., Snyder, C., Sorense...

    arXiv 2022

  4. [4]

    A., Khashabi, D

    Wang, W., Kordi, Y., Mishra, S., Liu, P., Smith, N. A., Khashabi, D. (2023). How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources. arXiv preprint arXiv:2306.04751

    arXiv 2023

  5. [5]

    S., Le, T.-T., Ngo Van, L., Nguyen, T

    Luong, T. S., Le, T.-T., Ngo Van, L., Nguyen, T. H. (2024). Realistic Evaluation of Toxicity in Large Language Models. arXiv preprint arXiv:2405.10659

    arXiv 2024

  6. [6]

    Zheng, L., Wang, Y., Chang, K.-W. (2024). Generative AI for Peer Assess- ment Helpfulness Evaluation. arXiv preprint arXiv:2405.01805

    arXiv 2024

  7. [7]

    Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., Testuggine, D. (2021). Dynabench: Rethinking Benchmarking in NLP. arXiv preprint arXiv:2104.14337

    arXiv 2021

  8. [8]

    M., Stiennon, N., Wu, J., Brown, T., Radford, A., Amodei, D., Christiano, P

    Ziegler, D. M., Stiennon, N., Wu, J., Brown, T., Radford, A., Amodei, D., Christiano, P. F., Irving, G. (2019). Fine-Tuning Language Models from Human Preferences. arXiv preprint arXiv:1909.08593

    arXiv 2019

Show all 10 references
  1. [9]

    Pérez, J., Marasović, A., Ferrando, A. (2022). Red Teaming Language Models with Language Models. arXiv preprint arXiv:2202.03286

    2022 arXiv

  2. [10]

    N., Baral, C

    Saeidi, A., Verma, S., Uddin, M. N., Baral, C. (2024). Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks. arXiv preprint arXiv:2404.14723. 16 A REPRODUCIBILITY We have open-sourced our codebase and the evaluation dataset used for our analysisat htt...

    2024 arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.