Pith. sign in

REVIEW 5 cited by

Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2503.12854 v3 pith:5UTQ3P6R submitted 2025-03-17 cs.CL

Songjun Tu , Jiahao Lin , Xiangyu Tian , Qichao Zhang , Linjing Li , Yuqian Fu , Nan Xu , Wei He
show 3 more authors
Xiangyuan Lan Dongmei Jiang Dongbin Zhao
This is my paper · ORCID
classification cs.CL
keywords reasoningenhancingiterativemodelalternativecomputationallearningllms
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Recent advancements in post-training methodologies for large language models (LLMs) have highlighted reinforcement learning (RL) as a critical component for enhancing reasoning. However, the substantial computational costs associated with RL-based approaches have led to growing interest in alternative paradigms, such as Direct Preference Optimization (DPO). In this study, we investigate the effectiveness of DPO in facilitating self-improvement for LLMs through iterative preference-based learning. We demonstrate that a single round of DPO with coarse filtering significantly enhances mathematical reasoning performance, particularly for strong base model. Furthermore, we design an iterative enhancement framework for both the generator and the reward model (RM), enabling their mutual improvement through online interaction across multiple rounds of DPO. Finally, with simple verifiable rewards, our model DPO-VP achieves RL-level performance with significantly lower computational overhead. These findings highlight DPO as a scalable and cost-effective alternative to RL, offering a practical solution for enhancing LLM reasoning in resource-constrained situations.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Your Self-Play Algorithm is Secretly an Adversarial Imitator: Understanding LLM Self-Play through the Lens of Imitation Learning

    cs.LG 2026-02 conditional novelty 6.0 of 10

    LLM self-play finetuning is equivalent to adversarial imitation learning; the chi-squared regularized variant SPIF bounds rewards and improves stability.

  2. TeaRAG: A Token-Efficient Agentic Retrieval-Augmented Generation Framework

    cs.IR 2025-11 conditional novelty 6.0 of 10

    TeaRAG shows that hybrid chunk+triplet retrieval with Personalized PageRank and an iterative process-aware DPO reward keeps QA accuracy while cutting reasoning tokens by roughly 60%.

  3. Mathesis: Towards Formal Theorem Proving from Natural Languages

    cs.AI 2025-06 conditional novelty 6.0 of 10

    An RL-trained autoformalizer plus a Lean prover solves 18% of Chinese Gaokao proof problems end-to-end from natural language, and 64.3% of MiniF2F at pass@32.

  4. RoboGPT-R1: Enhancing Robot Task Planning with Reinforcement Learning

    cs.AI 2025-10 conditional novelty 5.0 of 10

    A 3B VLM trained with SFT plus GRPO and an LCS-based reward reaches 55.3% on EmbodiedBench's EB-ALFRED, beating GPT-4o-mini and the 7B REBP planner.

  5. Enhancing Large Language Models through Structured Reasoning

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Structured reasoning tags plus a max-flow reward let a 1.5B model match the math accuracy of models trained for far longer, but the gains are within statistical noise.

Pith tools