Pith. sign in

REVIEW 15 cited by

ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.10505 v4 pith:3B3QFTDF submitted 2023-10-16 cs.LG

Ziniu Li , Tian Xu , Yushun Zhang , Zhihang Lin , Yang Yu , Ruoyu Sun , Zhi-Quan Luo This is my paper
classification cs.LG
keywords remaxrlhftraininglearningllmsmodelmodelsreinforcement
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Reinforcement Learning from Human Feedback (RLHF) is key to aligning Large Language Models (LLMs), typically paired with the Proximal Policy Optimization (PPO) algorithm. While PPO is a powerful method designed for general reinforcement learning tasks, it is overly sophisticated for LLMs, leading to laborious hyper-parameter tuning and significant computation burdens. To make RLHF efficient, we present ReMax, which leverages 3 properties of RLHF: fast simulation, deterministic transitions, and trajectory-level rewards. These properties are not exploited in PPO, making it less suitable for RLHF. Building on the renowned REINFORCE algorithm, ReMax does not require training an additional value model as in PPO and is further enhanced with a new variance reduction technique. ReMax offers several benefits over PPO: it is simpler to implement, eliminates more than 4 hyper-parameters in PPO, reduces GPU memory usage, and shortens training time. ReMax can save about 46% GPU memory than PPO when training a 7B model and enables training on A800-80GB GPUs without the memory-saving offloading technique needed by PPO. Applying ReMax to a Mistral-7B model resulted in a 94.78% win rate on the AlpacaEval leaderboard and a 7.739 score on MT-bench, setting a new SOTA for open-source 7B models. These results show the effectiveness of ReMax while addressing the limitations of PPO in LLMs.

Discussion (0). Sign in to comment.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Formalizing Learning from Language Feedback with Provable Guarantees

    cs.LG 2025-06 conditional novelty 7.0 of 10

    Introduces a formal framework for learning from language feedback, a transfer eluder dimension complexity measure, and HELiX, a no-regret algorithm whose regret scales with this dimension.

  2. When RLVR Shrinks the Reasoning Boundary: Diagnosing Pass@k Inversion

    cs.LG 2026-07 conditional novelty 6.0 of 10

    RLVR can destroy rare-but-recoverable correct trajectories on 'boundary' prompts; the PBA frozen-base anchor preserves them, improving both pass@1 and pass@256 over matched GRPO on Omni-MATH-Test.

  3. Learning Decentralized LLM Collaboration with Multi-Agent Actor Critic

    cs.AI 2026-01 unverdicted novelty 6.0 of 10

    Multi-agent actor-critic methods with a centralized critic improve decentralized LLM collaboration over Monte Carlo baselines in long-horizon and sparse-reward settings.

  4. Inpainting-Guided Policy Optimization for Diffusion Large Language Models

    cs.LG 2025-09 reject novelty 6.0 of 10

    IGPO injects partial ground-truth reasoning hints into masked diffusion LLM generation during RL sampling, restoring non-zero advantages and improving math benchmarks, but as written the RL training set overlaps the e...

  5. DeepForm: Reasoning Large Language Model for Communication System Formulation

    cs.LG 2025-06 conditional novelty 6.0 of 10

    DeepForm, a 7B LLM fine-tuned on the new CSFRC dataset, reports the highest accuracy on a communication system formulation test, surpassing larger models such as DeepSeek R1.

  6. Thompson Sampling in Online RLHF with General Function Approximation

    cs.LG 2025-05 reject novelty 6.0 of 10

    A model-free posterior sampling algorithm for online RLHF is shown to achieve O(sqrt(T)) regret when the completed function class has low Bellman eluder dimension.

  7. RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents

    cs.LG 2026-07 conditional novelty 5.5 of 10

    Reward-Swap Policy Optimization uses dense-reward exploration to generate diverse trajectories that then train the final policy strictly under true outcome rewards, lifting multi-turn agent performance.

  8. Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Models

    cs.CL 2026-07 conditional novelty 5.0 of 10

    Cross-fitted residualization of recurring teacher–reference disagreement lets offline OPD work when the SFT generator and distillation teacher differ, beating uncorrected Lightning OPD on math and code.

  9. QueryBandits for Hallucination Mitigation: Exploiting Semantic Features for No-Regret Rewriting

    cs.CL 2025-08 conditional novelty 5.0 of 10

    A contextual bandit that chooses among five query-rewrite strategies, conditioned on 17 linguistic features, reduces LLM hallucination on QA benchmarks and beats static prompting and no-rewrite baselines.

  10. Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)

    cs.LG 2025-07 conditional novelty 5.0 of 10

    SFT on curated data is a lower bound on a sparse-reward RL objective, and an importance-weighted variant, iw-SFT, tightens the bound and beats plain SFT on AIME 2024 and GPQA.

  11. Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Training

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Reinforcement fine-tuning largely prevents catastrophic forgetting during continual post-training of a multimodal LLM, while supervised fine-tuning degrades both task and general performance.

  12. Variance-Aware Baselines and Adaptive Learning Rates for Reinforcement Learning with Verifiable Rewards

    stat.ML 2025-11 reject novelty 4.0 of 10

    OBLR-PO combines an SNR-based learning rate and a gradient-weighted baseline for RLVR, but the main theory is undermined by a flawed smoothness proof.

  13. Reasoner for Real-World Event Detection: Scaling Reinforcement Learning via Adaptive Perplexity-Aware Sampling Strategy

    cs.LG 2025-07 conditional novelty 4.0 of 10

    APARL combines a pass-rate-based adaptive sampler with KL-regularized DAPO reinforcement learning and reports F1 improvements of 17.19% in-domain and 9.59% out-of-domain for customer service anomaly detection.

  14. Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective

    cs.LG 2025-06 conditional novelty 3.0 of 10

    Under the assumption that the response reward equals the discounted sum of token rewards, response-level rewards suffice for unbiased token-level policy gradients in LLMs.

  15. Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges

    cs.AI 2025-07 reject novelty 1.0 of 10

    A broad survey of LLM alignment that catalogs objectives, benchmarks, SFT/RLHF/DPO methods, and safety challenges, without contributing new experimental or theoretical results.

Pith tools