Pith. sign in

REVIEW 11 cited by

Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2005.12729 v1 pith:MBXQASGD submitted 2020-05-25 cs.LG cs.ROstat.ML

Logan Engstrom , Andrew Ilyas , Shibani Santurkar , Dimitris Tsipras , Firdaus Janoos , Larry Rudolph , Aleksander Madry This is my paper · ORCID
classification cs.LGcs.ROstat.ML
keywords policydeeptrpoalgorithmalgorithmscaseimportanceoptimization
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We study the roots of algorithmic progress in deep policy gradient algorithms through a case study on two popular algorithms: Proximal Policy Optimization (PPO) and Trust Region Policy Optimization (TRPO). Specifically, we investigate the consequences of "code-level optimizations:" algorithm augmentations found only in implementations or described as auxiliary details to the core algorithm. Seemingly of secondary importance, such optimizations turn out to have a major impact on agent behavior. Our results show that they (a) are responsible for most of PPO's gain in cumulative reward over TRPO, and (b) fundamentally change how RL methods function. These insights show the difficulty and importance of attributing performance gains in deep reinforcement learning. Code for reproducing our results is available at https://github.com/MadryLab/implementation-matters .

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multi-Turn On-Policy Distillation with Prefix Replay

    cs.LG 2026-07 conditional novelty 6.0 of 10

    ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.

  2. Understanding electricity consumption behaviour through Inverse Reinforcement Learning

    cs.LG 2026-07 conditional novelty 6.0 of 10

    AIRL-recovered reward functions show that the 2022 energy crisis and heatwave reshaped Italian households' cooling responses heterogeneously, sometimes durably, with time-of-use as an independent dimension.

  3. Personalized Observation Normalization for Federated Reinforcement Learning in Simulation Environments with Heterogeneity

    cs.LG 2026-04 conditional novelty 6.0 of 10

    Personalized local running-mean/variance observation normalization prevents weight-norm overshadowing in FedAvg and improves FedRL-PPO on heterogeneous MuJoCo morphology variants.

  4. SERA: Soft-Verified Efficient Repository Agents

    cs.CL 2026-01 conditional novelty 6.0 of 10

    SERA trains competitive coding agents via supervised finetuning on synthetic trajectories verified only by patch-overlap recall, cutting data-generation cost by roughly two orders of magnitude.

  5. Shared Control of Holonomic Wheelchairs through Reinforcement Learning

    cs.RO 2025-07 conditional novelty 6.0 of 10

    An RL policy trained in Isaac Gym and tested in Gazebo and on a real DAA V1 wheelchair translates 2D joystick commands into collision-free 3D motion for a holonomic wheelchair.

  6. On the Effect of Regularization in Policy Mirror Descent

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A large empirical sweep shows that in Policy Mirror Descent, MDP and Drift regularizers are partly substitutable, yet their precise combination determines temperature robustness.

  7. Thompson Sampling in Online RLHF with General Function Approximation

    cs.LG 2025-05 reject novelty 6.0 of 10

    A model-free posterior sampling algorithm for online RLHF is shown to achieve O(sqrt(T)) regret when the completed function class has low Bellman eluder dimension.

  8. Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environments

    cs.LG 2026-03 conditional novelty 5.0 of 10

    PPO plateaus can be avoided by increasing the number of parallel environments, which reduces both the outer-loop step size and update noise; scaling to 1M environments sustained improvement to 1T transitions.

  9. Shallow Preference Signals: Large Language Model Aligns Even Better with Truncated Data?

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Preference signals in LLM alignment are concentrated in early response tokens, so models trained on data truncated to the first half perform as well as or better than those trained on full responses.

  10. ODYSSE: Episode-wise Policy Optimization for Personalized Agentic Reasoning

    cs.AI 2026-07 conditional novelty 4.0 of 10

    An episode-level variant of GRPO (ESPO) improves personalized GUI-agent reasoning on the 102-episode SmartSpot benchmark, outperforming step-wise and outcome-only training baselines.

  11. Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges

    cs.AI 2025-07 reject novelty 1.0 of 10

    A broad survey of LLM alignment that catalogs objectives, benchmarks, SFT/RLHF/DPO methods, and safety challenges, without contributing new experimental or theoretical results.

Pith tools