Pith. sign in

REVIEW 3 major objections 6 minor 13 references

Standard reinforcement learning still forgets on new multimodal tasks; CPO cuts that forgetting without replaying old data.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 16:21 UTC pith:GIHBBBL6

load-bearing objection Solid empirical correction to the “RL forgets less” story on a modern multimodal sequence, with a practical replay-free regularizer; the Fisher-proxy story is under-validated but the main result still stands. the 3 major comments →

arxiv 2607.04364 v2 pith:GIHBBBL6 submitted 2026-07-05 cs.LG

RL Forgets! Towards Continual Policy Optimization

Mao-Lin Luo , Zhe-Xu Wang , Zi-Hao Zhou , Bo Ye , Jian Zhao , Min-Ling Zhang , Tong Wei This is my paper
classification cs.LG
keywords continual learningreinforcement learningvision-language modelscatastrophic forgettingpolicy optimizationbehavioral KLparameter-movement regularizationMRCL benchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper challenges the popular claim that reinforcement learning is naturally safe for continual post-training of vision-language models. On a new benchmark of recent, diverse multimodal reasoning tasks (medical, navigation, geometry, puzzles, finance), ordinary RL methods such as GRPO and GSPO still lose large amounts of earlier skill while learning the next task. The authors locate the failure in a mismatch: common KL regularizers only look at the current task, whereas forgetting is caused by the policy drifting on earlier task distributions. They replace that mismatched term with Continual Policy Optimization (CPO), a replay-free method that protects the parameters that moved most on previous tasks. Across 2B–8B models, CPO keeps more prior-task accuracy and even improves several out-of-domain pretrained capabilities relative to the base model.

Core claim

On the MRCL benchmark of recent multimodal reasoning tasks, standard policy-optimization algorithms still suffer severe catastrophic forgetting. The relevant constraint is a prior-task behavioral KL that cannot be computed without old data; CPO’s cumulative sparse parameter-movement regularizer is a practical surrogate for that constraint and substantially reduces forgetting while preserving plasticity.

What carries the argument

Continual Policy Optimization (CPO): after each task, add the top-p% largest-moving parameters to a cumulative protected set and apply a masked L1 penalty on those coordinates during the next RL update, serving as a gradient-free proxy for the old-task Fisher / prior-task behavioral KL.

Load-bearing premise

The method assumes that the parameters that moved the most on a finished task are the ones most important for preserving that task’s behavior, so ranking by absolute movement is a reliable stand-in for Fisher importance.

What would settle it

Measure the actual rank correlation between absolute parameter movement and the old-task Fisher diagonal (or the true prior-task KL) on the same VLM; if the top-movement mask systematically fails to protect the highest-Fisher parameters, CPO’s forgetting reductions should disappear.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues that reinforcement learning is not intrinsically resistant to catastrophic forgetting in continual VLM post-training. It introduces MRCL, a five-task multimodal reasoning benchmark built from 2025+ datasets, and shows that GRPO/GSPO still forget substantially. The authors attribute this to an objective mismatch: standard KL penalties are evaluated on current-task data, whereas forgetting is controlled by prior-task behavioral KL (Prop. 3.1). They propose Continual Policy Optimization (CPO), which relaxes the prior-task KL via a Fisher approximation and then uses cumulative top-p absolute parameter movement as a gradient-free ranking proxy (Prop. 3.2), applying masked L1 regularization on the protected set without replay. Across Qwen3-VL-2B/4B/8B, CPO improves MFN over GSPO (e.g., +13.67 pp on 8B) and often raises average pretrained capability on external benchmarks.

Significance. If the empirical findings hold, the paper usefully corrects an emerging narrative that RL post-training is largely forgetting-free, and supplies a timely, contamination-aware multimodal continual-learning benchmark. CPO is a practical, replay-free add-on to modern policy optimizers with released code, stage-wise matrices, multi-scale results, external capability tables, and an L1-vs-L2 ablation. Prop. 3.1 is a carefully proved finite-horizon KL–TV–return bound (Appendix A.1) that cleanly separates current-task from prior-task KL. These are real contributions. The significance of the theory-to-method claim, however, depends on whether movement ranking is a valid Fisher proxy on the models used; that link is currently untested.

major comments (3)
  1. Prop. 3.2 and §3.4–3.5: the central claim that CPO is “grounded in a prior-task behavioral KL objective” rests on absolute parameter movement ranking the old-task Fisher diagonal when C_k is approximately homogeneous (or under the pairwise condition Eq. 66). The manuscript never estimates F_kk, C_k, ρ_k, q_k, or any rank correlation (Spearman/Kendall) between |Δθ_k| and F_kk on any Qwen3-VL checkpoint. Without that check, the protected set S^(i) may not control prior-task KL, and the 13.7-point MFN gain cannot be attributed to the claimed surrogate. At minimum, report rank correlation on a subset of layers/checkpoints, or reframe CPO as an effective sparse-movement regularizer whose KL interpretation is conjectural.
  2. §4.1 and Appendix E: SFT/LoRA baselines use Direct (System-1) prompts while GRPO/GSPO/CPO use CoT prompts with boxed answers. This confounds the headline comparison that “RL forgets less than SFT” (Fig. 1; Tables 2–4). The fair within-RL comparison (CPO vs GSPO/GRPO) is not confounded, but claims about RL vs SFT forgetting should either match the prompt regime or be explicitly limited to same-prompt settings. Please re-run at least one SFT baseline under CoT (or one RL baseline under Direct) on one model scale.
  3. §3.5 / Algorithm 1 and experimental setup: p=10% and λ=100 are fixed by hand with no sensitivity study in the main results. Because the method’s only free design choices are exactly these two, a small grid (e.g., p ∈ {5,10,20}%, λ over an order of magnitude) on at least the 2B model is needed to show that the MFN gains are not a narrow hyperparameter sweet spot. The L1-vs-L2 ablation (Appendix C) is helpful but does not replace a p/λ sensitivity analysis.
minor comments (6)
  1. Abstract and §1: “reduces forgetting by 13.7%” is a percentage-point MFN gap (75.46 vs 61.79), not a relative reduction; state units consistently (pp vs %).
  2. Fig. 1 axes (CAD/CAI) are defined only in Appendix B; a one-sentence definition in the caption would help readers.
  3. Table 4 omits GRPO* for the 8B model while Tables 2–3 include it; either add the run or note the omission.
  4. Eq. (15): the normalization by ∥m∥_0 is sensible but not discussed; a brief justification would clarify scale invariance across stages as the protected set grows.
  5. Related work could more explicitly position CPO against online EWC / SI / MAS-style importance weighting in the RL (not only SFT) setting, beyond the brief Fisher discussion in §3.3.
  6. Typographical: title spacing (“TOWARDSCONTINUALPOLICY”), “T arget”, “Prior T asks” in Fig. 1, and occasional missing spaces in headings.

Circularity Check

0 steps flagged

No significant circularity: prior-task KL bound, Fisher relaxation, and movement proxy are independent derivations; reported gains are held-out measurements, not tautologies of the inputs.

full rationale

Walking the load-bearing chain: (i) Prop. 3.1 bounds old-task return degradation by mean token-level KL under the old-policy prefix distribution via TV + Pinsker + autoregressive chain rule—standard information-theoretic argument, not defined from the evaluation metrics. (ii) Eqs. 6–10 are the usual second-order expansion of that KL around the old checkpoint, with Hessian = Fisher; again independent of MRCL accuracies. (iii) Prop. 3.2 only claims that expected squared movement ranks like the Fisher diagonal when C_k is roughly homogeneous (or pairwise under an inequality); it does not define F_kk from movement, nor does the paper fit C_k and then “predict” Fisher ranks. (iv) CPO’s top-p mask and λ are free hyperparameters; MFT/MFN/MTA and external-benchmark averages are measured on held-out task splits and out-of-domain suites never used to set those knobs, so the 13.7-point MFN gain is not forced by construction. Self-citations (KeepLoRA, spectral-imbalance work) appear only as baselines/related work and do not underwrite the forgetting bound or the CPO objective. Unvalidated homogeneity of C_k is a correctness/assumption risk, not circularity. Derivation is self-contained; score 0.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The central empirical claim rests on standard RL/CL math plus two modeling choices that are not independently measured on the VLMs: local second-order KL≈Fisher and movement-as-Fisher-rank. Free knobs λ and p directly control the stability–plasticity trade-off reported in the headline numbers. No new physical entities; the invented objects are the benchmark and the algorithm.

free parameters (4)
  • λ (masked L1 strength)
    Set to 100 for CPO; scales the movement penalty in Eq. 15 and directly trades plasticity for stability. Not derived; chosen for experiments.
  • p (top movement ratio)
    Top 10% coordinates enter the cumulative protected set S^(i) (Eq. 16). Determines which parameters are frozen-ish; hand-chosen.
  • GRPO* KL coefficient β
    β=0.04 when comparing current-task KL; used to argue current-task KL fails. Baseline hyperparameter, not CPO’s core, but affects the mismatch narrative.
  • GSPO/GRPO clip ranges and rollout count
    ε_low/ε_high and 8 rollouts, T=0.8, max length 2048 are fixed training choices that affect absolute MFT/MFN levels across all RL methods.
axioms (5)
  • domain assumption Terminal rewards on each task have finite range B_{i-1}, enabling the Pinsker/TV return bound in Prop. 3.1.
    Stated in §3.2; true for the paper’s 0–1 style accuracy rewards but would fail for unbounded rewards.
  • standard math Local second-order expansion of prior-task mean token KL around the old checkpoint equals the old-policy token-level Fisher (Eqs. 6–8), with higher-order remainder neglected.
    Standard KL–Fisher identity under local smoothness; invoked to justify parameter-space surrogates.
  • domain assumption Locally stationary SGD on task T_{i-1} so that Fisher, prefix distribution, and reward-score moments are treated as fixed while measuring movement (Prop. 3.2).
    Required for E[|Δθ_k|²] ∝ C_k F_kk; large multimodal RL runs are only approximately stationary.
  • ad hoc to paper Ranking by absolute parameter movement preserves Fisher ranking when C_k is roughly uniform (or pairwise condition Eq. 66 holds).
    Load-bearing for replacing Fisher estimation with movement masks; not verified empirically on the models.
  • domain assumption Prior-task data are unavailable (privacy/efficiency), so replay-based Fisher/EWC is impractical for large VLM post-training.
    Motivates the entire replay-free design (§1, §3.2).
invented entities (2)
  • MRCL benchmark independent evidence
    purpose: Provide a recent, diverse multimodal continual-learning testbed less contaminated by pretraining overlap.
    Constructed from MedBookVQA, ThinkMorph Navigation/Puzzle, We-Math2.0, FinMME with custom splits; evaluation artifact, not a physical entity.
  • Continual Policy Optimization (CPO) protected-set mechanism independent evidence
    purpose: Replay-free surrogate for prior-task behavioral KL via cumulative top-movement masked L1 regularization.
    Algorithmic construct defined by Eqs. 15–16 and Algorithm 1; falsifiable via ablation (L1 vs L2, vs GSPO) which the paper partially provides.

pith-pipeline@v1.1.0-grok45 · 35208 in / 4051 out tokens · 49926 ms · 2026-07-14T16:21:44.113897+00:00 · methodology

0 comments
read the original abstract

Continual post-training is becoming a central paradigm for adapting vision-language models to evolving tasks. Recent work has increasingly favored reinforcement learning over supervised fine-tuning, driven by the belief that reinforcement learning is inherently less prone to forgetting. However, the belief remains insufficiently validated, as existing evidence is largely drawn from outdated or homogeneous benchmarks. We revisit this assumption under recent and diverse multimodal reasoning tasks. To this end, we introduce MRCL, a Multimodal Reasoning Continual Learning benchmark. Experiments on MRCL show that standard reinforcement learning still suffers from severe catastrophic forgetting during continual post-training. We trace this failure to an objective mismatch: the KL regularization used in common policy optimization methods is evaluated on current-task data, whereas forgetting is caused by behavioral drift on prior-task distributions. To address this problem, we propose Continual Policy Optimization (CPO), a replay-free framework grounded in a prior-task behavioral KL objective. CPO relaxes the intractable historical KL constraint into sparse parameter-movement regularization, limiting policy drift without storing old data. Extensive experiments across multiple model scales show that CPO consistently reduces forgetting while preserving, and in some cases improving, pretrained model capabilities. On Qwen3-VL-8B, CPO reduces forgetting by 13.7% and improves pretrained capability by 7.0%. The implementation code is available at https://github.com/MaolinLuo/CPO.

Figures

Figures reproduced from arXiv: 2607.04364 by Bo Ye, Jian Zhao, Mao-Lin Luo, Min-Ling Zhang, Tong Wei, Zhe-Xu Wang, Zi-Hao Zhou.

Figure 1
Figure 1. Figure 1: Plasticity–stability Pareto frontier during continual post-training. The y-axis reports the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: An overview of MRCL benchmark. It incorporates newly released tasks designed to [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Parameter movement heatmaps of CPO and its masked [PITH_FULL_IMAGE:figures/full_fig_p020_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Regularization loss curves of CPO with masked [PITH_FULL_IMAGE:figures/full_fig_p020_4.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 10 linked inside Pith

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  2. [2]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025a

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025a. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zha...

    Pith/arXiv arXiv

  3. [3]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities.arXiv preprint arXiv:2507.06261,

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities.arXiv preprint arXiv:2507.06261,

    Pith/arXiv arXiv

  4. [4]

    RL fine-tuning heals ood forgetting in sft.arXiv preprint arXiv:2509.12235,

    Hangzhan Jin, Sitao Luan, Sicheng Lyu, Guillaume Rabusseau, Reihaneh Rabbany, Doina Pre- cup, and Mohammad Hamdaqa. RL fine-tuning heals ood forgetting in sft.arXiv preprint arXiv:2509.12235,

    Pith/arXiv arXiv

  5. [5]

    Reinforcement fine-tuning naturally mitigates forgetting in continual post-training.arXiv preprint arXiv:2507.05386,

    12 Preprint Song Lai, Haohan Zhao, Rong Feng, Changyi Ma, Wenzhuo Liu, Hongbo Zhao, Xi Lin, Dong Yi, Qingfu Zhang, Hongbin Liu, et al. Reinforcement fine-tuning naturally mitigates forgetting in continual post-training.arXiv preprint arXiv:2507.05386,

    Pith/arXiv arXiv

  6. [6]

    Evaluating object hal- lucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hal- lucination in large vision-language models. InProceedings of the 2023 conference on empirical methods in natural language processing, pp. 292–305,

    2023

  7. [7]

    Overcoming catastrophic forgetting in visual continual learning with reinforcement fine-tuning.arXiv preprint arXiv:2605.09640,

    Meng Lou, Hanzhong Guo, Linwei Chen, and Yizhou Yu. Overcoming catastrophic forgetting in visual continual learning with reinforcement fine-tuning.arXiv preprint arXiv:2605.09640,

    Pith/arXiv arXiv

  8. [8]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. InInternational Conference on Learning Representations, volume 2024, pp. 23439–23554,

    2024

  9. [9]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    Pith/arXiv arXiv

  10. [10]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  11. [11]

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hong- sheng Li

    doi: 10.1002/9781118576618.ch7. Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hong- sheng Li. Measuring multimodal mathematical reasoning with math-vision dataset.Advances in Neural Information Processing Systems, 37:95095–95169, 2024a. 13 Preprint Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive ...

    doi:10.1002/9781118576618.ch7 2023

  12. [12]

    Sau Lai Yip, Sunan He, Yuxiang Nie, Shu Pui Chan, Yilin Ye, Sum Ying Lam, and Hao Chen

    Accessed: 2025-04-26. Sau Lai Yip, Sunan He, Yuxiang Nie, Shu Pui Chan, Yilin Ye, Sum Ying Lam, and Hao Chen. Medbookvqa: A systematic and comprehensive medical benchmark derived from open-access book.arXiv preprint arXiv:2506.00855,

    Pith/arXiv arXiv 2025

  13. [13]

    Group sequence policy optimization.arXiv preprint arXiv:2507.18071,

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization.arXiv preprint arXiv:2507.18071,

    Pith/arXiv arXiv