SSR-Zero: Simple Self-Rewarding Reinforcement Learning for Machine Translation

Mao Zheng; Mingyang Song; Sitong Wang; Wenjie Yang; Zheng Li

arxiv: 2505.16637 · v4 · submitted 2025-05-22 · 💻 cs.CL · cs.AI· cs.LG

SSR-Zero: Simple Self-Rewarding Reinforcement Learning for Machine Translation

Wenjie Yang , Mao Zheng , Mingyang Song , Zheng Li , Sitong Wang This is my paper

Pith reviewed 2026-05-22 13:47 UTC · model grok-4.3

classification 💻 cs.CL cs.AIcs.LG

keywords machine translationself-rewarding reinforcement learninglarge language modelsreinforcement learningself-improvementreference-free training

0 comments

The pith

Self-rewarding reinforcement learning trains a 7B model to outperform larger machine translation systems without reference data.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that a simple self-rewarding reinforcement learning approach can train effective machine translation models using only the model's own self-judgments as rewards. This matters because it removes dependence on expensive human-annotated references or pre-trained reward models that are difficult to scale. Training with 13K monolingual examples on a Qwen-2.5-7B backbone produces SSR-Zero-7B, which surpasses MT-specific models like TowerInstruct-13B and larger general models like Qwen2.5-32B-Instruct on English-Chinese tasks across WMT23, WMT24, and Flores200. Adding external signals such as COMET creates an even stronger model that reaches state-of-the-art among open models under 72B parameters.

Core claim

The paper claims that its Simple Self-Rewarding (SSR) RL framework, which is reference-free and fully online and relies solely on self-judging rewards, enables SSR-Zero-7B to outperform existing MT-specific LLMs such as TowerInstruct-13B and GemmaX-28-9B as well as larger general LLMs like Qwen2.5-32B-Instruct in English ↔ Chinese translation tasks from WMT23, WMT24, and Flores200 benchmarks. Augmenting SSR with external supervision from COMET further yields SSR-X-Zero-7B that achieves state-of-the-art performance among open-source models under 72B parameters.

What carries the argument

Simple Self-Rewarding (SSR) Reinforcement Learning, a reference-free online RL method that uses the model's self-generated judgments on its own outputs to supply reward signals for improving translation quality.

If this is right

Self-rewarding RL can scale machine translation training without relying on costly external supervision or reference translations.
Combining self-rewards with trained metrics like COMET produces further gains and state-of-the-art results among open models.
The self-rewarding mechanism shows advantages over external LLM-as-a-judge approaches for machine translation.
The findings indicate that self-improving RL methods hold promise for reducing supervision needs in language tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The approach could extend to other text generation tasks where reference data is scarce, enabling lighter supervision regimes.
Iterative self-reward loops might eventually allow models to improve autonomously across multiple rounds of training.
Testing whether self-judgments remain stable when applied to languages or domains outside English-Chinese would reveal broader limits.

Load-bearing premise

The model's self-generated judgments provide sufficiently reliable and unbiased reward signals to drive consistent improvements in translation quality via RL without reward hacking or performance collapse on held-out benchmarks.

What would settle it

A sustained drop in translation quality scores on WMT or Flores benchmarks during SSR training, or a clear mismatch between the model's self-judgments and independent quality metrics such as COMET or human ratings, would falsify the claim.

read the original abstract

Large language models (LLMs) have recently demonstrated remarkable capabilities in machine translation (MT). However, most advanced MT-specific LLMs heavily rely on external supervision signals during training, such as human-annotated reference data or trained reward models (RMs), which are often expensive to obtain and challenging to scale. To overcome this limitation, we propose a Simple Self-Rewarding (SSR) Reinforcement Learning (RL) framework for MT that is reference-free, fully online, and relies solely on self-judging rewards. Training with SSR using 13K monolingual examples and Qwen-2.5-7B as the backbone, our model SSR-Zero-7B outperforms existing MT-specific LLMs, e.g., TowerInstruct-13B and GemmaX-28-9B, as well as larger general LLMs like Qwen2.5-32B-Instruct in English $\leftrightarrow$ Chinese translation tasks from WMT23, WMT24, and Flores200 benchmarks. Furthermore, by augmenting SSR with external supervision from COMET, our strongest model, SSR-X-Zero-7B, achieves state-of-the-art performance in English $\leftrightarrow$ Chinese translation, surpassing all existing open-source models under 72B parameters and even outperforming closed-source models. Our analysis highlights the effectiveness of the self-rewarding mechanism compared to the external LLM-as-a-judge approach in MT and demonstrates its complementary benefits when combined with trained RMs. Our findings provide valuable insight into the potential of self-improving RL methods. We have publicly released our code, data and models.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

Self-rewarding RL lets a 7B model beat larger ones on En-Zh MT benchmarks from monolingual data, but self-judgment reliability still needs direct checks.

read the letter

SSR-Zero shows that you can train a decent MT model with self-rewarding RL on a small set of monolingual sentences, and it beats some established larger models on standard tests. What stands out is the simple, fully online setup. They take Qwen-2.5-7B, use 13K monolingual examples, have the model judge its own translations for rewards, and run RL. The resulting SSR-Zero-7B does better than TowerInstruct-13B, GemmaX-28-9B, and Qwen2.5-32B-Instruct on English to Chinese and back on WMT23, WMT24, and Flores200. They also have a hybrid version with COMET that pushes further. Releasing the code, data, and models is a plus for reproducibility. The soft spot is in the self-rewarding core. The concern is real that the model's own judgments might not be unbiased or stable enough. If the judge favors certain styles or lengths that don't match real quality, the RL could optimize for that instead. The paper claims the self-rewarding works better than using an external LLM judge and combines well with trained rewards, but without seeing the actual correlation numbers between self-scores and external metrics on the training outputs, it's difficult to rule out some degree of reward hacking. The abstract lacks specifics on the reward formulation and RL hyperparameters, though the public code should let people verify that. This paper is for folks in MT or LLM alignment who are looking for ways to cut down on supervised data. A reader who wants to experiment with self-improvement loops would get value from the concrete setup and results. It deserves a serious referee. The claims are testable with the released artifacts, and the idea addresses a real pain point in scaling MT. I would recommend sending it for peer review.

Referee Report

3 major / 2 minor

Summary. The paper introduces SSR-Zero, a reference-free and fully online self-rewarding reinforcement learning framework for machine translation. Using only 13K monolingual source examples and Qwen-2.5-7B as backbone, the resulting SSR-Zero-7B model is claimed to outperform MT-specific LLMs such as TowerInstruct-13B and GemmaX-28-9B as well as larger general models like Qwen2.5-32B-Instruct on En↔Zh tasks from WMT23, WMT24 and Flores200. The work also reports further gains when SSR is augmented with COMET supervision, highlights advantages over external LLM-as-judge baselines, and releases code, data and models.

Significance. If the results are robust, the contribution would be significant because it provides evidence that self-generated judgments can serve as effective rewards for RL-based MT improvement without external references or trained reward models. The public release of code, data and models is a clear strength that supports reproducibility and allows independent verification of the self-rewarding loop.

major comments (3)

§3 (SSR framework): the exact self-judgment prompt template, output format (pairwise vs. scalar), and reward normalization procedure used to produce the RL signal are not specified. This information is load-bearing for evaluating whether the self-reward mechanism is unbiased or susceptible to the reward-hacking risks noted in the skeptic analysis.
§4.3 (Ablations and analysis): no experiment freezes the judge weights while continuing policy optimization, nor reports the correlation between self-judgment scores and held-out COMET/human scores on the exact translations produced during RL training. These controls are needed to substantiate that gains arise from reliable self-rewards rather than other training dynamics.
§4.1 (Experimental setup): the manuscript provides no RL algorithm hyperparameters, number of PPO/GRPO steps, or statistical significance tests (p-values, confidence intervals) for the reported benchmark improvements over TowerInstruct-13B and Qwen2.5-32B-Instruct.

minor comments (2)

Abstract: the phrase 'complementary benefits when combined with trained RMs' is stated without a brief description of the combination procedure (e.g., reward mixing weights).
Figure 1 or method diagram: the caption and arrows should explicitly label the self-judgment step and the RL update to improve clarity for readers unfamiliar with the loop.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their constructive feedback on our manuscript. We value the positive remarks regarding the significance of our self-rewarding approach and the reproducibility benefits from releasing code, data, and models. We address each of the major comments below, indicating the revisions we will make to strengthen the paper.

read point-by-point responses

Referee: §3 (SSR framework): the exact self-judgment prompt template, output format (pairwise vs. scalar), and reward normalization procedure used to produce the RL signal are not specified. This information is load-bearing for evaluating whether the self-reward mechanism is unbiased or susceptible to the reward-hacking risks noted in the skeptic analysis.

Authors: We concur that the precise details of the self-judgment prompt, output format, and reward normalization are critical for assessing the integrity of the self-rewarding process and potential issues like reward hacking. To address this, we will revise Section 3 to include the complete self-judgment prompt template, clarify the output format (specifying if it is pairwise comparison or scalar scoring), and provide a detailed description of the reward normalization procedure. These additions will enable better evaluation of the mechanism's reliability. revision: yes
Referee: §4.3 (Ablations and analysis): no experiment freezes the judge weights while continuing policy optimization, nor reports the correlation between self-judgment scores and held-out COMET/human scores on the exact translations produced during RL training. These controls are needed to substantiate that gains arise from reliable self-rewards rather than other training dynamics.

Authors: We appreciate the suggestion for these additional analyses to more rigorously demonstrate that the performance gains are due to effective self-rewards. For the correlation, we will calculate and report the Pearson or Spearman correlation between the self-judgment scores and held-out COMET scores (and human scores where available) for the translations generated during the RL process in the updated Section 4.3. Regarding freezing the judge weights, since our framework involves joint updating of the model acting as both policy and judge, we will add an ablation study where the judge component is frozen after a certain point while continuing policy optimization. These will be included in the revised manuscript. revision: yes
Referee: §4.1 (Experimental setup): the manuscript provides no RL algorithm hyperparameters, number of PPO/GRPO steps, or statistical significance tests (p-values, confidence intervals) for the reported benchmark improvements over TowerInstruct-13B and Qwen2.5-32B-Instruct.

Authors: We agree that specifying the RL hyperparameters, training steps, and statistical tests enhances the transparency and credibility of our experimental results. In the revised version, we will expand Section 4.1 to include all relevant RL algorithm hyperparameters (such as learning rate, clip range, and batch sizes), the exact number of PPO or GRPO steps performed, and statistical significance measures including p-values and confidence intervals for the improvements over the compared models like TowerInstruct-13B and Qwen2.5-32B-Instruct. revision: yes

Circularity Check

0 steps flagged

No significant circularity; external benchmarks ground the claims

full rationale

The paper defines SSR as a reference-free RL method that uses the model's self-generated judgments as the reward signal during training on 13K monolingual examples. The headline performance claims (SSR-Zero-7B outperforming TowerInstruct-13B, GemmaX-28-9B, and Qwen2.5-32B-Instruct on WMT23/WMT24/Flores200 En↔Zh) are measured on held-out public benchmarks that are independent of the self-reward loop. No equation or result is shown to reduce by construction to a fitted parameter, self-citation, or renamed input; the self-reward mechanism is the explicit training procedure rather than a definitional tautology. The derivation therefore remains self-contained against external evaluation.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests primarily on the domain assumption that LLM self-judgments correlate with translation quality sufficiently for RL improvement; no free parameters or new invented entities are introduced in the abstract description.

axioms (1)

domain assumption LLM self-judgments on its own translations can serve as effective reward signals for reinforcement learning without reference data or external models.
This premise is required for the SSR method to function as described and is not independently validated within the abstract.

pith-pipeline@v0.9.0 · 5833 in / 1376 out tokens · 49157 ms · 2026-05-22T13:47:01.481037+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

rself = Mself(src, trans)/100 , rself ∈[0,1] ... Overall Reward rall = rself + rformat if rformat ≠ 0 else 0

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translation
cs.CL 2025-04 unverdicted novelty 3.0

A literature survey that organizes prompting, fine-tuning, preference optimization, and context-aware techniques for LLM-based machine translation with emphasis on low-resource languages.