Pith. sign in

REVIEW 4 major objections 4 minor 64 references

RIVAL: Reinforcement Learning with Iterative and Adversarial Optimization for Machine Translation

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper establishes iterative RM-LLM adversarial training as a viable alternative to SFT for LLM translation, with gains in WMT BLEU and smaller out-of-distribution degradation.

desk verdict Solid recipe for iterative adversarial reward modeling in MT, but the OOD-preservation claim is overstated and Equation 8 has a typo; deserves peer review. read the letter →

arxiv 2506.05070 v2 pith:URJVTQFQ submitted 2025-06-05 cs.CL

Tianjiao Li , Mengran Yu , Chenyu Shi , Yanjun Zhao , Xiaojing Liu , Qiang Zhang , Qi Zhang , Xuanjing Huang
show 1 more author
Jiayin Wang
This is my paper
classification cs.CL
keywords machinetranslationRLHFrewardhackingdistributionalshiftadversarialtrainingiterativeoptimizationsubtitleGRPO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reinforcement learning from feedback is meant to sharpen translation quality, but the paper reports that ordinary offline RLHF fails on colloquial subtitle translation: the fixed reward model drifts out of sync with the translator as it trains, and the model starts adding content that is not in the source text to please the reward model. RIVAL instead runs reward-model training and translator training as an iterative min-max game, with the reward model repeatedly retrained to widen the gap between strong and weak translations and the translator trained to close that gap, guided by both qualitative preference and a BLEU-based quantitative reward predicted by the same reward model. On WMT Chinese-English test sets, two RIVAL iterations with both rewards reach BLEU 39.39/33.42 and COMETkiwi 72.60/73.61, above the SFT baseline of 38.77/32.22 and 71.39/71.88, while out-of-distribution English-German quality degrades far less than under SFT. The paper's central claim is that iteratively updated adversarial reward modeling is a viable alternative to supervised fine-tuning for LLM-based machine translation.

What carries the argument

The load-bearing object is the RIVAL min-max objective $$\min_{r_\phi}\max_{\pi_\$\theta$} \mathbb{E}[r_\$\varphi$(x,\pi_\$\theta$(y|x))] - \mathbb{E}_{y\sim P_{\text{strong}}}[r_\$\varphi$(x,y)]$$ subject to a KL constraint, which frames the RM as a discriminator that keeps chasing the current translator while the LLM acts as a generator that keeps closing the quality gap. Two concrete mechanisms carry the result: the RM step retrains $r_\phi$ with a pair-wise rank loss (the qualitative preference reward) on strong-versus-current-weak translation pairs, and the LLM step uses GRPO to maximize the RM reward; the quantitative preference reward is a second output head on the same RM that regresses BLEU scores through a noise-robust MAE loss, injecting lexical fidelity without making the reward a brittle reference-dependent metric.

What would settle it

Run a blind side-by-side human evaluation on held-out subtitle and WMT samples, comparing RIVAL-Iter2-Qual+Quant, SFT, and GPT-4o; if human raters do not prefer RIVAL over SFT in the same direction GPT-4o's scores do, the reward proxy is not faithfully tracking human translation quality.

Watch

Extended reading notes

Core claim

The paper claims that the failure of vanilla RLHF in translation is caused by distributional shift: as the LLM improves during RL training, its output distribution moves away from the weak translations the offline reward model was trained on, so reward scores stop tracking true quality and reward hacking appears. The proposed fix is RIVAL, which rewrites the RLHF pipeline as a min-max game between the reward model and the LLM: the RM is trained to maximize the score gap between outputs of a strong translator, defined as gold references plus GPT-4o outputs, and the current LLM, while the LLM is trained to minimize that gap, with the RM rebuilt each iteration from current LLM outputs plus replayed earlier outputs. To keep the qualitative preference signal from drifting during open-ended exploration, the RM also predicts a quantitative preference reward that approximates BLEU, trained on noisy data with a multi-head loss $L_{\text{qualitative}} + \alpha L_{\text{quantitative}}$. With this setup, two iterations on WMT EN-ZH reach BLEU 39.39 and 33.42 versus 38.77 and 32.22 for SFT, and the out-of-distribution EN-DE results degrade far less than SFT, which the paper reads as evidence that RIVAL learns more general translation strategies rather than memorized patterns.

Load-bearing premise

The entire pipeline assumes that GPT-4o generates genuinely strong translations and that its scores faithfully track human quality on the subtitle task, since GPT-4o both writes the strong translations the reward model learns from and evaluates the final outputs.

Editorial extensions

If this is right

  • If RIVAL is right, iterative adversarial RM updating is a working alternative to SFT for LLM translation, and the gains are not bought by sacrificing BLEU: quantitative and qualitative rewards together raise both lexical and semantic metrics.
  • The method preserves out-of-distribution language performance far better than SFT: on WMT EN-DE and DE-EN the RIVAL models sit close to the original Qwen2.5-7B-Chat, whereas SFT drops sharply.
  • Because the reward model is rebuilt each round from current LLM outputs, the framework should keep providing reliable supervision as the model improves, avoiding the reward-hacking failure mode documented in the pilot experiment.
  • The paper's own results suggest further iterations may push the model past strong translators; it explicitly lists surpassing the strong translator as a plausible continuation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's claims, the framework implies that reward hacking in translation is primarily a distribution-shift problem rather than a pure signal-design problem; if so, periodically re-fitting the RM is more important than the exact preference loss, a hypothesis testable by ablating RIVAL against offline RM refreshment without adversarial framing.
  • Because the subtitle references are GPT-4o-generated and GPT-4o also scores the outputs, the reported quality gains may partly reflect imitation of that model's style; a human side-by-side evaluation on a fresh subtitle sample would separate real quality gain from proxy alignment.
  • A direct extension would be to replace the BLEU head with a learned reference-free quality estimator as the quantitative reward; if the multi-head stabilization effect persists, the framework's success does not depend on BLEU specifically.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes RIVAL, an iterative adversarial RL framework for LLM-based machine translation. The reward model and the translation LLM are alternately updated in a min-max game: the RM learns to rank strong translations above the current policy's outputs (qualitative preference reward), while an additional head predicts BLEU-like quantitative scores to stabilize training. Experiments on a new colloquial Chinese-English subtitle dataset and on WMT ZH-EN (with EN-DE and Medical ZH-DE as OOD tasks) report that RIVAL-Iter2-Qual+Quant outperforms SFT on BLEU and COMETkiwi in-domain while degrading less than SFT on OOD tasks. The paper also releases the processed subtitle dataset. The central in-domain WMT result is plausible if the reported training objective is corrected and the OOD claim is restated precisely.

Significance. If the corrected training objective reproduces the reported numbers, RIVAL is a useful contribution: it offers a concrete alternative to SFT for LLM-based MT, with an adversarial reward-model update that addresses distributional shift, and it provides an open subtitle-translation resource for a challenging colloquial domain. The paper's strongest evidence is the in-domain WMT comparison in Table 2, where RIVAL-Iter2-Qual+Quant reaches BLEU 39.39/33.42 and COMETkiwi 72.60/73.61 versus SFT's 38.77/32.22 and 71.39/71.88 on EN-ZH/ZH-EN. The MAE-vs-MSE analysis (Table 3) is a concrete, useful ablation. However, the paper does not ship code or machine-checked proofs, and the central claims rest on an equation that is printed in a way that cannot train the quantitative head and on a contribution statement that is contradicted by the paper's own OOD numbers. These issues are correctable but load-bearing.

major comments (4)
  1. [§4.2, Eq. (8) and Algorithm 1] Equation (8) defines the quantitative term as E_DRM[|ys_BLEU - yw_BLEU|]. This expression does not involve the reward model parameters rϕ, so its gradient with respect to the RM is zero and the quantitative head cannot be trained by this loss as printed. Algorithm 1, line 7, stores (x, BLEU(ys_i, ys_i), BLEU(ys_i, yw_i)), which suggests the intended loss is an MAE between predicted scores and these BLEU targets, e.g., |rϕ(x,ys)-BLEU(ys,ys)| + |rϕ(x,yw)-BLEU(ys,yw)|. Please correct the equation and confirm that the Qual+Quant results in Tables 2 were obtained with a loss that actually updates the quantitative head; if they were, the printed formula is a typo, but it is load-bearing and must be fixed.
  2. [§1, Contribution 2 and Table 2 (OOD rows)] The claim that RIVAL improves in-domain translation quality 'without compromising the out-of-distribution performance of the model' is not supported by Table 2. Relative to the base Qwen2.5-7B-Chat, RIVAL-Iter2-Qual+Quant decreases OOD WMT EN-DE COMETkiwi from 71.53 to 68.74 and BLEU from 27.23 to 25.25, and DE-EN COMETkiwi from 74.18 to 72.23 and BLEU from 36.85 to 35.25. The paper's own discussion later narrows the claim to 'significantly less degradation' than SFT, which is a different statement. Please revise Contribution 2 and the relevant text to state the comparison explicitly, and if the degradation relative to the base model is considered acceptable, justify that standard with evidence.
  3. [§3.1, §5.1, Appendix C] The subtitle evaluation and the construction of training data are both anchored to GPT-4o: GPT-4o produces the 'strong translations' used as RM targets, and GPT-4o is used as the primary judge. The only human check (Appendix C, Figure 5) reports three annotators but provides no numeric agreement, correlation, inter-annotator reliability, or error bars. This makes it difficult to rule out that the subtitle gains reflect the RM and the judge favoring GPT-4o-like outputs rather than human-perceived quality. Please report quantitative human-model agreement (e.g., Pearson/Spearman correlation, quadratic-weighted kappa, per-dimension agreement) and, if possible, add a reference-based metric or human evaluation on a subset to break the circularity.
  4. [§4, Eqs. (4) and (7), Appendix D] The formal objectives in Equations (4) and (7) impose the constraint KL[πθ∥πref] < η, but Appendix D states that the KL penalty coefficient β is set to 0 and the constraint is 'effectively removed.' If no KL regularization is used, the distributional-shift story motivating RIVAL is tested without the safeguard present in the formal method. Please clarify whether the constraint is active, and if β=0 is intentional, justify this choice and report the actual divergence between πθ and πref over training iterations.
minor comments (4)
  1. [Tables 1 and 2] The captions say all results are averaged over 3 runs, but no standard deviations, confidence intervals, or significance tests are reported. Some differences are small (e.g., +0.62 BLEU for EN-ZH in Table 2), so variance information is needed to assess whether the gains are reliable.
  2. [Appendix C, Figure 5] The figure alone does not support the statement that GPT-4o assessments show 'a high degree of consistency' with human judgments. Please add numeric correlation coefficients, per-dimension agreement, inter-annotator agreement, and the number of annotated items.
  3. [Section 5.2 and Table 1] The claim that 'more iterations lead to better performance' is not consistently supported by Table 1: RIVAL-Iter2-Qual has lower GPT-4o average (3.53) than RIVAL-Iter1-Qual (3.68), and COMETkiwi only improves from 66.27 to 66.49. Please restrict the claim to the Qual+Quant setting or qualify it with the observed variance.
  4. [Related Work and Appendix D] There are small presentation errors: 'accross' appears in Related Work, and the text refers to the Verl framework while citing Sheng et al. (2024), whose system is named HybridFlow. These should be harmonized.

Circularity Check

1 steps flagged · score 4.0 of 10

Subtitle pipeline is a GPT-4o supervision-and-evaluation loop; WMT claims rest on external references.

  1. self definitional [Section 3.1 (Dataset Construction, step 3) and Section 5.1 (Evaluation Metrics)]
    "We utilize GPT-4o to produce strong translations that served as targets and employ Qwen2.5-7B-Chat as our baseline model to generate weak translations, representing the initial performance that we aim to improve. [...] On the subtitle dataset, we use COMETKiwi and GPT-4o for evaluation because (1) subtitle tasks prioritize semantic content, making BLEU less appropriate; and (2) the references are GPT-4o-generated, making it infeasible to compute BLEU scores against genuine reference translations."

    The subtitle 'strong translations' are GPT-4o outputs, and the RM is trained by Eq. 6 to assign them higher reward than weak outputs. The LLM is then trained to maximize this RM reward (Eq. 7). The reported subtitle quality is measured by GPT-4o scoring against the same GPT-4o-generated references. So the outcome variable on the subtitle task is essentially closeness to GPT-4o, the same source that supplied the training target; the subtitle gains are not an independently defined measure of translation quality. The Appendix C human check and COMETkiwi provide partial external support, but the primary subtitle metric is entangled with the training signal.

full rationale

No self-citations of the authors appear; the method is a standard adversarial RL loop (GAN-inspired min-max with GRPO). The central WMT in-domain claim (Table 2) is independently supported because BLEU and COMETkiwi are computed against external WMT references, not against GPT-4o outputs; so the main contribution does not reduce to its inputs. The OOD-preservation claim is an internal empirical contradiction with Table 2 (COMETkiwi drops 2.79 EN->DE and 1.95 DE->EN), but that is a correctness risk, not circularity. The circular component is confined to the subtitle evaluation: GPT-4o both generates the strong translations that train the RM and scores the final outputs, making the subtitle quality metric partly self-definitional. Since WMT results are independent and the subtitle conclusions are partially supported by COMETkiwi and a small human comparison, the score is 4 rather than higher.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the definition of 'strong translations' and on the reward signals; those are inputs from GPT-4o and BLEU, not derived within the paper. The free parameters alpha, beta, and tau are chosen by hand or left unspecified. No new physical or conceptual entities are introduced.

free parameters (3)
  • alpha (quantitative loss weight) = 1
    Chosen by hand ('we do not assign it a specific value and thus set it to 1'); balances rank loss and BLEU MAE loss, with no tuning or sensitivity analysis reported.
  • beta (KL penalty coefficient) = 0
    Set to 0 in Appendix D, effectively removing the KL constraint stated in Eq. 4; this alters the optimization problem from the one formally defined.
  • tau (similarity threshold) = not given
    Algorithm 1 filters training pairs where sim(ys,yw) < tau; the threshold is never specified, yet it determines which samples the RM is trained on.
assumptions (6)
  • domain assumption GPT-4o outputs define the 'strong translator' distribution P_strong used as training target.
    Section 3.1 step 3: GPT-4o translations are treated as targets for the RM and SFT; if this 'strong' quality is mislabeled, the whole preference signal is biased.
  • domain assumption GPT-4o scoring is a reliable proxy for human evaluation of subtitle translations.
    Section 5.1 and Appendix C; only three annotators, no numeric agreement given, yet GPT-4o is used as the primary evaluator.
  • standard math The Bradley-Terry model is appropriate for modeling translation preferences.
    Equation 2 uses the Bradley-Terry rank loss to train the RM; standard in RLHF.
  • standard math GRPO with clipping is a valid policy optimization for this setting.
    Equation 3; standard RLHF practice, but beta=0 in implementation removes the KL constraint in Eq. 4.
  • ad hoc to paper Training the RM to approximate BLEU with an MAE head improves stability and generalization.
    Section 4.2 motivates this via noisy-training and Tikhonov regularization analogies, but the exact mechanism and the Equation 8 loss as written are not sound.
  • ad hoc to paper The min-max game between RM and LLM converges to a useful fixed point without additional constraints.
    No convergence analysis; empirical stability is claimed from Figure 4 and two iterations of experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RIVAL: Reinforcement Learning with Iterative and Adversarial Optimization for Machine Translation." pith.science (2026). https://pith.science/paper/URJVTQFQ

@misc{pith2026250605070,
  author       = {Pith},
  title        = {Pith review of: RIVAL: Reinforcement Learning with Iterative and Adversarial Optimization for Machine Translation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/URJVTQFQ}},
  note         = {Machine review of arXiv:2506.05070}
}
read the original abstract

Large language models (LLMs) possess strong multilingual capabilities, and combining Reinforcement Learning from Human Feedback (RLHF) with translation tasks has shown great potential. However, we observe that this paradigm performs unexpectedly poorly when applied to colloquial subtitle translation tasks. In this work, we investigate this issue and find that the offline reward model (RM) gradually diverges from the online LLM due to distributional shift, ultimately leading to undesirable training outcomes. To address this, we propose RIVAL, an adversarial training framework that formulates the process as a min-max game between the RM and the LLM. RIVAL iteratively updates the both models, with the RM trained to distinguish strong from weak translations (qualitative preference reward), and the LLM trained to enhance its translation for closing this gap. To stabilize training and improve generalizability, we also incorporate quantitative preference reward (e.g., BLEU) into the RM, enabling reference-free quality modeling aligned with human evaluation. Through extensive experiments, we demonstrate that the proposed adversarial training framework significantly improves upon translation baselines.

Figures

Figures reproduced from arXiv: 2506.05070 by the authors.

Figure 1
Figure 1. The illustration highlights the issue aris [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Scoring Differential of the RM and GPT￾4o between strong translations and the translations generated by the optimized model. To investigate the causes of the aforemen￾tioned phenomena, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The RIVAL framework. In the RM updating step, the RM is trained to maximize the score gap [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Scoring Differential of the RM and GPT-4o between strong and weak translations. (a) for [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Human Evaluation vs GPT-4o Evaluation To assess the reliability of GPT-4o in evalu￾ating translation quality, we conduct a compar￾ative analysis between human and model-based judgments on the subtitle dataset. Specifically, we invite three human annotators—none of whom…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

64 extracted references · 18 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    Ahmed M Ahmed, Rafael Rafailov, Stepan Sharkov, Xuechen Li, and Sanmi Koyejo. 2024. Scalable ensembling for mitigating reward overoptimisation. arXiv preprint arXiv:2406.01013

    arXiv 2024

  5. [5]

    Duarte M Alves, Jos \'e Pombal, Nuno M Guerreiro, Pedro H Martins, Jo \ a o Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, et al. 2024. Tower: An open multilingual large language model for translation-related tasks. arXiv preprint arXiv:2402.17733

    arXiv 2024

  6. [6]

    Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. https://arxiv.org/pdf/1606.06565.pdf Concrete problems in ai safety . arXiv preprint arXiv:1606.06565

    arXiv 2016

  7. [7]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  8. [8]

    Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch

    Trapit Bansal, Jakub W. Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. 2017. https://api.semanticscholar.org/CorpusID:3921978 Emergent complexity via multi-agent competition . ArXiv, abs/1710.03748

    arXiv 2017

Show all 64 references
  1. [9]

    Yamini Bansal, Behrooz Ghorbani, Ankush Garg, Biao Zhang, Colin Cherry, Behnam Neyshabur, and Orhan Firat. 2022. Data scaling laws in nmt: The effect of noise and architecture. In International Conference on Machine Learning, pages 1466--1482. PMLR

    2022

  2. [10]

    Marco Benedetti and Enrico Ventura. 2024. Training neural networks with structured noise improves classification and generalization. Journal of Physics A: Mathematical and Theoretical, 57(41):415001

    2024

  3. [11]

    Chris M Bishop. 1995. Training with noise is equivalent to tikhonov regularization. Neural computation, 7(1):108--116

    1995

  4. [12]

    Ralph Allan Bradley and Milton E. Terry. 1952. http://www.jstor.org/stable/2334029 Rank analysis of incomplete block designs: I. the method of paired comparisons . Biometrika, 39(3/4):324--345

    1952

  5. [13]

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017 a . Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30

    2017

  6. [14]

    Christiano, Jan Leike, Tom B

    Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017 b . Deep reinforcement learning from human preferences. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, page 4302–4310, Red Hoo...

    2017

  7. [15]

    Pau de Jorge Aranda, Adel Bibi, Riccardo Volpi, Amartya Sanyal, Philip Torr, Gr \'e gory Rogez, and Puneet Dokania. 2022. Make some noise: Reliable and efficient single-step adversarial training. Advances in Neural Information Processing Systems, 35:12881--12893

    2022

  8. [16]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, and Ruoyu Zhang. 2025. http://arxiv.org/abs/2501.12948 Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    2025 arXiv

  9. [17]

    Carson Denison, Monte MacDiarmid, Fazl Barez, David Duvenaud, Shauna Kravec, Samuel Marks, Nicholas Schiefer, Ryan Soklaski, Alex Tamkin, Jared Kaplan, et al. 2024. Sycophancy to subterfuge: Investigating reward-tampering in large language models. arXiv preprint arXiv:2406.10162

    2024 arXiv

  10. [18]

    Rati Devidze, Parameswaran Kamalaruban, and Adish Singla. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/266c0f191b04cbbbe529016d0edc847e-Paper-Conference.pdf Exploration-guided reward shaping for reinforcement learning under sparse rewards . In Advances in N...

    2022

  11. [19]

    Sharkey, Jacob Pfau, and David Krueger

    Lauro Langosco di Langosco, Jack Koch, Lee D. Sharkey, Jacob Pfau, and David Krueger. 2021. https://api.semanticscholar.org/CorpusID:249954130 Goal misgeneralization in deep reinforcement learning . In International Conference on Machine Learning

    2021

  12. [20]

    Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Marc ' Aurelio Ranzato. 2018. https://doi.org/10.18653/v1/N18-1033 Classical structured prediction losses for sequence to sequence learning . In Proceedings of the 2018 Conference of the North A merican Chapter of the ...

    2018 doi

  13. [21]

    Tom Everitt and Marcus Hutter. 2019. https://api.semanticscholar.org/CorpusID:199552156 Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective . ArXiv, abs/1908.04734

    2019 arXiv

  14. [22]

    Tom Everitt, Victoria Krakovna, Laurent Orseau, and Shane Legg. 2017. https://api.semanticscholar.org/CorpusID:3075935 Reinforcement learning with a corrupted reward channel . In International Joint Conference on Artificial Intelligence

    2017

  15. [23]

    Zhaopeng Feng, Shaosheng Cao, Jiahan Ren, Jiayuan Su, Ruizhe Chen, Yan Zhang, Zhe Xu, Yao Hu, Jian Wu, and Zuozhu Liu. 2025. http://arxiv.org/abs/2504.10160 Mt-r1-zero: Advancing llm-based machine translation via r1-zero-like reinforcement learning

    2025 arXiv

  16. [24]

    Zhaopeng Feng, Ruizhe Chen, Yan Zhang, Zijie Meng, and Zuozhu Liu. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.860 Ladder: A model-agnostic framework boosting LLM -based machine translation to the next level . In Proceedings of the 2024 Conference on Empirical Methods in...

    2024 doi

  17. [25]

    Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. 2019. Adversarial policies: Attacking deep reinforcement learning. arXiv preprint arXiv:1905.10615

    2019 arXiv

  18. [26]

    Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Advances in neural information processing systems, 27

    2014

  19. [27]

    Prasoon Goyal, Scott Niekum, and Raymond J. Mooney. 2019. https://doi.org/10.24963/ijcai.2019/331 Using natural language for reward shaping in reinforcement learning . In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19 , pag...

    2019 doi

  20. [28]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  21. [29]

    Minggui He, Yilun Liu, Shimin Tao, Yuanchang Luo, Hongyong Zeng, Chang Su, Li Zhang, Hongxia Ma, Daimeng Wei, Weibin Meng, Hao Yang, Boxing Chen, and Osamu Yoshie. 2025. http://arxiv.org/abs/2502.19735 R1-t1: Fully incentivizing translation capability in llms via reasoning learning

    2025 arXiv

  22. [30]

    Samuel Kiegeland and Julia Kreutzer. 2021. https://doi.org/10.18653/v1/2021.naacl-main.133 Revisiting the weaknesses of reinforcement learning for neural machine translation . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computatio...

    2021 doi

  23. [31]

    Julia Kreutzer, Shahram Khadivi, Evgeny Matusov, and Stefan Riezler. 2018. https://doi.org/10.18653/v1/N18-3012 Can neural machine translation be improved with user feedback? In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computatio...

    2018 doi

  24. [32]

    Yebin Lee, Imseong Park, and Myungjoo Kang. 2024. Fleur: An explainable reference-free evaluation metric for image captioning using a large multimodal model. arXiv preprint arXiv:2406.06004

    2024 arXiv

  25. [33]

    Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulieu, Peter J Bentley, Samuel Bernard, Guillaume Beslon, David M Bryson, et al. 2020. The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation a...

    2020

  26. [34]

    Wang Luo, Haoran Li, Zicheng Zhang, Congying Han, Jiayu Lv, and Tiande Guo. 2024. Sambo-rl: Shifts-aware model-based offline reinforcement learning. arXiv preprint arXiv:2408.12830

    2024

  27. [35]

    Tasnim Mohiuddin, Philipp Koehn, Vishrav Chaudhary, James Cross, Shruti Bhosale, and Shafiq Joty. 2022. Data selection curriculum for neural machine translation. arXiv preprint arXiv:2203.13867

    2022 arXiv

  28. [36]

    Khanh Nguyen, Hal Daum \'e III, and Jordan Boyd-Graber. 2017. https://doi.org/10.18653/v1/D17-1153 Reinforcement learning for bandit neural machine translation with simulated human feedback . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Proces...

    2017 doi

  29. [37]

    OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, and Alex Beutel. 2024. http://arxiv.org/abs/2412.16720 Openai o1 system card

    2024 arXiv

  30. [38]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022

  31. [39]

    Alexander Pan, Kush Bhatia, and Jacob Steinhardt. 2022. https://api.semanticscholar.org/CorpusID:245837268 The effects of reward misspecification: Mapping and mitigating misaligned models . ArXiv, abs/2201.03544

    2022 arXiv

  32. [40]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

    2002

  33. [41]

    Romain Paulus, Caiming Xiong, and Richard Socher. 2017. https://api.semanticscholar.org/CorpusID:21850704 A deep reinforced model for abstractive summarization . ArXiv, abs/1705.04304

    2017 arXiv

  34. [42]

    Miguel Moura Ramos, Tomás Almeida, Daniel Vareta, Filipe Azevedo, Sweta Agrawal, Patrick Fernandes, and André F. T. Martins. 2025. http://arxiv.org/abs/2411.05986 Fine-grained reward optimization for machine translation using error severity mappings

    2025

  35. [44]

    Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. http://arxiv.org/abs/1511.06732 Sequence level training with recurrent neural networks

    2016 arXiv

  36. [45]

    Ricardo Rei, Marcos Treviso, Nuno M Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, Jos \'e GC De Souza, Taisiya Glushkova, Duarte M Alves, Alon Lavie, et al. 2022. Cometkiwi: Ist-unbabel 2022 submission for the quality estimation shared task. arXiv preprint arXiv...

    2022 arXiv

  37. [46]

    Manon Revel, Matteo Cargnelutti, Tyna Eloundou, and Greg Leppert. 2025. Seal: Systematic error analysis for value alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 27599--27607

    2025

  38. [47]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

    2017 arXiv

  39. [48]

    Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text generation. arXiv preprint arXiv:2004.04696

    2020 arXiv

  40. [49]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    2024 arXiv

  41. [50]

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2024. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256

    2024 arXiv

  42. [51]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053

    2019 arXiv

  43. [52]

    Artem Sokolov, Julia Kreutzer, Stefan Riezler, and Christopher Lo. 2016. https://proceedings.neurips.cc/paper_files/paper/2016/file/795c7a7a5ec6b460ec00c5841019b9e9-Paper.pdf Stochastic structured prediction under bandit feedback . In Advances in Neural Information Processing ...

    2016

  44. [53]

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008--3021

    2020

  45. [54]

    Shaomu Tan and Christof Monz. 2025. http://arxiv.org/abs/2504.13630 Remedy: Learning machine translation evaluation from human preferences with reward modeling

    2025 arXiv

  46. [55]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  47. [56]

    Jonathan Uesato, Ramana Kumar, Victoria Krakovna, Tom Everitt, Richard Ngo, and Shane Legg. 2020. Avoiding tampering incentives in deep rl via decoupled approval. arXiv preprint arXiv:2011.08827

    2020 arXiv

  48. [57]

    Chaojun Wang and Rico Sennrich. 2020. https://doi.org/10.18653/v1/2020.acl-main.326 On exposure bias, hallucination and domain shift in neural machine translation . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3544--3552, On...

    2020 doi

  49. [58]

    Lilian Weng. 2024. https://lilianweng.github.io/posts/2024-11-28-reward-hacking/ Reward hacking in reinforcement learning. lilianweng.github.io

    2024

  50. [59]

    Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. 2022. Large language models are better reasoners with self-verification. arXiv preprint arXiv:2212.09561

    2022 arXiv

  51. [60]

    Sam Wiseman and Alexander M. Rush. 2016. https://doi.org/10.18653/v1/D16-1137 Sequence-to-sequence learning as beam-search optimization . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1296--1306, Austin, Texas. Association for...

    2016 doi

  52. [61]

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. 2022. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasin...

    2022

  53. [62]

    Lijun Wu, Fei Tian, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. 2018. https://doi.org/10.18653/v1/D18-1397 A study of reinforcement learning for neural machine translation . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3612--362...

    2018 doi

  54. [63]

    Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2023. A paradigm shift in machine translation: Boosting translation performance of large language models. arXiv preprint arXiv:2309.11674

    2023 arXiv

  55. [64]

    Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2024 a . https://openreview.net/forum?id=farT6XXntP A paradigm shift in machine translation: Boosting translation performance of large language models . In The Twelfth International Conference on Learning Representations

    2024

  56. [65]

    Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024 b . Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. arXiv preprint arXiv:2401.08417

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.