{"id":"19320e19-2300-41b5-b59f-9014534d6b74","arxiv_id":"2608.10812","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"GRPO with a language-gated average of XCOMET and COMETKiwi rewards, plus SFT-RL checkpoint interpolation, improves reference-free and reference-based neural scores across 46 languages.","lead":"This paper post-trains open multilingual translation models using reinforcement learning that rewards translations scored by two reference-free quality estimators, then interpolates the result with the supervised model. The resulting 12B model scores higher than Google Translate, Gemini 3 Pro, and GPT-5 on the reference-free metrics used by the authors across 46 languages.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on optimizing the same XCOMET/COMETKiwi metrics that are then used as the headline evaluation; without an independent metric or human evaluation, the reported quality gains may be reward overfitting, not translation quality.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the reward used for training is also the primary evaluation metric. I agree with that diagnosis. The paper is internally consistent, releases models and code, and reports one partially independent quantity, reference-based XCOMET on FLORES+, which improves even as spBLEU declines. Those facts give some support to the claim, but they do not break the circularity. Reference-based XCOMET is a learned neural metric in the same COMET family as the reward components, so it is not a fully independent arbiter of translation quality, and spBLEU moves in the opposite direction. Since the headline comparison against proprietary systems is made entirely in reference-free XCOMET/COMETKiwi terms, the strongest claim of the paper depends on those metrics being faithful proxies for human quality. The obvious failure mode is reward hacking: GRPO can exploit systematic blind spots in the two QE models while producing outputs that are fluent but less faithful or otherwise worse by human standards. The paper itself cites reward hacking as a motivation, and the language gate only addresses wrong-language outputs, not other blind spots. A human MQM study on a modest, stratified sample would settle whether the ordering v1.0 > v0.1 and v1.0-12B > proprietary systems is real. Because the reader already returned CONDITIONAL, my stress-test does not move the verdict; it sharpens the condition that must be met before the central claim can be accepted.","tokens_in":105617,"tokens_out":4478,"duration_ms":47191,"concrete_test":"Run a blind MQM human evaluation on a stratified sample (e.g., 200 segments x 8 language directions spanning high/mid/low resource, drawn from held-out portions of WMT24++ and FLORES+) comparing MiLMMT-46-12B-v1.0 with v0.1 and with Google Translate, Gemini 3 Pro, and GPT-5. If the XCOMET/COMETKiwi ordering does not reproduce in MQM, especially for v1.0 versus v0.1, the reward/evaluation circularity is confirmed as load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reward in Section 2 is R = 1/2(s_X + s_K) when OpenLID-v3 predicts the target language, and 0 otherwise. Section 3 then evaluates with the same two 10B QE models, and the headline WMT24++ comparisons in Table 1 are exclusively reference-free XCOMET/COMETKiwi. GRPO is therefore directly maximizing the reported headline metric. The FLORES+ reference-based XCOMET column is not identical to the reward, but it is another COMET-family neural metric and may share blind spots; the only surface-level metric, spBLEU, decreases on average by 1.21 points. No human evaluation is included, and the abstract's 'improve translation quality' goes beyond 'improve the trained QE scores.' Thus the central claim that v1.0 produces better translation, and in particular that 12B-v1.0 beats Google Translate/Gemini/GPT-5, is not yet established by the evidence presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies reference-free post-training for multilingual machine translation. Starting from the supervised-finetuned MiLMMT-46-v0.1 models, it applies GRPO with a reward defined as the average of XCOMET and COMETKiwi scores, gated by OpenLID-v3 language identification, and then linearly interpolates the SFT and RL checkpoints with coefficient alpha=0.5 to obtain MiLMMT-46-v1.0. Experiments across 1B, 4B, and 12B scales and 46 languages report consistent gains on WMT24++ and FLORES+ over the SFT starting point, favorable comparisons with open and proprietary baselines, and an on-policy distillation analysis showing that distillation reaches but does not exceed the RL-plus-interpolation frontier. Models and code are released.","tokens_in":105862,"tokens_out":4588,"duration_ms":48962,"significance":"If the headline claims were independently verified, this would be a valuable contribution: it demonstrates a scalable, reference-free RL recipe across three model scales and 46 languages, with a clean checkpoint-interpolation mechanism and a careful distillation study. The paper ships released models and code, detailed per-direction results, and reproducible hyperparameters, which are concrete strengths. However, the primary evaluation uses the same two QE scorers (XCOMET and COMETKiwi) that constitute the training reward, so the reported gains are, to a substantial degree, direct optimization of the evaluation metric rather than independent evidence of translation quality. The observed average spBLEU decrease of 1.21 points and the absence of human evaluation leave the abstract's 'improve translation quality' claim unsubstantiated. The contribution is better framed as a recipe for improving the optimized reference-free QE metrics, with the quality claim requiring additional evidence.","major_comments":[{"comment":"The reward is R(x,y,ell) = 0.5(s_X + s_K) when OpenLID-v3 predicts the target language and 0 otherwise, and Section 3 evaluates with exactly the same XCOMET and COMETKiwi scorers on both benchmarks. The headline gains on WMT24++ and FLORES+ are therefore direct reward optimization, not an independent test of translation quality. The abstract's 'improve translation quality' and 'leading reference-free scores' overstate the evidence: the paper demonstrates improvements on the trained QE metrics. The only surface-level metric reported, spBLEU, decreases by 1.21 points on average (Section 4.1), and reference-based XCOMET, while not identical to the reward, is a COMET-family metric that may share the reward models' blind spots. The conclusion's own statement that 'more robust and human-aligned reward models' are future work concedes the reward may not be human-aligned. To support the quality claim, the authors should either add human evaluation or independent, non-COMET reference-based metrics, or explicitly reframe the contribution as improving the optimized reference-free scores.","section":"Section 2 (Reward design) and Section 3 (Evaluation)"},{"comment":"The interpolation coefficient alpha=0.5 is selected from the FLORES+ spBLEU/reference-based XCOMET trade-off curves, and the same FLORES+ benchmark is then used as a main evaluation set in Table 1. This creates a model-selection-on-the-evaluation-set effect: the reported FLORES+ results for MiLMMT-46-v1.0 are not independent of the alpha selection procedure. The authors should choose alpha on a separate validation set, or report the full range of alpha values and show that the main conclusions are robust to the choice. The WMT24++ results, which were not used for selection, are less affected, but the abstract and Section 4.1 draw conclusions from both benchmarks. At minimum, the paper should clearly state that alpha was tuned on FLORES+ and treat the FLORES+ numbers as partially selected rather than as a fully held-out evaluation.","section":"Section 4.2 (Figure 2) and Table 1"},{"comment":"No error bars, significance tests, or multiple decoding runs are reported. The claim that MiLMMT-46-12B-v1.0 'achieves leading reference-free scores' against proprietary systems rests on single greedy-decoding point estimates, and several comparisons in Table 1 are within a few tenths of a point on COMETKiwi or reference-based XCOMET. Without confidence intervals, bootstrap estimates, or at least multiple runs, the comparative claims are fragile. Given that the paper positions the 12B model as outperforming Google Translate, Gemini 3 Pro, and GPT-5, the authors should provide uncertainty estimates for the key comparisons and, ideally, a human evaluation to interpret the QE-score differences. This is especially important because the QE scores are also the training objective, so metric noise directly affects the validity of the quality claims.","section":"Section 3 (Evaluation) and Section 4.1 (Main Results)"}],"minor_comments":[{"comment":"The inequality '0.30< mu_x <0.95' lacks spacing and is hard to read; please format it consistently as '0.30 < mu_x < 0.95' and similarly for 'sigma_x >= 0.05'.","section":"Section 2 (RL data)"},{"comment":"The discussion of the spBLEU decrease would be strengthened by also reporting a second reference-based metric such as chrF, which is less sensitive to lexical overlap, to demonstrate that the decrease is not merely an artifact of the single-reference spBLEU measure.","section":"Section 4.1"},{"comment":"The column headers 'QE' and 'Ref.' are ambiguous when combined with the 'WMT24++' and 'FLORES+' blocks; please define the metric pairs explicitly in the caption, as done in the body text.","section":"Table 1 caption"},{"comment":"The term 'k1 estimator' in Appendix E is not defined in the main text; please provide a brief definition or citation at first use.","section":"Section 5 (On-policy distillation)"},{"comment":"The hyperparameter table lists 'KL loss type' and 'KL coefficient beta' but does not report the clipping parameter epsilon used in the GRPO objective from Section 2; please add it for completeness.","section":"Appendix C, Table 5"}],"recommendation":"major_revision","confidential_remarks":"The circularity concern raised in the reader's report is real and load-bearing: the reward in Section 2 is the same XCOMET/COMETKiwi average used as the primary evaluation in Section 3. I agree that this does not sink the paper's technical contribution—the recipe for improving those QE metrics is sound and clearly presented, with strong reproducibility features—but the abstract and conclusion must be reframed, and independent evaluation (human or non-COMET reference-based) is needed to support the 'translation quality' claim. The alpha-selection-on-FLORES+ issue is also fixable by reporting a separate validation set or a sensitivity analysis. The paper is within scope for cs.CL and the empirical scale is a strength; I recommend major revision rather than rejection because the central method is defensible once the claims are appropriately scoped."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Tom,\n\nHere's my take on arXiv:2608.10812. The paper is a clean empirical study of reference-free RL for multilingual MT. It takes the MiLMMT-46-v0.1 SFT models, applies GRPO with a reward that is the average of XCOMET and COMETKiwi scores (gated by language ID), then linearly interpolates SFT and RL checkpoints. They do this at 1B, 4B, and 12B, across 46 languages, and also compare on-policy distillation. The released models and code are real artifacts.\n\nWhat's genuinely useful: the interpolation analysis is well done. The trade-off between spBLEU and reference-based XCOMET is characterized clearly, and alpha=0.5 is a reasonable operating point. The OPD experiments are honest—distillation reaches but doesn't beat the RL+interpolation frontier. The scale of the evaluation is impressive, and the per-direction results are all there.\n\nThe problem is the reward-evaluation entanglement. The reward is exactly (XCOMET + COMETKiwi)/2, and the headline numbers are exactly those two metrics. So the fact that v1.0 improves on those metrics over v0.1 is partly the objective working as intended. That's not nothing—it shows the RL pipeline optimizes the chosen reward—but it doesn't establish better translation quality. spBLEU drops by 1.21 points on average, which the authors wave off. The comparison against Google Translate, Gemini 3 Pro, and GPT-5 is unfair in a specific sense: those systems were not trained with this QE reward, so the reference-free scores are optimized for your model and merely measured for theirs.\n\nThe choice of alpha=0.5 is also selected from the FLORES+ curves, so the final benchmark is used for model selection. That's a leak, though a modest one. There are no human evaluations and no error bars. The central claim in the abstract—'improve translation quality'—outruns the evidence.\n\nI'd still send this to serious review. The artifacts are valuable, the interpolation trick is likely to be reused, and the OPD results are a useful data point. But the authors should be pushed to add human evaluation or at least an independent reference-based metric outside the COMET family, and to justify alpha without peeking at the final test sets. If they add that, the quality claim becomes credible. Without it, the paper remains a solid engineering report with a circular evaluation.\n\nWho needs to read it: anyone doing RL-based MT post-training, and anyone building QE-guided methods; it's a good case study in why the metric you optimize must not be the only metric you report.\n\nRecommendation: accept for peer review, with major revision focused on independent evaluation.\n\nBest,\n[Your name]","headline":"Reward-evaluation entanglement keeps the headline quality claim unproven, but the released models, interpolation analysis, and honest OPD comparison make this worth reviewing.","tokens_in":106379,"tokens_out":2836,"would_cite":true,"duration_ms":31609,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reference-free post-training with a language-gated quality reward plus SFT–RL checkpoint interpolation consistently improves open multilingual translation models across 1B, 4B, and 12B scales and 46 languages.","keywords":["multilingual machine translation","reference-free quality estimation","reinforcement learning","GRPO","checkpoint interpolation","language identification gate","on-policy distillation","open large language models"],"falsifier":"A human-annotation study on a stratified sample of FLORES+ and WMT24++ outputs comparing MiLMMT-46-12B-v1.0, MiLMMT-46-12B-v0.1, Google Translate, and Gemini would settle the central claim: if human raters do not prefer v1.0 over v0.1 more often than chance, or rank the proprietary systems above it despite lower reference-free scores, the claimed quality gains fail.","tokens_in":105439,"feed_emoji":"🌐","tokens_out":6779,"duration_ms":57808,"temperature":0.7,"pith_summary":"This paper claims that a reference-free post-training recipe can push open multilingual translation models beyond their supervised fine-tuning starting point without needing parallel reference sentences. The recipe applies GRPO with a reward that is the average of two reference-free quality estimators (XCOMET and COMETKiwi), gated by a language-identification check so that fluent but wrong-language outputs score zero. Then the SFT and RL checkpoints are linearly interpolated at weight 0.5 to recover most of the lexical accuracy lost during RL while keeping most of the neural-quality gain. Across 46 languages and model scales of 1B, 4B, and 12B parameters, the resulting MiLMMT-46-v1.0 models beat their v0.1 SFT counterparts on learned quality metrics and, at 12B, post the best reference-free scores among the open and proprietary systems tested.","feed_headline":"Open 12B model tops proprietary systems on reference-free scores","feed_subtitle":"RL plus checkpoint interpolation lifts translation quality across 46 languages and all three scales.","key_machinery":"Group Relative Policy Optimization (GRPO) with a gated reward R = ½(s_X + s_K) if the predicted language of the candidate equals the intended target language, and 0 otherwise, combined with linear interpolation θ_α = αθ_SFT + (1−α)θ_RL between the supervised and RL checkpoints. The reward channels policy optimization toward translations that two learned reference-free quality estimators agree on, while the language gate blocks wrong-language outputs; the interpolation controls drift from the SFT behavior.","core_discovery":"Reference-free RL from a supervised-finetuned multilingual model, with a reward formed by averaging XCOMET and COMETKiwi and zeroed when OpenLID-v3 predicts a language other than the intended target, consistently improves translation quality as measured by those same models, while SFT–RL checkpoint interpolation at α=0.5 turns the RL gains into a better operating point than either endpoint. The paper's central empirical discovery is that this recipe transfers across three model scales and 46 languages, that the 12B version reaches the top reference-free scores against Google Translate, Gemini models, GPT-5, NLLB, and recent open translation systems on shared language subsets, and that on-policy distillation from the 12B teacher can approach but not surpass the RL+interpolation frontier.","pith_inferences":["A testable extension of the paper's recipe is to evaluate MiLMMT-46-v1.0 against its SFT predecessor with human annotations on a held-out set of sentences across low-resource directions; that would show whether the reference-free score gains correspond to human-judged quality rather than the preferences of the two QE models.","The choice α=0.5 was read off FLORES+ curves; an obvious follow-up is to sweep α on a different benchmark or language group to see whether a single interpolation coefficient generalizes, or whether the optimal operating point moves with language resource level.","The RL data filter keeps only instances with mid-range mean reward and non-negligible group variance, which likely biases post-training toward directions where the QE models disagree; one implication is that very-high- and very-low-quality directions may be under-touched by RL, and a test would be to measure per-direction gains against the filter's per-direction retention rate.","On-policy distillation's near-parity with RL+interpolation suggests that the teacher's quality preferences can be transferred without a QE reward at the student scale, which would be useful for language pairs where QE models are unavailable."],"forward_implications":["Reference-free post-training works when the reward combines two QE models and a language gate; the gate is sufficient to suppress wrong-language reward hacking.","Interpolation at α=0.5 recovers several spBLEU points lost during RL while sacrificing less than one point of reference-based XCOMET, giving a cheap, no-training way to tune the quality trade-off.","A 1B-parameter v1.0 model outperforms TranslateGemma-4B on every reported WMT24++ and FLORES+ metric, showing the recipe lets a small model compete with a four-times-larger open system.","On-policy distillation transfers most of the 12B teacher's gains to 1B and 4B students but does not surpass the RL+interpolation frontier, making it an alternative rather than an improvement.","The same RL recipe applies unchanged at 1B, 4B, and 12B scales, so the reported gains are not an artifact of one particular model size."],"supporting_citations":[{"why":"Supplies the supervised-finetuned MiLMMT-46-v0.1 starting checkpoints and SFT data from which the RL stage begins.","marker":"Shang et al., 2026"},{"why":"Provides GRPO, the policy-optimization algorithm that carries the RL stage.","marker":"Shao et al., 2024"},{"why":"Defines XCOMET, one of the two reward components and evaluation metrics.","marker":"Guerreiro et al., 2024"},{"why":"Defines COMETKiwi, the second reward component and evaluation metric.","marker":"Rei et al., 2023"},{"why":"Provides OpenLID-v3, the language-identification gate that zeros the reward for wrong-language outputs.","marker":"Fedorova et al., 2026"},{"why":"Documents reward hacking in multilingual translation RL and motivates the gate and interpolation design.","marker":"Liu et al., 2026"},{"why":"Tower+ is a GRPO-trained multilingual translation baseline and a precedent for carefully designed RL rewards.","marker":"Rei et al., 2025"},{"why":"HY-MT2 is one of the strongest open baseline systems that MiLMMT-46-v1.0 is compared against.","marker":"Zheng et al., 2026"},{"why":"Defines the WMT24++ benchmark, one of the two evaluation datasets.","marker":"Deutsch et al., 2025"},{"why":"Defines the FLORES+ benchmark, the other evaluation dataset and source of spBLEU and reference-based XCOMET metrics.","marker":"NLLB Team et al., 2024"}],"fun_headline_variants":["Reference-free RL lifts open MT above proprietary","Open 12B MT beats Google, Gemini, GPT-5 without refs","RL post-training gains edge with no quality references","46-language model tops commercial MT via ref-free reward"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the average of XCOMET and COMETKiwi is a valid proxy for translation quality, because the same scores are both the training reward and the headline evaluation metric.","fun_headline_variants_meta":{"raw":{"variants":["Reference-free RL lifts open MT above proprietary","Open 12B MT beats Google, Gemini, GPT-5 without refs","RL post-training gains edge with no quality references","46-language model tops commercial MT via ref-free reward"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000114,"raw_usage":{"total_tokens":1030,"prompt_tokens":871,"completion_tokens":159,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":93}},"tokens_in":487,"tokens_out":159,"duration_ms":2536,"temperature":1.0,"reasoning_tokens":93,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:49:35.293289+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A human-annotation study on a stratified sample of FLORES+ and WMT24++ outputs comparing MiLMMT-46-12B-v1.0, MiLMMT-46-12B-v0.1, Google Translate, and Gemini would settle the central claim: if human raters do not prefer v1.0 over v0.1 more often than chance, or rank the proprietary systems above it despite lower reference-free scores, the claimed quality gains fail.","supporting_citations":[],"review_version":1}