Calibrating Translation Decoding with Quality Estimation on LLMs

Christof Monz; Di Wu; Yibin Lei

REVIEW 3 major objections 3 minor 34 references

Calibrating Translation Decoding with Quality Estimation on LLMs

T0 review · 3 major / 3 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Optimizing the Pearson correlation between translation likelihood and external quality, on about 2,000 sentences per direction, substantially improves LLM translation and makes its likelihood a strong reference-free quality estimator.

desk verdict A genuinely new training-time calibration objective with broad, consistent MT gains; the QE claim is the soft spot, since the paper likely sells teacher distillation as emergent quality estimation and never plots the actual teacher. read the letter →

arxiv 2504.19044 v3 pith:FFX7C3J2 submitted 2025-04-26 cs.CL

Di Wu , Yibin Lei , Christof Monz This is my paper · ORCID

classification cs.CL

keywords machinetranslationLLMdecodinglikelihoodcalibrationPearsoncorrelationqualityestimationCometKiwibeamsearchpreferenceoptimization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Machine translation systems pick the highest-probability hypothesis, but a hypothesis's likelihood and its true quality are only weakly correlated, so beam search often lands on mediocre or pathological outputs. This paper claims the fix can happen at training time: sample several hypotheses per prompt, score each with a reference-free neural quality metric (CometKiwi-XXL), and directly maximize the Pearson correlation between the hypotheses' log-likelihoods and those quality scores. With about 2,000 training sentences per direction and low-rank (LoRA) fine-tuning, the method improves translation across nine target languages, beats contrastive preference optimization (CPO) on the same data, and lets a 7B model with plain beam search match the quality of a 70B model that reranks 100 sampled hypotheses, roughly 200 times faster. The calibrated log-likelihood then doubles as a reference-free quality estimator whose correlation with expert human MQM judgments rivals or surpasses CometKiwi on en→de and en→ru. The paper's larger thesis is that quality optimization and quality estimation are one problem: a translator that has learned what a good translation is should both produce it and recognize it in its own output.

What carries the argument

The load-bearing object is the Pearson correlation loss. For each prompt, the model—or a fixed external sampler, in the off-policy variant—generates $k$ hypotheses by nucleus sampling, each carrying a log-likelihood $z_\theta(y_i|x)$ and a quality score $q(y_i|x)$ from CometKiwi-XXL; the loss is the negative Pearson correlation between the two lists, $-\frac{1}{k}\sum_{i=1}^{k} \frac{z_i-\mu_z}{\sigma_z}\cdot\frac{q_i-\mu_q}{\sigma_q}$, estimated under the nucleus-truncated sampling distribution. Because Pearson correlation is invariant to scale and shift, the objective never prescribes absolute likelihood values; it only reorders hypotheses by quality, which is exactly the granularity that decoding needs. The loss is differentiable and reduces to the dot product of two mean-centered, $\ell^2$-normalized vectors, so it costs almost nothing to compute. A supervised fine-tuning term on the best-scoring hypotheses anchors the absolute scale, and nucleus sampling concentrates the correlation on the likely region of the decoding space, following prior evidence that correlations in that region are what decoding cares about.

What would settle it

The decisive experiment is to run the identical training recipe with the quality signal inverted—using the negative of the CometKiwi score as $q(y|x)$—and then evaluate the resulting translations with expert human MQM judgments. If human-evaluated quality still improves, the correlation mechanism itself carries the gains; if human quality tracks the teacher's sign, the method is a distillation of CometKiwi's preferences and will inherit its failures wherever the teacher is wrong. A complementary check: use the calibrated likelihood as a quality estimator on a direction or domain outside the teacher's training distribution, where any genuinely acquired quality awareness—rather than a memorized ranking—should still correlate with human judgment.

Watch

Extended reading notes

Core claim

The paper's central claim is that likelihood-quality miscalibration—the reason the most probable hypotheses are often not the best translations—is cheaply repairable at training time. For each source sentence it draws $k$ hypotheses by nucleus sampling, computes each hypothesis's log-likelihood $z_\theta(y_i|x)$ and its quality score $q(y_i|x)$ from CometKiwi-XXL, and minimizes the negative Pearson correlation between the two sets of values, with a supervised fine-tuning term on the highest-scoring hypotheses to anchor the absolute likelihood scale. On WMT24, TowerInstruct-Mistral-7B calibrated this way and decoded with beam search of width 5 reaches translation quality comparable to Tower-70B-v2 with 100-sample MBR/TRR—a 70B system that samples 100 candidates and reranks them with metric ensembles—while outperforming both supervised fine-tuning on best-of-n samples and CPO across CometKiwi-XL, XCOMET, and COMET, with corroborating human evaluation. The paper further claims that the calibrated model's average log-likelihood correlates with expert WMT22 MQM judgments better than CometKiwi itself on en→de and en→ru, and it reads this as evidence that a well-calibrated translation model inherently knows what a good translation is: quality optimization and quality estimation are two sides of one objective.

Load-bearing premise

CometKiwi-XXL, the neural metric that supplies every training-time quality score, is a faithful and unbiased proxy for human translation quality across all nine target languages, so that teaching the model to rank hypotheses by CometKiwi's scores improves translation itself rather than merely copying the teacher's preferences.

Editorial extensions

If this is right

A 7B translator calibrated on about 2,000 sentences per direction and decoded with beam search of width 5 reaches the translation quality of a 70B system that samples 100 hypotheses and reranks them with metric ensembles (MBR/TRR), at roughly 200 times lower inference cost—expensive test-time optimization is replaceable by a short training step.
The calibrated model's average log-likelihood becomes a usable reference-free quality estimator, rivaling or surpassing CometKiwi on en→de and en→ru WMT22 MQM judgments; one model can translate and score its own output without a separate QE system.
Calibration gains are additive on top of supervised fine-tuning and hold across the evaluated metric families (CometKiwi-XL, CometKiwi-XXL, XCOMET, COMET) and in human evaluation, which the paper takes as evidence that the improvement is not mere metric hacking.
Quality keeps improving as more hypotheses per sentence are sampled during training, so the method's ceiling has not been reached at the tested sampling sizes.
Calibration makes efficient MAP decoding (beam search) competitive with sampling-based decoding, lowering the latency barrier for online translation deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The result that calibrated likelihood 'surpasses CometKiwi' is best read as distillation: the model absorbs CometKiwi's ordering and re-expresses it in a cheaper, inference-free form. A targeted probe would evaluate the calibrated likelihood on a domain or direction where CometKiwi is known to be wrong and check whether the transferred ranking inherits the teacher's blind spots.
Because the objective is correlation rather than reward maximization, it should be insensitive to monotone rescaling or systematic offset in the quality signal; that property could make Pearson calibration useful in other conditional generation tasks—summarization, code generation, or grammatical error correction—where likelihood-quality miscalibration is also documented.
The SFT regularizer's role is under-explored relative to its importance: the Pearson term alone constrains only the ordering of likelihoods, so an ablation that varies or removes the SFT weight would reveal how much of the gain comes from absolute-scale anchoring rather than correlation.
In principle, the calibrated likelihood could replace the external metric inside test-time decoding, reranking sampled hypotheses with the model's own score; whether internal reranking matches external-metric reranking would directly measure how much of the teacher's knowledge transferred.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing to know: this paper is a solid empirical methods contribution with a genuinely new objective, and its main translation-quality claim largely holds. The QE subclaim is where the paper overreaches, and that overreach is partly visible in the paper's own choices.

The new bit is the loss: negative Pearson correlation between sampled hypothesis log-likelihoods and an external quality score, plus an SFT regularizer. That is not the same as DPO/CPO contrastive objectives or test-time reranking, and it is simple enough to matter. The authors apply it to Tower and ALMA in 7B/13B variants, cover nine language directions, test on- and off-policy, include human evals, and release code, data, and annotations. The gains are consistent and sometimes large, e.g., +2.8 Kiwi-XL average over TowerInstruct-Mistral-7B, and the calibrated 7B/13B models beat CPO across the board. Table 2 is the right kind of control: calibrating with COMET, XCOMET, or Kiwi-XXL all improve held-out metrics, which is decent evidence against simple metric gaming.

Soft spots, in proportion. First, the headline "comparable to Tower-70B-v2 + MBR/TRR" is generous. On the four directions where the 70B numbers are given, the calibrated 7B is lower by about 0.7–1.8 Kiwi-XL points, and the 13B is still slightly lower. Close, but not quite matching. The abstract's "closely approximating or even surpassing" language should be toned down.

Second, the QE claim. The training teacher is CometKiwi-XXL, yet the QE plots compare against CometKiwi-XL and the older CometKiwi, never the teacher itself. If the calibrated likelihood had a Spearman correlation with human MQM above the teacher's, that would be a real emergence story. As presented, the result is consistent with distillation of the teacher's preferences into likelihood space. The authors should plot XXL's own Spearman correlation on the same WMT22 MQM data and state directly whether their likelihood exceeds or only approaches the teacher. Relatedly, "without relying on human-annotated data" is misleading: the teacher was trained on human direct assessment judgments, so the claim should be "no additional human data beyond the teacher."

Third, methodology hygiene: no error bars or multiple seeds, learning rate and checkpoint selected on the validation set, and the SFT loss-weighting coefficient is not stated. These are fixable but need to be in the paper.

Overall, the central argument about calibration improving MAP decoding and making beam search competitive with expensive sampling is well supported. The QE overreach is a substantial caveat, not a fatal flaw.

This paper deserves a serious referee. I would send it out, with a request to add the teacher baseline to the QE plots, add error bars, disclose the loss weight, and temper the claims. For an MT or LLM post-training researcher, it is a useful read.

Referee Report

3 major / 3 minor

Summary. The paper proposes a training-time method to calibrate the log-likelihoods of machine-translation hypotheses against external quality scores. For each source sentence, several hypotheses are sampled, and the model is trained with a loss that is the negative Pearson correlation between the hypothesis log-likelihoods and quality scores (e.g., CometKiwi-XXL), optionally regularized by a supervised fine-tuning term on the highest-scoring samples. The authors apply this to LLM-based translation models (ALMA and Tower, 7B and 13B) using about 2,000 training sentences per direction, and report substantial gains over the base models, over SFT and preference-optimization baselines (CPO), across multiple metrics (CometKiwi-XL, XCOMET, COMET), in both off-policy and on-policy training, and in a human evaluation on three directions. They further claim that the calibrated log-likelihood itself becomes a strong reference-free quality estimator, surpassing CometKiwi in Spearman correlation with human MQM on en-de and en-ru, and that calibration makes beam search competitive with Best-of-100 sampling. The paper includes cross-metric training experiments (Table 2), sensitivity analyses, and ablations on sampling size.

Significance. If the main claims hold, the method is important: it offers a simple, training-time alternative to expensive test-time decoding strategies such as Best-of-N reranking and MBR, and it makes a conceptual connection between quality optimization and quality estimation through a shared Pearson objective. The empirical scope is broad for a paper of this type: multiple base models, multiple source languages, both on- and off-policy training, cross-metric training, checkpoint- and sensitivity analyses, and human evaluation. The paper also releases code and human evaluation data. However, the most surprising claim, that calibrated likelihood surpasses CometKiwi as a quality estimator, is weakened by the fact that the training signal is itself a CometKiwi-family model (CometKiwi-XXL), and the teacher is never plotted on the same human-annotated data. The absence of error bars or multiple-seed results and the practice of selecting learning rates and checkpoints on the validation metric also temper confidence in the exact magnitudes. These issues are addressable; the core decoding-quality result is supported by many consistent, if variance-unreported, measurements.

major comments (3)

[§5.2, Figure 2, Eq. (3)] The claim that calibrated log-likelihood surpasses CometKiwi as a QE metric is not backed by the appropriate comparison. The training signal q(y|x) in Eq. (3) is set to CometKiwi-XXL (Section 4.2), but Figure 2 plots only "CometKiwi" and "CometKiwi-XL" as baselines against human MQM; the actual teacher, CometKiwi-XXL, is never evaluated on the same WMT22 MQM segments. Since the loss directly enforces a monotone (Pearson) relationship between log-likelihood and CometKiwi-XXL scores, the calibrated likelihood's Spearman correlation with human MQM is expected to be at most the teacher's correlation (up to estimation noise). Observing superiority over an older/smaller CometKiwi checkpoint is consistent with distilling a newer, stronger teacher into likelihood space, not with the model independently knowing translation quality. Please add the teacher's own Spearman and Kendall scores on the identical WMT22 MQM data for en-de and en-ru, and rephrase the conclusion that this is achieved "without relying on human-annotated data" (Section 7), since CometKiwi-XXL was trained on human DA judgments.
[§4.2, §5.1, Tables 1 and 3] The headline magnitudes lack statistical support. The paper reports single runs, with learning rates searched from 1e-5 to 1e-4 and checkpoints selected by validation XCOMET on NTREX, and then reports "the best results for all settings"; no standard deviations, confidence intervals, or multiple seeds are given. The reported gains, e.g., +2.8 KIWI-XL and +2.7 XCOMET over TowerInstruct-Mistral-7B (Section 5.1), are treated as conclusive, but without variance estimates the reader cannot tell whether the differences between calibration and, say, CPO are significant. Please provide at least three seeds (or bootstrap confidence intervals) for the main off-policy results, and report the exact checkpoint-selection rule and the loss weighting between L_pearson and L_sft, which is not specified anywhere in Section 3 or 4.
[§3, Appendix F.3] The on-policy training formulation ignores the gradient through the sampling distribution. In Eq. (3) the hypotheses y_i are drawn from tilde{p}_θ (or from p_θ in the on-policy case), but the paper states that gradients are propagated only through z_θ, µ_z, and σ_z, implicitly treating the samples as fixed. If the true gradients through the sampling procedure are non-negligible, the on-policy results in Appendix F.3 optimize an unstated surrogate rather than the stated Pearson objective. Please either justify why the score-function/rep-gradient term can be dropped (with an ablation), or include it in the derivation. This matters for the claimed generality 'under different training dynamics' but is secondary to the off-policy central claim; still, the current text is under-specified.

minor comments (3)

[Figure 2 and Section 4.1] The baseline 'CometKiwi' in Figure 2 is not explicitly defined. Section 4.1 lists CometKiwi-XL and CometKiwi-XXL with model versions, but Figure 2 uses a third label; please state which checkpoint (e.g., wmt22-cometkiwi-da) this refers to, for reproducibility.
[Table 1 caption and rows] The table layout mixes TowerInstruct and TowerBase rows with '+ SFT on BoN data', '+ CPO', and '+ Calibration' subrows; the caption should clarify that the '+' rows are trained on the immediately preceding base model (TowerBase or TowerInstruct), since the current text is ambiguous for readers skimming the table.
[Section 5.1, latency comparison] The 'approximately 200 times faster' claim is based on a rough 10x model-size latency ratio and 100x sampling; please give the actual measured or estimated wall-clock times, or state explicitly that this is an order-of-magnitude estimate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: calibrated likelihood is validated against independent human MQM, cross-metric benchmarks, and a held-out human study.

full rationale

The paper's load-bearing claims are checked against external signals rather than against the training objective itself. Equation (3) optimizes Pearson correlation between sampled hypothesis log-likelihoods and CometKiwi-XXL quality scores, but the headline quality-estimation result (Section 5.2, Figure 2) is measured as Spearman correlation with WMT22 human MQM annotations, and the translation-quality results are measured with CometKiwi-XL, XCOMET, COMET, and a held-out human study (Section 6.2). These evaluation quantities are not the training target and are not definitionally equal to Eq. (3). Using CometKiwi-XXL as the training signal does make the calibrated likelihood a potential distillation of that teacher, and the paper would be stronger if it plotted the teacher's own WMT22 QE score; likewise, the conclusion's phrase 'without relying on human-annotated data' overstates the situation because CometKiwi-XXL was trained on human DA. But distillation and benchmark-selection concerns are correctness or limitation issues, not circular derivations: the student's Spearman correlation with human MQM on a held-out set is an empirical generalization, and Table 2 shows improvements transfer across metric families. There is no load-bearing self-citation chain, and no equation in the paper reduces the reported predictions to their inputs.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central claim rests on standard calculus (Pearson correlation gradients), a sampling approximation of the decoding space, and the fidelity of an external metric as a quality teacher. No new physical or linguistic entities are postulated. The main free choices are the sampling size, the unstated loss weighting, and hyperparameters selected by validation.

free parameters (6)

Training sampling size k = 16 (off-policy), 5 (on-policy top-k)
Number of hypotheses sampled per source sentence to estimate Pearson correlation; chosen by hand, with sensitivity tested in Figure 5.
Loss weighting between L_pearson and L_sft = not reported; appears to be equal unit weighting
Section 3 gives L_cal = L_pearson + L_sft with no coefficient. Since L_pearson is bounded near [-1,1] and L_sft is a token-level negative log-likelihood, the effective balance is an unstated tuning choice.
Learning rate = 1e-5 to 1e-4, best selected per setting
Authors report best results across the sweep; not fixed a priori.
LoRA hyperparameters = rank 8, alpha 32, dropout 0.05
Standard but hand-chosen; affect the capacity and behavior of the fine-tuned model.
Calibration dataset size = 2,009 English sentences from Flores dev/devtest
Chosen as an available small set; only English source sentences are used, so calibration data for other source languages is not directly tested.
Nucleus sampling top-p / top-k = top-p 0.98 (off-policy), top-k 5 (on-policy)
Defines the sampling distribution used to approximate the decoding space; changes the effective training distribution.

assumptions (6)

standard math Pearson correlation is differentiable and its gradient through z_theta, mu_z, and sigma_z is usable with Adam
Appendix C.1 shows Pearson correlation reduces to a dot product of centered, normalized vectors; true but requires treating sampled log-likelihoods as reparameterized outputs.
domain assumption Nucleus sampling with small k approximates the full output distribution well enough for calibration
Section 3 Eq. 3 uses the nucleus-induced distribution p~; authors acknowledge the bias and justify by focusing on likely hypotheses per Ott et al. 2018.
domain assumption CometKiwi-XXL quality scores are a valid ground-truth signal for translation quality
Section 4.2 uses CometKiwi-XXL as the training signal; all downstream claims depend on this teacher's fidelity.
ad hoc to paper Improved likelihood-quality correlation under the sampled distribution transfers to improved MAP/beam-search decoding
Section 5.3 claims beam search improves after calibration; there is no formal transfer argument, only empirical observation.
domain assumption SFT on highest-scoring samples anchors the absolute scale without distorting the correlation objective
Section 3 introduces L_sft to avoid absolute-scale drift; the interaction between the two loss terms is not analyzed.
ad hoc to paper The gradient through the sampling distribution p_theta is negligible in on-policy training
Section 3 says gradients are propagated through z_theta, mu_z, and sigma_z only; for on-policy sampling the samples also depend on theta, so the update is a biased estimator of the Pearson loss gradient.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Calibrating Translation Decoding with Quality Estimation on LLMs." pith.science (2026). https://pith.science/paper/FFX7C3J2

@misc{pith2026250419044,
  author       = {Pith},
  title        = {Pith review of: Calibrating Translation Decoding with Quality Estimation on LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FFX7C3J2}},
  note         = {Machine review of arXiv:2504.19044}
}

read the original abstract

Neural machine translation (NMT) systems typically employ maximum a posteriori (MAP) decoding to select the highest-scoring translation from the distribution mass. However, recent evidence highlights the inadequacy of MAP decoding, often resulting in low-quality or even pathological hypotheses -- the decoding objective is not aligned with real-world translation quality. This paper proposes calibrating hypothesis likelihoods with translation quality from a distribution view by directly optimizing their Pearson correlation -- thereby enhancing the effectiveness of translation decoding. With our method, translation on large language models (LLMs) improves substantially after limited training (2K instances per direction). This improvement is orthogonal to those achieved through supervised fine-tuning, leading to substantial gains across a broad range of metrics and human evaluations -- even when applied to top-performing translation-specialized LLMs fine-tuned on high-quality translation data, such as Tower, or when compared to recent preference optimization methods, like CPO. Moreover, the calibrated translation likelihood can directly serve as a strong proxy for translation quality, closely approximating or even surpassing some state-of-the-art translation quality estimation models, like CometKiwi. Lastly, our in-depth analysis demonstrates that calibration enhances the effectiveness of MAP decoding, thereby enabling greater efficiency in real-world deployment. The resulting state-of-the-art translation model, which covers 10 languages, along with the accompanying code and human evaluation data, has been released to the community: https://github.com/moore3930/calibrating-llm-mt.

Figures

Figures reproduced from arXiv: 2504.19044 by the authors.

**Figure 2.** The Spearman coefficient and the corresponding translation performance in [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** The average performance (en→de,es,ru,zh) measured by COMET score for both TowerInstruct-Mistral-7B and our calibrated model, when varying sampling size for Best-of-N (BoN) sampling and applying beam search with a beam size of 5 [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Human evaluation results in en→zh direction: score distribution and an aggregated summary. TScore and CScore denote the average scores over 200 annotations for TowerInstructMistral-7B and our calibrated model, respectively. Win, Loss, and Tie indicate the number of sa…

**Figure 5.** Figure 5: Average performance variation on the WMT24 dataset across different sampling sizes [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

**Figure 6.** Figure 6: Detailed annotation results illustrate the number of changes in [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]

**Figure 7.** Figure 7: Human evaluation results in en→ru direction: score distribution and an aggregated summary. TScore and CScore denote the average scores over 200 annotations for TowerInstructMistral-7B and our calibrated model, respectively. Win, Loss, and Tie indicate the number of sa…

**Figure 8.** Figure 8: The Spearman coefficient and the corresponding translation performance in [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]

**Figure 9.** Figure 9: The Kendall coefficient and the corresponding translation performance in [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]

**Figure 10.** Figure 10: The Kendall coefficient and the corresponding translation performance in [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 8 canonical work pages

[1]

Tower: An open multilingual large language model for translation-related tasks.arXiv preprint arXiv:2402.17733,

Duarte M Alves, José Pombal, Nuno M Guerreiro, Pedro H Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, et al. Tower: An open multilingual large language model for translation-related tasks.arXiv preprint arXiv:2402.17733,

arXiv
[3]

Each data point represents the number of samples corresponding to a specific pair of scores before and after calibration (vertical and horizontal axes), respectively

18 en-zhCalibrated Model0 1 2 3 4 5 6 TowerInstruct-Mistral-7B 0 0 0 0 0 0 0 31 0 0 0 0 0 0 02 0 0 0 1 1 0 13 0 0 0 4 4 2134 0 0 0 3 6 5295 0 0 0 1 5 8476 0 0 0 0 51844 Figure 6: Detailed annotation results illustrate the number of changes in en→zh translations. Each data point represents the number of samples corresponding to a specific pair of scores be...

work page arXiv 2023
[6]

doi: 10.18653/v1/2022.sum eval-1.4

Association for Computational Linguistics. doi: 10.18653/v1/2022.sum eval-1.4. URLhttps://aclanthology.org/2022.sumeval-1.4/. Patrick Fernandes, António Farinhas, Ricardo Rei, José G. C. de Souza, Perez Ogayo, Graham Neubig, and Andre Martins. Quality-aware decoding for neural machine translation. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan V...

work page doi:10.18653/v1/2022.sum 2022
[7]

doi: 10.18653/v1/2022.naacl-main.100

Association for Computational Linguistics. doi: 10.18653/v1/2022.naacl-main.100. URL https://aclanthology.org/2022.naacl-main.100/. Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. Unsupervised quality estimation for neural machine translation.Transactions o...

doi:10.18653/v1/2022.naacl-main.100 2022
[10]

doi: 10.18653/v1/2023.wmt-1.51

Association for Computational Linguistics. doi: 10.18653/v1/2023.wmt-1.51. URLhttps://aclanthology.org/2023.wmt-1.51/. Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback.arXiv preprint arXiv:2402.04792,

arXiv 2023
[11]

How good are gpt models at machine translation? a comprehensive evaluation.arXiv preprint arXiv:2302.09210,

Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Mat- sushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. How good are gpt models at machine translation? a comprehensive evaluation.arXiv preprint arXiv:2302.09210,

arXiv
[12]

Evaluation of best-of-n sampling strategies for language model alignment.arXiv preprint arXiv:2502.12668,

Yuki Ichihara, Yuu Jinnai, Tetsuro Morimura, Kaito Ariu, Kenshi Abe, Mitsuki Sakamoto, and Eiji Uchibe. Evaluation of best-of-n sampling strategies for language model alignment.arXiv preprint arXiv:2502.12668,

arXiv
[14]

doi: 10.18653/v1/2023.wmt-1.63

Association for Computational Linguistics. doi: 10.18653/v1/2023.wmt-1.63. URL https://aclanthology.o rg/2023.wmt-1.63/. Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. Metricx-24: The google submission to the wmt 2024 metrics shared task.arXiv preprint arXiv:2410.03983,

arXiv 2023

Show all 34 references

[15]

Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980,

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980,

arXiv
[17]

URLhttps://aclanthology.org/2022.wmt-1.1/

Association for Computational Linguistics. URLhttps://aclanthology.org/2022.wmt-1.1/. Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondˇrej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Kar...

2022
[19]

doi: 10.18653/v1/2024.wmt-1 .109

Association for Computational Linguistics. doi: 10.18653/v1/2024.wmt-1 .109. URLhttps://aclanthology.org/2024.wmt-1.109/. Aviral Kumar and Sunita Sarawagi. Calibration of encoder decoder models for neural machine translation.arXiv preprint arXiv:1903.00802,

2024 arXiv
[25]

COMET: A neural framework for MT evaluation

Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. COMET: A neural framework for MT evaluation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors,Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–270...

2020
[26]

doi: 10.18653/v 1/2020.emnlp-main.213

Association for Computational Linguistics. doi: 10.18653/v 1/2020.emnlp-main.213. URLhttps://aclanthology.org/2020.emnlp-main.213/. Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte ...

2020 arXiv
[27]

doi: 10.18653/v1/2020 .acl-main.278

Association for Computational Linguistics. doi: 10.18653/v1/2020 .acl-main.278. URLhttps://aclanthology.org/2020.acl-main.278/. Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. A paradigm shift in ma- chine translation: Boosting translation performance of large ...

2020 arXiv
[28]

Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation.arXiv preprint arXiv:2401.08417,

Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation.arXiv preprint arXiv:2401.08417,

arXiv
[29]

Direct preference optimization for neural machine translation with minimum Bayes risk decoding

Guangyu Yang, Jinghong Chen, Weizhe Lin, and Bill Byrne. Direct preference optimization for neural machine translation with minimum Bayes risk decoding. In Kevin Duh, Helena Gomez, and Steven Bethard, editors,Proceedings of the 2024 Conference of the North American Chapter of ...

2024
[30]

doi: 10.18653/v1/2024.naacl-short.34

Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-short.34. URL https://aclanthology.org/2024. naacl-short.34/. Chrysoula Zerva, Frédéric Blain, Ricardo Rei, Piyawat Lertvittayakumjorn, José G. C. de Souza, Steffen Eger, Diptesh Kanojia, Duarte Alves, Cons...

2024 doi
[31]

URLhttps://aclanthology.org/2022.wmt-1.3/

Association for Computational Linguistics. URLhttps://aclanthology.org/2022.wmt-1.3/. Yao Zhao, Misha Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu. Calibrating sequence likelihood improves conditional language generation.arXiv preprint arXiv:2210.00045,

2022 arXiv
[32]

A preference-driven paradigm for enhanced translation with large language models

Dawei Zhu, Sony Trenous, Xiaoyu Shen, Dietrich Klakow, Bill Byrne, and Eva Hasler. A preference-driven paradigm for enhanced translation with large language models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors,Proceedings of the 2024 Conference of the North American...

2024
[33]

doi: 10.18653/v1/2024.naacl-long.186

Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.186. URL https://aclanthology.org/2024.naacl-long.186/. 15 A Limitations In this paper, we examined the impact of our calibration method under an approximate maximuma posteriori(MAP) decoding (i.e., be...

2024 doi
[1952]

Large language monkeys: Scaling inference compute with repeated sampling

Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787,

arXiv
[2002]

doi: 10.3115/1073083.1073135

Association for Computational Linguistics. doi: 10.3115/1073083.1073135. URLhttps://aclanthology.org/P02-1040/. Robin L Plackett. The analysis of permutations.Journal of the Royal Statistical Society Series C: Applied Statistics, 24(2):193–202,
[2004]

URL https://aclanthology.org/N04-1022/

Association for Computational Linguistics. URL https://aclanthology.org/N04-1022/. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T\" ulu 3: Pushing frontiers in ...

arXiv
[2014]

Findings of the 2022 conference on machine translation (WMT22)

Tom Kocmi, Rachel Bawden, Ondˇrej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin ...

2022
[2015]

doi: 10.18653/v1/W15-3049

Association for Computational Linguistics. doi: 10.18653/v1/W15-3049. URLhttps://aclanthology.org/W15-3049/. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a...

doi
[2017]

doi: 10.18653/v1/W17-3204

Association for Computational Linguistics. doi: 10.18653/v1/W17-3204. URLhttps://aclanthology.org/W17-3204/. Geza Kovacs, Daniel Deutsch, and Markus Freitag. Mitigating metric bias in minimum Bayes risk decoding. In Barry Haddow, Tom Kocmi, Philipp Koehn, and Christof Monz, ed...

doi
[2018]

doi: 10.18653/v1/ W18-6322

Association for Computational Linguistics. doi: 10.18653/v1/ W18-6322. URLhttps://aclanthology.org/W18-6322/. Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. Analyzing uncertainty in neural machine translation. InInternational Conference on Machine Learning, ...

doi
[2019]

Minimum Bayes-risk decoding for statistical machine translation

12 Shankar Kumar and William Byrne. Minimum Bayes-risk decoding for statistical machine translation. InProceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 169–176, Boston,...

2004
[2020]

URLhttps://aclanthology.org/2020.tacl-1.35/

doi: 10.1162/tacl_a_00330. URLhttps://aclanthology.org/2020.tacl-1.35/. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions...

2020 doi
[2021]

URLhttps://aclanthology.org/2021.tacl-1.87/

doi: 10.1162/ tacl_a_00437. URLhttps://aclanthology.org/2021.tacl-1.87/. Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. High quality rather than high model probability: Minimum Bayes risk decoding with neural metrics.Transactions of the Association for Computation...

2021 doi
[2022]

Kto: Model alignment as prospect theoretic optimization.arXiv preprint arXiv:2402.01306,

Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization.arXiv preprint arXiv:2402.01306,

arXiv
[2023]

No language left behind: Scaling human-centered machine translation.arXiv preprint arXiv:2207.04672,

Marta R Costa-Jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centered machine translation.arXiv preprint arXiv:2207.04672,

arXiv
[2024]

Open problems and fundamental limitations of reinforcement learning from human feedback.arXiv preprint arXiv:2307.15217,

Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback.arXiv preprint arXiv:23...

arXiv
[2025]

MetricX-23: The Google submission to the WMT 2023 metrics shared task

Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, and Markus Freitag. MetricX-23: The Google submission to the WMT 2023 metrics shared task. In Philipp Koehn, Barry Haddow, Tom Kocmi, and Christof Monz, editors,Proceedings of the Eighth Confer...

2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Tower: An open multilingual large language model for translation-related tasks.arXiv preprint arXiv:2402.17733,

Duarte M Alves, José Pombal, Nuno M Guerreiro, Pedro H Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, et al. Tower: An open multilingual large language model for translation-related tasks.arXiv preprint arXiv:2402.17733,

arXiv

[2] [3]

Each data point represents the number of samples corresponding to a specific pair of scores before and after calibration (vertical and horizontal axes), respectively

18 en-zhCalibrated Model0 1 2 3 4 5 6 TowerInstruct-Mistral-7B 0 0 0 0 0 0 0 31 0 0 0 0 0 0 02 0 0 0 1 1 0 13 0 0 0 4 4 2134 0 0 0 3 6 5295 0 0 0 1 5 8476 0 0 0 0 51844 Figure 6: Detailed annotation results illustrate the number of changes in en→zh translations. Each data point represents the number of samples corresponding to a specific pair of scores be...

work page arXiv 2023

[3] [6]

doi: 10.18653/v1/2022.sum eval-1.4

Association for Computational Linguistics. doi: 10.18653/v1/2022.sum eval-1.4. URLhttps://aclanthology.org/2022.sumeval-1.4/. Patrick Fernandes, António Farinhas, Ricardo Rei, José G. C. de Souza, Perez Ogayo, Graham Neubig, and Andre Martins. Quality-aware decoding for neural machine translation. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan V...

work page doi:10.18653/v1/2022.sum 2022

[4] [7]

doi: 10.18653/v1/2022.naacl-main.100

Association for Computational Linguistics. doi: 10.18653/v1/2022.naacl-main.100. URL https://aclanthology.org/2022.naacl-main.100/. Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. Unsupervised quality estimation for neural machine translation.Transactions o...

doi:10.18653/v1/2022.naacl-main.100 2022

[5] [10]

doi: 10.18653/v1/2023.wmt-1.51

Association for Computational Linguistics. doi: 10.18653/v1/2023.wmt-1.51. URLhttps://aclanthology.org/2023.wmt-1.51/. Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback.arXiv preprint arXiv:2402.04792,

arXiv 2023

[6] [11]

How good are gpt models at machine translation? a comprehensive evaluation.arXiv preprint arXiv:2302.09210,

Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Mat- sushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. How good are gpt models at machine translation? a comprehensive evaluation.arXiv preprint arXiv:2302.09210,

arXiv

[7] [12]

Evaluation of best-of-n sampling strategies for language model alignment.arXiv preprint arXiv:2502.12668,

Yuki Ichihara, Yuu Jinnai, Tetsuro Morimura, Kaito Ariu, Kenshi Abe, Mitsuki Sakamoto, and Eiji Uchibe. Evaluation of best-of-n sampling strategies for language model alignment.arXiv preprint arXiv:2502.12668,

arXiv

[8] [14]

doi: 10.18653/v1/2023.wmt-1.63

Association for Computational Linguistics. doi: 10.18653/v1/2023.wmt-1.63. URL https://aclanthology.o rg/2023.wmt-1.63/. Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. Metricx-24: The google submission to the wmt 2024 metrics shared task.arXiv preprint arXiv:2410.03983,

arXiv 2023