REVIEW 7 cited by
MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
In this paper, we present the MetricX-24 submissions to the WMT24 Metrics Shared Task and provide details on the improvements we made over the previous version of MetricX. Our primary submission is a hybrid reference-based/-free metric, which can score a translation irrespective of whether it is given the source segment, the reference, or both. The metric is trained on previous WMT data in a two-stage fashion, first on the DA ratings only, then on a mixture of MQM and DA ratings. The training set in both stages is augmented with synthetic examples that we created to make the metric more robust to several common failure modes, such as fluent but unrelated translation, or undertranslation. We demonstrate the benefits of the individual modifications via an ablation study, and show a significant performance increase over MetricX-23 on the WMT23 MQM ratings, as well as our new synthetic challenge set.
Forward citations
Cited by 7 Pith papers
-
From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set
Using per-example in-context demonstrations built from historical same-source human MQM ratings makes an LLM judge dramatically better at fine-grained MT evaluation on WMT'23 and WMT'24.
-
TQLite: Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation
Distilling agreement-filtered multi-LRM jury annotations into Gemma-3-12B improves MQM translation quality evaluation from 52.63% to 55.03% average segment-level accuracy, approaching closed LRMs.
-
ConsistencyChecker: Tree-based Evaluation of LLM Generalization Capabilities
ConsistencyChecker ranks LLMs by how well they survive chains of reversible transformations, and those scores track WMT 2024 translation quality rankings (r > 0.7) without using WMT paired data.
-
Calibrating Translation Decoding with Quality Estimation on LLMs
Optimizing the Pearson correlation between hypothesis likelihood and an external quality score during fine-tuning improves LLM translation quality and turns log-likelihood into a competitive reference-free quality estimator.
-
Quality-Aware Decoding: Unifying Quality Estimation and Decoding
A uni-directional token-level QE model that scores partial translations is merged into beam search, improving NMT quality over N-best re-ranking on WMT23 English-German and Chinese-English.
-
How to Select Datapoints for Efficient Human Evaluation of NLG Models?
Selecting human-evaluation items by metric variance, metric consistency, output diversity, or IRT-based informativeness matches random-sampling ranking accuracy with roughly 70% of the annotation budget in WMT23 and SummEval.
-
TACTIC: Translation Agents with Cognitive-Theoretic Interactive Collaboration
TACTIC, a cognitive-inspired six-agent workflow, improves LLM translation quality over direct prompting on FLORES-200 and WMT24, with the best DeepSeek-V3 setup reaching 96.19 XCOMET on English-to-X.
Discussion (0). Continue with ORCID to comment.