Pith. sign in

REVIEW 5 cited by

MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.03983 v1 pith:4CRQR54W submitted 2024-10-04 cs.CL

Juraj Juraska , Daniel Deutsch , Mara Finkelstein , Markus Freitag This is my paper · ORCID
classification cs.CL
keywords metricratingsmetricsmetricx-24previoussharedsubmissionsynthetic
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In this paper, we present the MetricX-24 submissions to the WMT24 Metrics Shared Task and provide details on the improvements we made over the previous version of MetricX. Our primary submission is a hybrid reference-based/-free metric, which can score a translation irrespective of whether it is given the source segment, the reference, or both. The metric is trained on previous WMT data in a two-stage fashion, first on the DA ratings only, then on a mixture of MQM and DA ratings. The training set in both stages is augmented with synthetic examples that we created to make the metric more robust to several common failure modes, such as fluent but unrelated translation, or undertranslation. We demonstrate the benefits of the individual modifications via an ablation study, and show a significant performance increase over MetricX-23 on the WMT23 MQM ratings, as well as our new synthetic challenge set.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TQLite: Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation

    cs.CL 2026-08 conditional novelty 6.0 of 10

    Distilling agreement-filtered multi-LRM jury annotations into Gemma-3-12B improves MQM translation quality evaluation from 52.63% to 55.03% average segment-level accuracy, approaching closed LRMs.

  2. ConsistencyChecker: Tree-based Evaluation of LLM Generalization Capabilities

    cs.AI 2025-06 conditional novelty 6.0 of 10

    ConsistencyChecker ranks LLMs by how well they survive chains of reversible transformations, and those scores track WMT 2024 translation quality rankings (r > 0.7) without using WMT paired data.

  3. Quality-Aware Decoding: Unifying Quality Estimation and Decoding

    cs.CL 2025-02 conditional novelty 6.0 of 10

    A uni-directional token-level QE model that scores partial translations is merged into beam search, improving NMT quality over N-best re-ranking on WMT23 English-German and Chinese-English.

  4. How to Select Datapoints for Efficient Human Evaluation of NLG Models?

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Selecting human-evaluation items by metric variance, metric consistency, output diversity, or IRT-based informativeness matches random-sampling ranking accuracy with roughly 70% of the annotation budget in WMT23 and SummEval.

  5. TACTIC: Translation Agents with Cognitive-Theoretic Interactive Collaboration

    cs.CL 2025-06 conditional novelty 4.0 of 10

    TACTIC, a cognitive-inspired six-agent workflow, improves LLM translation quality over direct prompting on FLORES-200 and WMT24, with the best DeepSeek-V3 setup reaching 96.19 XCOMET on English-to-X.

Pith tools