Pith. sign in

REVIEW 2 cited by

Preliminary WMT24 Ranking of General MT Systems and LLMs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.19884 v1 pith:P253ZE6C submitted 2024-07-29 cs.CL

Tom Kocmi , Eleftherios Avramidis , Rachel Bawden , Ondrej Bojar , Anton Dvorkovich , Christian Federmann , Mark Fishel , Markus Freitag
show 13 more authors
Thamme Gowda Roman Grundkiewicz Barry Haddow Marzena Karpinska Philipp Koehn Benjamin Marie Kenton Murray Masaaki Nagata Martin Popel Maja Popovic Mariya Shmatova Stein{th}ór Steingrímsson Vilém Zouhar
This is my paper · ORCID
classification cs.CL
keywords rankinggeneralpreliminaryautomaticsystemswmt24duringevaluation
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

This is the preliminary ranking of WMT24 General MT systems based on automatic metrics. The official ranking will be a human evaluation, which is superior to the automatic ranking and supersedes it. The purpose of this report is not to interpret any findings but only provide preliminary results to the participants of the General MT task that may be useful during the writing of the system submission.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ConsistencyChecker: Tree-based Evaluation of LLM Generalization Capabilities

    cs.AI 2025-06 conditional novelty 6.0 of 10

    ConsistencyChecker ranks LLMs by how well they survive chains of reversible transformations, and those scores track WMT 2024 translation quality rankings (r > 0.7) without using WMT paired data.

  2. How to Select Datapoints for Efficient Human Evaluation of NLG Models?

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Selecting human-evaluation items by metric variance, metric consistency, output diversity, or IRT-based informativeness matches random-sampling ranking accuracy with roughly 70% of the annotation budget in WMT23 and SummEval.

Pith tools