Pith. sign in

REVIEW 4 major objections 5 minor 91 references

TQLite: Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Distilling a multi-LRM jury into small open models lets a 12B evaluator reach 55% segment-level accuracy, outperforming all open reasoning-model baselines and approaching closed ones.

desk verdict Useful benchmark and a sensible distillation recipe, but the central 12B result is tuned on the test set and the synthetic labels are unvalidated; treat 55.03% as an upper bound until held-out checks. read the letter →

arxiv 2608.02975 v1 pith:FZZHQ6JT submitted 2026-08-04 cs.CL cs.AIcs.LG

Bhavin Jawade , Cameron R. Wolfe This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords translationqualityevaluationMQMknowledgedistillationlargereasoningmodelssmalllanguageLLM-as-a-JudgesyntheticdataWMT22metrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a small, open language model can be trained to judge translation quality almost as well as the largest commercial reasoning models, at a fraction of the compute. MQM (Multidimensional Quality Metrics) is the rubric used: evaluators mark error spans with a category and severity. The method generates synthetic MQM annotations with a panel of large reasoning models, keeps only segments where the panel agrees, merges duplicate annotations with a meta-judge, and fine-tunes a small model on those labels. The load-bearing result is that Gemma-3-12B-it distilled through this pipeline reaches 55.03% average segment-level pairwise accuracy on the WMT22 metrics test set, up from 52.63% for the base model and above every open reasoning-model baseline tested. If true, this gives a practical path to cheap, real-time translation quality evaluation that does not depend on closed APIs.

What carries the argument

The central object is the multi-LRM jury with agreement filtering. A panel of large reasoning models (o1 and o3) independently annotates each segment; the score range $\Delta_i = \max_m S_i^{(m)} - \min_m S_i^{(m)}$ across the jury categorizes examples from perfect agreement ($\Delta_i = 0$) to no agreement ($\Delta_i > 10$). Only segments with perfect or high agreement go into the training set, and a meta-judge model consolidates duplicate error spans and assigns canonical categories and severities. This consensus-based data curation is what carries the argument: it filters out noisy teacher labels and gives the small student a clean, consistent target to imitate.

What would settle it

Sample segments where the jury had perfect agreement, ask human annotators to judge whether the synthetic error spans are correct, and measure the false-positive rate; if a large share of agreed-upon annotations are wrong, the student is inheriting systematic teacher bias.

Watch

Extended reading notes

Core claim

The paper's central claim is that the evaluation capability of frontier reasoning models can be transferred to a small student by distilling the jury's consensus rather than any single teacher. Given a pool of source-translation pairs, the method asks each member of a panel of large reasoning models to output MQM error spans and scores, computes the score range across the panel, and keeps only examples with perfect or high agreement. A meta-judge merges the surviving annotations into one canonical set, and the student is trained by supervised fine-tuning with low-rank adaptation to reproduce that structured output from the same prompt. On the WMT22 metrics test set, the best student, Gemma-3-12B-it, reaches 55.03% average segment-level accuracy, outperforming all open reasoning models tested and approaching closed reasoning models, while running much faster and cheaper.

Load-bearing premise

The synthetic labels produced by the reasoning-model jury and cleaned by the meta-judge are accurate enough to train on, meaning that high jury agreement is a valid proxy for annotation quality.

Editorial extensions

If this is right

  • A 12-billion-parameter open model can serve as a practical replacement for closed APIs in segment-level MQM evaluation, with system-level accuracy of 85.04% and segment-level accuracy of 55.03% on WMT22.
  • Jury agreement is a usable proxy for label quality in synthetic-data distillation: training on lower-agreement samples degrades the student from 53.54% (perfect agreement) to 51.24% (low agreement) at 20K samples.
  • Distillation continues to improve with more curated data, rising from 52.59% segment-level accuracy at 5K samples to 55.29% at 99K samples.
  • LoRA rank offers a capacity tradeoff: rank 128 gives the best 20K-sample result (53.80%), and rank 256 slightly overfits on that small set.
  • The distilled student shows the best accuracy-per-inference-time tradeoff among the evaluators tested, making real-time translation quality evaluation feasible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is applying the same agreement-filtering recipe to other fine-grained evaluation tasks, such as summarization error detection or factuality annotation, where multiple judges can propose overlapping spans and a meta-judge can merge them.
  • The paper does not validate the synthetic labels against human MQM annotations on the training distribution; an obvious test is comparing students trained on agreement-filtered synthetic data with students trained on an equal-sized set of human labels.
  • Because the student inherits teacher biases, comparing the student's error-category distribution to human annotators on a held-out set would reveal whether distillation amplifies systematic over-flagging or under-flagging.
  • The monotonic scaling trend suggests that larger curated pools beyond 100K samples could close the remaining gap to closed reasoning models, provided agreement filtering continues to yield high-precision labels.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents an extensive empirical study of LLM- and LRM-based MQM translation-quality evaluation on the WMT22 metrics test set, and proposes TQLite, a distillation framework in which a jury of two LRMs (OpenAI o1 and o3) generates synthetic MQM error-span annotations, a meta-judge (GPT-4.1-mini) merges and cleans the annotations, and agreement-based filtering selects high-confidence segments for supervised fine-tuning of small open models (Gemma-3-4b-it and Gemma-3-12b-it) via LoRA. The headline result is that the distilled 12B model reaches 55.03% average segment-level accuracy (acc*) versus 52.63% for the base model, outperforming open LRM baselines and approaching closed LRMs, while the 4B model does not improve over its base. The paper also reports system-level accuracy, inference-time trade-offs, and ablations over training-set size, LoRA rank, and agreement threshold.

Significance. If the central empirical claim holds, TQLite is a practically valuable contribution: it provides a cost-effective, open, small-model alternative to expensive LRM-based MQM evaluators, and the accompanying benchmark of many LLMs and LRMs under a standardized protocol is a useful reference. The paper uses official WMT22 evaluation scripts, covers a broad model zoo, and explicitly commits to releasing code and data, which strengthens reproducibility. However, the central claim currently rests on hyperparameters and the final model configuration being selected using the WMT22 test set itself, with no held-out validation or error bars, and the synthetic training labels are never checked against human MQM annotations; both issues directly affect the validity of the reported 55.03% result and the general claim that SLMs can be distilled to approach LRM-level evaluation.

major comments (4)
  1. [§6.2, Table 3, Figs. 4–6] The final TQLite configuration (perfect-agreement filter, 99,214 training samples, LoRA rank 256) is selected by monitoring average segment-level accuracy on the WMT22 test set: Figures 4, 5, and 6 plot test-set acc* against training-sample count, LoRA rank, and jury-agreement threshold, respectively, and Table 3 reports the resulting best configuration. No held-out split is used and no error bars or significance tests are reported, so the headline 55.03% versus 52.63% comparison is a test-set-selected estimate and is likely to be an optimistic upper bound. The authors should re-run the hyperparameter selection on a validation split and report the resulting test-set numbers, or otherwise quantify selection bias.
  2. [§6.2, Table 3] Table 3 shows that the distilled Gemma-3-4b-it model (53.13% average segment-level acc*) is worse than its pretrained base (53.36%), which directly contradicts the abstract's general claim that 'SLMs trained via TQLite achieve strong MQM evaluation performance that far exceeds off-the-shelf evaluation capabilities of standard SLMs.' The claim is only supported for the 12B student; the paper should either qualify the claim to the 12B model or analyze why distillation fails for the 4B model.
  3. [§5, Synthetic data generation] The 99,214 synthetic training annotations produced by the o1/o3 jury and cleaned by the meta-judge are never validated against human MQM labels. The assumption that high jury agreement (Δ_i = 0) is a valid proxy for annotation quality is load-bearing: if the jury systematically agrees on incorrect error annotations, the student will inherit that bias, and the measured improvement over the base model could reflect learning output formatting or test-set-specific patterns rather than genuine MQM evaluation ability. The authors should validate the synthetic labels against human MQM annotations on a held-out sample of the training distribution, or at least report agreement statistics between the filtered synthetic labels and available human-labeled data.
  4. [§6.1, Implementation Details] The manuscript refers to the student models as 'Gemma-12B-it' and 'Gemma-27B-it' in Section 5 and Section 6.1, but Table 3 evaluates 'gemma-3-12b-it' and 'gemma-3-4b-it'; no results are shown for a 27B student. This naming inconsistency makes it difficult to identify exactly which models were trained and evaluated, and should be corrected for reproducibility.
minor comments (5)
  1. [Table 3 caption] The caption of Table 3 reads 'Performance of LRMs on the WMT22 test set,' but the table reports pretrained and TQLite-distilled Gemma SLMs, not LRMs; this is misleading and should be corrected.
  2. [§6.2, Figure 4] The text says accuracy rises to 0.5529 at 99K training samples, while Table 3 reports 55.03% for the 99,214-sample model; the relationship between these numbers should be clarified to avoid apparent inconsistency.
  3. [Abstract and throughout] The abstract and several passages refer to a 'multi-LLM jury,' but the jury described in Section 5 and 6.1 consists of LRMs (o1 and o3), not standard LLMs; the terminology should be made consistent.
  4. [§5, Finetuning the student model] The statement that the jury models 'can exceed 10^11–10^12 parameters' is unsupported, since the parameter counts of closed LRMs such as o1 and o3 are not public; this claim should be removed or replaced with a verifiable statement.
  5. [§6.2, Discussion] No confidence intervals, standard deviations, or significance tests are reported for any of the accuracy differences (e.g., 55.03% vs 52.63%, or the small differences among LRM baselines), even though many of these gaps are only a few tenths of a percentage point; reporting some measure of uncertainty would strengthen the comparisons.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the distilled student is scored against human MQM labels on WMT22, independent of the LRM jury outputs used for supervision.

full rationale

The paper's derivation chain is linear: benchmark LLM/LRM judges on WMT22 (Section 4), select a multi-LRM jury, generate synthetic ESA annotations on a separate 3M-segment corpus (Section 5), filter by jury agreement, merge via a meta-judge, fine-tune Gemma students, and then evaluate the students on the WMT22 human-labeled MQM test set (Section 6). The headline comparison in Table 3 is between the student's predicted error spans and human MQM labels, not between the student and the teacher jury's own scores. The student is never trained on WMT22 human labels, so the reported 55.03% segment-level accuracy is not forced by construction to match the synthetic supervision. The teacher panel selection, agreement-threshold choice, and LoRA-rank tuning do use WMT22 test-set performance, which is an in-sample selection risk and a legitimate reproducibility concern, but it is not a circular reduction: the final accuracy is an independent measurement against human gold labels. The Limitations section explicitly acknowledges that synthetic labels may inherit teacher bias and that low-resource generalization is untested; these are validity limitations, not evidence that the derivation is equivalent to its inputs. No load-bearing self-citations, imported uniqueness theorems, or ansatz-via-citation steps appear in the manuscript. Therefore no specific circular step can be exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim depends on several hand-chosen thresholds and design choices (agreement buckets, LoRA rank, sample size, jury composition) and on the assumption that jury agreement equals label quality; no new theoretical entities are introduced.

free parameters (4)
  • Agreement threshold for data filtering = Perfect agreement (delta_i=0)
    The paper buckets jury agreement into five levels and selects Perfect-Agreement as the main training filter; this choice affects the student's performance and was made after comparing accuracy on WMT22 (Figure 6).
  • LoRA rank = 256
    Used for TQLite-distilled models; Figure 5 shows peak performance at rank 128 on 20K samples, so the rank is a tuned hyperparameter.
  • Number of training samples = 99,214
    The main results use the full Perfect-Agreement set; scaling in Figure 4 shows accuracy still rising at the maximum, so the data size is a capacity/filter tradeoff choice.
  • Jury composition = OpenAI o1 and o3; meta-judge GPT-4.1-mini
    The choice of teachers and meta-judge determines the synthetic labels and is a design choice; changing the jury likely changes results.
assumptions (5)
  • domain assumption WMT22 metrics test set human MQM labels are valid gold-standard scores for translation quality.
    Section 3 uses these labels for all evaluation; if the human labels are noisy, all comparisons are affected.
  • domain assumption LLM/LRM outputs following the Gemba-MQM/TQLite prompt can be parsed into reliable error spans and MQM scores.
    Section 4 and Appendix A rely on heuristic parsers; the paper notes parsing changes can swing metrics even with fixed model outputs.
  • domain assumption The public corpora used for synthetic data (opus-100, europarl, etc.) are representative of translation evaluation distributions.
    Section 6.1 constructs 3M training pairs from these corpora; if the distribution differs from deployment data, distillation quality is limited.
  • ad hoc to paper High jury agreement implies accurate MQM annotations (the meta-judge correctly merges and cleans them).
    Section 5 assumes Perfect-Agreement segments are strong training candidates; this proxy is not validated against human labels on the training data.
  • domain assumption The student models (Gemma-3-4B/12B) have sufficient capacity to learn MQM evaluation from the synthetic labels.
    Section 6 relies on the chosen architectures; one of them (4B) did not improve, suggesting capacity and data interact with the assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TQLite: Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation." pith.science (2026). https://pith.science/paper/FZZHQ6JT

@misc{pith2026260802975,
  author       = {Pith},
  title        = {Pith review of: TQLite: Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FZZHQ6JT}},
  note         = {Machine review of arXiv:2608.02975}
}
read the original abstract

Large language models (LLMs) have demonstrated impressive performance in MQM-based translation quality (TQ) evaluation, and recent advances in large reasoning models (LRMs) promise even greater improvements. However, both LLMs and LRMs are computationally expensive to deploy at scale, while small language models (SLMs)---though much more efficient---struggle with the complex reasoning required for evaluation tasks. In this work, we present an extensive empirical study benchmarking SLMs, LLMs, and LRMs across a wide range of TQ evaluation setups, providing a comprehensive view of the current landscape and establishing best practices. To address the scalability challenge, we introduce TQLite, a novel distillation framework that enables SLMs to approach the MQM evaluation performance of the best LRM-based evaluators. Our approach leverages a multi-LRM jury to generate high-quality synthetic training data via practical data curation techniques and aggregation of evaluation responses across a diverse panel of models. Our results demonstrate that SLMs trained via TQLite achieve strong MQM evaluation performance that far exceeds off-the-shelf evaluation capabilities of standard SLMs, offering a scalable and cost-effective alternative to LLM- and LRM-based evaluators.

Figures

Figures reproduced from arXiv: 2608.02975 by the authors.

Figure 1
Figure 1. Our TQLite framework for distilling a multi￾LRM jury into an SLM for efficient TQ evaluation. Lavie, 2005) and tend to correlate poorly with hu￾man judgments (Stiennon et al., 2020), leading to the popularization of reference-free and open￾ended evaluation techniques like LLM-as-a-Judge (Zheng et al., 2023). The concept of LLM-as-a-Judge is simple: we just prompt a powerful LLM to evaluate a segment of text. Usually… view at source ↗
Figure 2
Figure 2. System-level (aggregate) and segment-level [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Aggregate system and segment-level accuracy [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Distillation Scaling - Change in average segment-level accuracy with increase in number of train￾ing samples for Gemma-3-12b-it. above, we train a smaller open-source language model to replicate the translation quality evalua￾tion capabilities of the LRM jury. In our e…
Figure 5
Figure 5. Figure 5: Impact of LoRA Rank (number of train￾able parameters) on average segment-level accuracy for Gemma-3-12b-it trained with 20,000 training samples [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Impact of changing data quality through incor [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Inference time to performance tradeoff. ness. Impact of training samples. In [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Our TQLite prompt, which is a slightly modi￾fied version of the prompt used by Gemba-MQM (Kocmi and Federmann, 2023a). • Temperature = 0.7 • TopP = 0.8 • TopK = 20 • MinP = 0 Similarly, when these models are used with think￾ing enabled in [PITH_FULL_IMAGE:figures/full…
Figure 9
Figure 9. Figure 9: Our TQLite-ST prompt, which modifies the TQLite prompt to fully reside in a single chat turn. explanation from standard LLMs along with their evaluation: Zero-Shot CoT Instruction Let’s think step-by-step. The source text and translation that you will evaluate are prov…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

91 extracted references · 17 canonical work pages

  1. [1]

    2024 , url =

    Gemini Team , title =. 2024 , url =

    2024

  2. [2]

    2025 , url =

    OpenAI , title =. 2025 , url =

    2025

  3. [3]

    Advances in neural information processing systems , volume=

    Learning to summarize with human feedback , author=. Advances in neural information processing systems , volume=

  4. [6]

    2025 , url =

    Meta , title =. 2025 , url =

    2025

  5. [7]

    Advances in neural information processing systems , volume=

    Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in neural information processing systems , volume=

  6. [8]

    2023 , publisher=

    Error analysis prompting enables human-like translation evaluation in large language models: A case study on chatgpt , author=. 2023 , publisher=

    2023

  7. [9]

    The Twelfth International Conference on Learning Representations , year=

    Prometheus: Inducing fine-grained evaluation capability in language models , author=. The Twelfth International Conference on Learning Representations , year=

  8. [17]

    Alpacaeval: An automatic evaluator of instruction-following models , author=

Show all 91 references
  1. [19]

    2023 , url =

    Vicuna Team , title =. 2023 , url =

    2023

  2. [20]

    Proceedings of the Sixth Conference on Machine Translation , pages=

    Are references really needed? unbabel-IST 2021 submission for the metrics shared task , author=. Proceedings of the Sixth Conference on Machine Translation , pages=

    2021

  3. [22]

    Proceedings of the Seventh Conference on Machine Translation (WMT) , pages=

    COMET-22: Unbabel-IST 2022 submission for the metrics shared task , author=. Proceedings of the Seventh Conference on Machine Translation (WMT) , pages=

    2022

  4. [23]

    CoRR , year=

    Ties Matter: Modifying Kendall's Tau for Modern Metric Meta-Evaluation , author=. CoRR , year=

  5. [26]

    Multidimensional quality metrics (MQM): A framework for declaring and describing translation quality metrics , author=. Tradum

  6. [27]

    Transactions of the Association for Computational Linguistics , volume=

    Experts, errors, and context: A large-scale study of human evaluation for machine translation , author=. Transactions of the Association for Computational Linguistics , volume=. 2021 , publisher=

    2021

  7. [29]

    Proceedings of the Seventh Conference on Machine Translation (WMT) , pages=

    Findings of the 2022 conference on machine translation (WMT22) , author=. Proceedings of the Seventh Conference on Machine Translation (WMT) , pages=

    2022

  8. [30]

    Proceedings of the Seventh Conference on Machine Translation (WMT) , pages=

    Results of WMT22 metrics shared task: Stop using BLEU--neural metrics are better and more robust , author=. Proceedings of the Seventh Conference on Machine Translation (WMT) , pages=

  9. [31]

    Advances in neural information processing systems , volume=

    Large language models are zero-shot reasoners , author=. Advances in neural information processing systems , volume=

  10. [33]

    arXiv e-prints , pages=

    The llama 3 herd of models , author=. arXiv e-prints , pages=

  11. [51]

    Proceedings of the Ninth Conference on Machine Translation , pages=

    Are LLMs breaking MT metrics? results of the WMT24 metrics shared task , author=. Proceedings of the Ninth Conference on Machine Translation , pages=

  12. [55]

    Transactions of the Association for Computational Linguistics , volume=

    xcomet: Transparent machine translation evaluation through fine-grained error detection , author=. Transactions of the Association for Computational Linguistics , volume=. 2024 , publisher=

    2024

  13. [56]

    Advances in neural information processing systems , volume=

    Language models are few-shot learners , author=. Advances in neural information processing systems , volume=

  14. [57]

    Advances in neural information processing systems , volume=

    Judging llm-as-a-judge with mt-bench and chatbot arena , author=. Advances in neural information processing systems , volume=

  15. [58]

    Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages=

    Bleu: a method for automatic evaluation of machine translation , author=. Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages=

  16. [59]

    Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization , pages=

    METEOR: An automatic metric for MT evaluation with improved correlation with human judgments , author=. Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization , pages=

  17. [60]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  18. [61]

    Publications Manual , year = "1983", publisher =

    1983

  19. [62]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    1981

  20. [63]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of. 2007 , url=

    2007

  21. [64]

    Dan Gusfield , title =. 1997

    1997

  22. [65]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  23. [66]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =. 2005 , url=

    2005

  24. [67]

    and Tukey, John W

    Cooley, James W. and Tukey, John W. , journal=. An algorithm for the machine calculation of complex. 1965 , url=

    1965

  25. [68]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    2023 arXiv

  26. [69]

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv preprint arXiv:2305.10403

    2023 arXiv

  27. [70]

    Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65--72

    2005

  28. [71]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  29. [72]

    Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937

    2023 arXiv

  30. [73]

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic c...

    2025 arXiv

  31. [74]

    John Dang, Shivalika Singh, Daniel D'souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, et al. 2024. Aya expanse: Combining research breakthroughs for a new multilingual frontier. arXiv preprint arXiv:2412.04261

    2024 arXiv

  32. [75]

    Daniel Deutsch, George F Foster, and Markus Freitag. 2023. Ties matter: Modifying kendall's tau for modern metric meta-evaluation. CoRR

    2023

  33. [76]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv e-prints, pages arXiv--2407

    2024

  34. [77]

    Sicheng Feng, Gongfan Fang, Xinyin Ma, and Xinchao Wang. 2025. Efficient reasoning models: A survey. arXiv preprint arXiv:2504.10903

    2025

  35. [78]

    Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, Andr \'e FT Martins, Graham Neubig, Ankush Garg, Jonathan H Clark, Markus Freitag, and Orhan Firat. 2023. The devil is in the errors: Leveraging large language models for fine-grained machine translation evalua...

    2023 arXiv

  36. [79]

    Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460--1474

    2021

  37. [80]

    Markus Freitag, Nitika Mathur, Daniel Deutsch, Chi-Kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Frederic Blain, Tom Kocmi, Jiayi Wang, et al. 2024. Are llms breaking mt metrics? results of the wmt24 metrics shared task. In Proceedings of the Ninth Conference on ...

    2024

  38. [81]

    Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and Andr \'e FT Martins. 2022. Results of wmt22 metrics shared task: Stop using bleu--neural metrics are better and more robust. In Proceedings o...

    2022

  39. [82]

    Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166

    2023 arXiv

  40. [83]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594

    2024 arXiv

  41. [84]

    Nuno M Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and Andr \'e FT Martins. 2024. xcomet: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12:979--995

    2024

  42. [85]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  43. [86]

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531

    2015 arXiv

  44. [87]

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556

    2022 arXiv

  45. [88]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  46. [89]

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720

    2024 arXiv

  47. [90]

    Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. 2024. Metricx-24: The google submission to the wmt 2024 metrics shared task. arXiv preprint arXiv:2410.03983

    2024 arXiv

  48. [91]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361

    2020 arXiv

  49. [92]

    Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. 2023. Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth International Conference on Learning Repr...

    2023

  50. [93]

    Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. Prometheus 2: An open source language model specialized in evaluating other language models. arXiv preprint arXiv:2405.01535

    2024 arXiv

  51. [94]

    Tom Kocmi, Rachel Bawden, Ond r ej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, et al. 2022. Findings of the 2022 conference on machine translation (wmt22). In Proceedings of the Seventh Conference on...

    2022

  52. [95]

    Tom Kocmi and Christian Federmann. 2023 a . Gemba-mqm: Detecting translation quality error spans with gpt-4. arXiv preprint arXiv:2310.13988

    2023 arXiv

  53. [96]

    Tom Kocmi and Christian Federmann. 2023 b . Large language models are state-of-the-art evaluators of translation quality. arXiv preprint arXiv:2302.14520

    2023 arXiv

  54. [97]

    Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. arXiv preprint arXiv:2107.10821

    2021 arXiv

  55. [98]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199--22213

    2022

  56. [99]

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. 2024. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124

    2024 arXiv

  57. [100]

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models

    2023

  58. [101]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634

    2023 arXiv

  59. [102]

    Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014. Multidimensional quality metrics (mqm): A framework for declaring and describing translation quality metrics. Tradum \`a tica , (12):0455--463

    2014

  60. [103]

    Qingyu Lu, Liang Ding, Kanjian Zhang, Jinxia Zhang, and Dacheng Tao. 2024. Mqm-ape: Toward high-quality error annotation predictors with automatic post-editing in llm translation evaluators. arXiv preprint arXiv:2409.14335

    2024 arXiv

  61. [104]

    Qingyu Lu, Baopu Qiu, Liang Ding, Liping Xie, and Dacheng Tao. 2023. Error analysis prompting enables human-like translation evaluation in large language models: A case study on chatgpt

    2023

  62. [105]

    Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. Tangled up in bleu: Reevaluating the evaluation of automatic machine translation evaluation metrics. arXiv preprint arXiv:2006.06264

    2020 arXiv

  63. [106]

    Meta. 2025. https://ai.meta.com/blog/llama-4-multimodal-intelligence/ The llama 4 herd: The beginning of a new era of natively multimodal ai innovation . Accessed: 2025-08-12

    2025

  64. [107]

    OpenAI. 2025. https://openai.com/index/introducing-o3-and-o4-mini/ Introducing openai o3 and o4-mini . Accessed: 2025-08-12

    2025

  65. [108]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

    2002

  66. [109]

    Ricardo Rei, Jos \'e GC De Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and Andr \'e FT Martins. 2022 a . Comet-22: Unbabel-ist 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine...

    2022

  67. [110]

    Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan Van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, Andr \'e FT Martins, and Alon Lavie. 2021. Are references really needed? unbabel-ist 2021 submission for the metrics shared task. In Proceedings of the Sixth Conference ...

    2021

  68. [111]

    Ricardo Rei, Marcos Treviso, Nuno M Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, Jos \'e GC De Souza, Taisiya Glushkova, Duarte M Alves, Alon Lavie, et al. 2022 b . Cometkiwi: Ist-unbabel 2022 submission for the quality estimation shared task. arXiv preprint ar...

    2022 arXiv

  69. [112]

    Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text generation. arXiv preprint arXiv:2004.04696

    2020 arXiv

  70. [113]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    2024 arXiv

  71. [114]

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008--3021

    2020

  72. [115]

    Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, and Yun-Nung Chen. 2024. Let me speak freely? a study on the impact of format restrictions on performance of large language models. arXiv preprint arXiv:2408.02442

    2024 arXiv

  73. [116]

    Gemini Team. 2024. https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/ Introducing gemini 2.0: our new ai model for the agentic era . Accessed: 2025-08-12

    2024

  74. [117]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram \'e , Morgane Rivi \`e re, et al. 2025 a . Gemma 3 technical report. arXiv preprint arXiv:2503.19786

    2025 arXiv

  75. [118]

    Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. 2025 b . Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534

    2025 arXiv

  76. [119]

    Vicuna Team. 2023. https://lmsys.org/blog/2023-03-30-vicuna/ Vicuna: An open-source chatbot impressing gpt-4 with 90\ Accessed: 2025-08-12

    2023

  77. [120]

    Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023 a . Is chatgpt a good nlg evaluator? a preliminary study. arXiv preprint arXiv:2303.04048

    2023 arXiv

  78. [121]

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023 b . Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926

    2023 arXiv

  79. [122]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  80. [123]

    Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. 2024. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116

    2024 arXiv

  81. [124]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388

    2025 arXiv

  82. [125]

    Zihan Yu, Liang He, Zhen Wu, Xinyu Dai, and Jiajun Chen. 2023. Towards better chain-of-thought prompting strategies: A survey. arXiv preprint arXiv:2310.04959

    2023 arXiv

  83. [126]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595--46623

    2023

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.