Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Specialist LLM raters beat the top MT metric by 54% to 119%.

desk verdict Strong controlled study of test-set-specific ICL for LLM-based MT evaluation, but the headline SOTA claim over zero-shot XCOMET needs a same-data baseline to be fair. read the letter →

arxiv 2411.15387 v2 pith:HDB5SHCW submitted 2024-11-23 cs.CL

Mara Finkelstein , Dan Deutsch , Parker Riley , Juraj Juraska , Geza Kovacs , Markus Freitag This is my paper · ORCID
classification cs.CL
keywords LLM-as-a-JudgeAutoratermachinetranslationevaluationMQMin-contextlearningtest-setspecializationpseudo-SxSratingcollectionqualityestimation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that, when evaluating a machine translation system on a fixed canonical test set, the most effective evaluator is not a general-purpose LLM judge but one specialized to that test set through in-context demonstrations built from historical human ratings. For each source segment, the prompt shows the model the human MQM error annotations of other systems' translations of that same segment, collected by a single rater; the resulting Specialist AutoMQM metric improves character-level F1 by 54% on WMT'23 and 119% on WMT'24 over the previous state-of-the-art XCOMET. Ablations attribute the gains to the same-source demonstrations rather than to extra demonstrations or rater matching alone, and the method is robust across three LLM backbones and extends to direct score prediction. The practical appeal is that it upgrades automatic evaluation without finetuning or new per-system human annotation, at the cost of a one-off pseudo-SxS rating effort per benchmark.

What carries the argument

The central mechanism is the Specialist method's per-example construction of in-context learning (ICL) examples: for each source segment $x_i$ in the test set, the prompt for evaluating a system $M^*$ includes the historical MQM ratings (error spans with severities and categories) for $N$ other systems' translations of the same $x_i$, with the pseudo-SxS constraint that those ratings were produced by a single fixed rater (Algorithm 1). Each test example therefore gets its own unique demonstration set, which distinguishes the method from fixed-demonstration baselines such as GEMBA-MQM. The demonstrations teach the model both which errors to flag and which spans to leave alone; the authors verify this with a 'Parrot' baseline that copies matching spans from ICL examples, which reaches only about half the Specialist's F1, and with an abstention analysis showing the Specialist predicts fewer copied errors than the shuffled baseline does.

What would settle it

A decisive check: on a pseudo-SxS-rated test set, compare Specialist AutoMQM with the fixed-different-source baseline (same ICL examples and same rater, but demonstrations drawn from a different source segment). The paper reports 51.59 vs 27.06 F1 on WMT'23; if on another such set this gap collapses to near zero, the same-source mechanism is not the driver of the result. The same test applies in reverse: if a non-pseudo-SxS test set still yields the full Specialist gain, then the pseudo-SxS requirement is not load-bearing.

Watch

Extended reading notes

Core claim

The central claim is that an LLM-based quality metric, which the paper calls AutoMQM, can be made state of the art for a fixed MT test set by prompting the LLM with historical human MQM error annotations of other systems' translations of the same source sentence, provided all those annotations come from a single rater per source (the pseudo-SxS constraint). Under hold-one-out evaluation on WMT'23 and WMT'24, this Specialist AutoMQM reaches character-level F1 of 51.59 and 35.59, respectively, versus 33.50 and 16.23 for XCOMET-XXL-QE. The paper shows that the same-source ingredient is decisive: using the same ICL examples but shuffled across sources, or drawn from a fixed different source with the same rater, does not beat XCOMET. It also claims the metric specializes to the rater as well as the test set, because replacing same-rater ICL examples with a different rater's ratings for the same source drops performance to the shuffled-baseline level, and the metric matches human inter-annotator agreement when the ICL rater and test-set rater differ.

Load-bearing premise

The method presupposes that the test set already carries human MQM ratings for at least three other translation systems, and that every translation of a given source segment was rated by one fixed rater; where that collection protocol is missing, such as WMT'24 en→es, the Specialist gain shrinks to a few F1 points.

Editorial extensions

If this is right

  • On the WMT'23 and WMT'24 MT test sets, a prompted LLM with no finetuning can outperform the previous state-of-the-art finetuned span-level quality-estimation metric XCOMET by 54% and 119% relative F1.
  • A one-time investment of pseudo-SxS human MQM ratings for a few systems makes a test set reusable: new systems can then be evaluated accurately without additional human annotation.
  • The method generalizes across LLM backbones (Gemini 1.5 Pro, GPT-4o, Claude 3.5 Sonnet), which suggests the gains are not an artifact of a single model.
  • The same-source demonstrations are the decisive ingredient: the 'Fixed, different source' and 'Shuffled sources' baselines do not reproduce the gains, so the effect is not merely extra ICL examples or rater matching.
  • The pattern extends to direct assessment (a 0-100 score prediction task), where the Specialist Scorer beats or matches the state-of-the-art MetricX-24 on most tested language pairs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the Specialist route is applied to other NLG evaluation benchmarks (summarization, code generation, instruction following), the pseudo-SxS rating protocol is likely the make-or-break requirement: the paper's own MT-Bench experiment failed to beat its shuffled baseline precisely because ratings were not collected that way.
  • The rater-specialization result suggests that a large share of LLM-as-a-Judge performance may be implicit rater imitation; meta-evaluating against a single rater's ground truth can therefore understate a metric that genuinely generalizes across raters.
  • A practical testable extension: on a new test set with pseudo-SxS ratings, only three historical systems should suffice for Specialist AutoMQM to beat XCOMET; if the margin is instead within shuffle noise, the protocol has not been satisfied.
  • The method suggests a route toward personalized judges: metrics calibrated to one rater's error behavior, which could predict how that particular rater would judge new systems.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Specialist, a method that specializes a prompted LLM-based autorater to a fixed test set by constructing per-example in-context learning (ICL) demonstrations from historical human MQM ratings of other translation systems on the same source segment, with the pseudo-SxS condition that a single rater annotated all translations of that source. The LLM then evaluates a held-out system's translation on that segment using those demonstrations. On WMT'23 and WMT'24, the authors report 54% and 119% relative character-level F1 improvements over XCOMET-XXL-QE. They support the central claim with extensive controls: shuffled-source ICL, fixed-different-source ICL, ICL-size scaling, a Parrot baseline that copies spans from demonstrations, filtering of exact-match demonstration errors, cross-rater rounds, multiple LLM backbones, and a transfer experiment to scalar score prediction.

Significance. If properly framed, the paper is a useful and carefully controlled contribution to transductive, test-set-specialized evaluation. The hold-one-out prompting protocol in Algorithm 1 is sound: the evaluated system's own ratings are never used as demonstrations, so the reported gains are not a hidden circularity. The controlled comparisons—Shuffled sources, Fixed different source, Parrot model, filtered ICL, and Round2-rater ICL—are strong evidence that same-source, same-rater demonstrations are the active ingredient rather than simply the number of ICL examples or naive copying. The paper's explicit accounting of rater variability is a further strength. However, the headline comparison to XCOMET-XXL-QE compares a transductive method that sees historical test-set labels with an inductive metric that does not; the 'state-of-the-art' claim is therefore overstated unless a baseline that consumes the same held-out ratings is added. The strict pseudo-SxS data requirement is also openly acknowledged in the Limitations, and it substantially narrows the practical applicability claimed in the introduction.

major comments (3)
  1. [Section 5, Table 1 (rows 1a vs 2a); Abstract] The headline 54%/119% F1 improvement over XCOMET-XXL-QE is not a same-setting comparison. Under Algorithm 1, Specialist receives, for every test example, the human MQM annotations of all other systems on the same source and from the same rater, whereas XCOMET-XXL-QE is applied without any of those test-set labels. The reported gain therefore conflates the paper's actual contribution—same-source ICL demonstrations, as supported by the Shuffled and Fixed-different-source controls—with the much weaker observation that access to historical pseudo-SxS ratings helps span prediction. To support the 'state-of-the-art' claim, the paper should add a strong baseline that consumes the same held-out ratings, for example XCOMET fine-tuned on the other systems' WMT'23/24 MQM labels, or a supervised span tagger trained on those labels. Without such a baseline, the abstract and Section 5 should be rephrased as improvements over zero-shot XCOMET in a transductive setting.
  2. [Algorithm 1, Section 4.1, Limitations] The method requires that the exact test set already have MQM ratings from at least three other translation systems, collected pseudo-SxS so that a single rater rated all translations of each source. The paper's own Table 5 shows that using ICL examples from a different rating round—even for the same source—drops F1 to approximately the shuffled baseline level, and the Limitations section states that no commonly used public benchmark follows the pseudo-SxS collection procedure and that an MT-Bench attempt showed no gain. This is a load-bearing scope limitation: it narrows the introductory claim that evaluation in practice is performed on fixed canonical test sets to the much more specific situation of WMT-style pseudo-SxS MQM collections. The paper should state precisely which existing test sets satisfy the prerequisites and, given that the method is being proposed as a general approach, discuss explicitly whether the contribution is also a recommendation for how future test sets should be collected.
  3. [Table 13, WMT'24 en→es row] The Shuffled-sources row for en→es reports F1=26.12, precision=19.67, recall=38.84, which are exactly the same values as the en→de row above it. Because the table's note and the text in Section 5 use the en→es result to conclude that gains are smaller when pseudo-SxS is not enforced, this duplicate row must be corrected or explained. If the en→es values are a typographical error, the corrected numbers could affect the interpretation of the non-pseudo-SxS condition.
minor comments (5)
  1. [Section 5, first paragraph] The sentence 'also note that the Shuffled sources baseline already performs on par ... and and outperforms XCOMET' contains a duplicated 'and'.
  2. [Appendix A, Figure 6 prompt] The direct-assessment prompt says 'Perfect meaning nd grammar'; this should read 'and grammar'.
  3. [Appendix B.1.2] The text says 'difference-source ICL examples'; this should be 'different-source'.
  4. [Algorithm 1] The notation is slightly inconsistent: line 5 initializes RM* to an empty list, line 14 says 'Append Ri M* to RM*', but the final 'Return RM*' is fine; consider adding an index to the append for clarity, e.g., RM*[i].
  5. [Figure 4] The cross-rater matrices would be easier to read with a colorbar and a short caption explaining that higher values on the diagonal indicate rater specialization; the current caption describes the axes but not the color scale.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluated system's own ratings are held out, and the same-source demonstrations are the method being tested, not a hidden reuse of the target labels.

full rationale

The central claim is that per-example, same-source historical MQM ratings, provided as ICL demonstrations, improve fine-grained MT evaluation. The load-bearing comparisons are not circular. In Algorithm 1, for every evaluated system M*, the prompt excludes that system's ratings: the ICL examples are the ratings of N other systems, and the prediction is evaluated against the held-out rating of M* (Section 4.3: 'for whichever system we are evaluating, we exclude that system's ratings from the ICL examples and prompt with the ratings from the remaining systems'). The target label is therefore never an input to the model, so the reported F1 scores are genuine predictions, not refits of the ground truth. The paper's positive controls are also empirical rather than definitional: the 'Shuffled sources' and 'Fixed, different source' baselines (Section 4.2) use the same global rating pool and the same rater constraint but break the same-source correspondence, and the 'Parrot' baseline (Section 5.4.2) directly quantifies how much of the gain could be explained by copying demonstration spans; Specialist AutoMQM substantially exceeds the Parrot's F1 (45.7 vs 27.6 en->de; 57.5 vs 36.5 zh->en), showing the result is not forced by the demonstration content. The rater-specialization result (Section 5.5) is similarly empirical: switching from Round1 to Round2 ICL examples for the same sources drops F1 to the shuffled-baseline level, and the en->es WMT'24 language pair, which was not collected pseudo-SxS (Section 4.1, Table 13), shows a much smaller Specialist gain. That is evidence that the rater-matching property is a genuine empirical condition, not an assumption smuggled in by construction. The paper does rely on several self-citations (e.g., Riley et al. 2024 for the value of pseudo-SxS collection, Deutsch et al. 2023 for the Acc23 meta-evaluation metric), but none of these is invoked as a uniqueness theorem or as the sole justification for the method's central premise; the paper's own ablations independently demonstrate the pseudo-SxS requirement. The headline comparison against XCOMET is protocol-asymmetric—XCOMET is applied zero-shot without access to the historical ratings that Specialist AutoMQM consumes—so the 54% and 119% relative gains conflate the method's mechanism with the availability of test-set labels.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No numeric free parameters are fitted; the only choices are prompt design and ICL construction, which are fixed by the method. The central claim rests on data-availability assumptions: same-test-set pseudo-SxS MQM ratings from multiple systems and an LLM that can exploit same-source demonstrations. These are not circular because the evaluated system's own ratings are held out.

assumptions (5)
  • domain assumption Historical MQM ratings for at least 3 other translation systems on the exact test set are available and usable as ICL demonstrations.
    Algorithm 1 requires a set R of ratings on X for N systems; Section 5.3 shows 3 systems suffice. Without this data the method cannot be instantiated.
  • domain assumption Ratings are collected in pseudo-SxS fashion, so a fixed rater rates all translations of each source and that same rater's labels are the evaluation ground truth.
    Section 4.1 and the Limitations state the pseudo-SxS constraint is required. Table 5 and the WMT'24 en->es results show non-pseudo-SxS data yields much smaller gains.
  • domain assumption The prompted LLM (Gemini 1.5 Pro, GPT-4o, Claude 3.5 Sonnet) is a competent MQM span annotator and is not contaminated by WMT test set exposure in a way that inflates results.
    The method assumes LLM in-context learning from demonstrations produces reliable error spans. The paper does not test for test-set contamination.
  • domain assumption Character-level F1 with 0.5 partial credit for severity mismatch is the right meta-evaluation for error span prediction.
    Section 4.4 adopts the WMT'23 QE Shared Task span F1; this choice shapes all reported numbers.
  • domain assumption Hold-one-out evaluation, which removes the evaluated system's ratings from ICL examples, approximates how the metric will perform on a new system.
    Section 4.3 uses hold-one-out prompting to meta-evaluate; deployment for an unseen system assumes the same behavior extends.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set." pith.science (2026). https://pith.science/paper/HDB5SHCW

@misc{pith2026241115387,
  author       = {Pith},
  title        = {Pith review of: From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HDB5SHCW}},
  note         = {Machine review of arXiv:2411.15387}
}
read the original abstract

As LLMs continue to become more powerful and versatile, human evaluation has quickly become intractable at scale and reliance on automatic metrics has become the norm. Recently, it has been shown that LLMs are themselves state-of-the-art evaluators for many tasks. These Autoraters are typically designed so that they generalize to new systems and test sets. In practice, however, evaluation is performed on a small set of fixed, canonical test sets, which are carefully curated to measure certain capabilities of interest and are not changed frequently. In this work, we design a method which specializes a prompted Autorater to a given test set, by leveraging historical ratings on the test set to construct in-context learning (ICL) examples. We evaluate our Specialist method on the task of fine-grained machine translation evaluation, and show that it dramatically outperforms the state-of-the-art XCOMET metric by 54% and 119% on the WMT'23 and WMT'24 test sets, respectively. We perform extensive analyses to understand the representations learned by our Specialist metrics, and how variability in rater behavior affects their performance. We also verify the generalizability and robustness of our Specialist method for designing automatic metrics across different numbers of ICL examples, LLM backbones, systems to evaluate, and evaluation tasks.

Figures

Figures reproduced from arXiv: 2411.15387 by the authors.

Figure 1
Figure 1. Illustration of the Specialist method, compared against the Fixed, different source baseline, for prompting an LLM-based Autorater. Both methods (i) construct a unique set of demonstrations (i.e., ICL examples) for every test set example, consisting of historical ratings from different system outputs for some fixed source, and (ii) provide demonstrations from the same rater as the test rating ground truth. The diffe… view at source ↗
Figure 2
Figure 2. Specialized AutoMQM performance per translation system. The Champion (row 2a) and Shuffled baseline (row 1c) models from [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Specialist AutoMQM performance (“Champion + Filter” setting; [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Cross-rater performance of AutoMQM and human annotators, computed using the extension to the (round 1) WMT’23 zh→en test set, whereby 10% of the test set (18 source segments × 15 systems = 270 examples) was rated by all 8 raters. In Figure (a), Specialist AutoMQM is pr…
Figure 5
Figure 5. Figure 5: shows the AutoMQM prompt template, [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Direct Assessment prompt, with placeholders for {source language}, {source} (for both ICL examples and the test example), {target language}, {translation} (again, for both ICL examples and the test example), and {score} (for ICL examples only). B. Supplemental Results …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progress

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Automatic MT metrics often rank on par with or above human annotators when both are scored against MQM human judgments, raising doubts about whether progress in MT evaluation can still be measured.

Reference graph

Works this paper leans on

39 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Constitutional ai: Harmlessness from ai feedback

    Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022

    arXiv 2022

  4. [4]

    M., Kanojia, D., de Souza, J

    Blain, F., Zerva, C., Rei, R., Guerreiro, N. M., Kanojia, D., de Souza, J. G., Silva, B., Vaz, T., Jingxuan, Y., Azadi, F., et al. Findings of the wmt 2023 shared task on quality estimation. In Proceedings of the Eighth Conference on Machine Translation, pp.\ 629--653, 2023

    work page 2023

  5. [5]

    Callison-Burch, C., Koehn, P., Monz, C., Schroeder, J., and Fordyce, C. S. Proceedings of the third workshop on statistical machine translation. In Proceedings of the Third Workshop on Statistical Machine Translation, 2008

    work page 2008

  6. [6]

    "Seeing the Big through the Small": Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations?

    Chen, B., Wang, X., Peng, S., Litschko, R., Korhonen, A., and Plank, B. " seeing the big through the small": Can llms approximate human judgment distributions on nli from a few explanations? arXiv preprint arXiv:2406.17600, 2024

    work page Pith review arXiv 2024

  7. [7]

    Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration

    Deutsch, D., Foster, G., and Freitag, M. Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration. arXiv preprint arXiv:2305.14324, 2023

    arXiv 2023

  8. [8]

    F., Neubig, G., Garg, A., Clark, J

    Fernandes, P., Deutsch, D., Finkelstein, M., Riley, P., Martins, A. F., Neubig, G., Garg, A., Clark, J. H., Freitag, M., and Firat, O. The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation. arXiv preprint arXiv:2308.07286, 2023

    arXiv 2023

Show all 39 references
  1. [9]

    Experts, errors, and context: A large-scale study of human evaluation for machine translation

    Freitag, M., Foster, G., Grangier, D., Ratnakar, V., Tan, Q., and Macherey, W. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9: 0 1460--1474, 2021

    2021

  2. [10]

    Results of wmt23 metrics shared task: Metrics might be guilty but references are not innocent

    Freitag, M., Mathur, N., Lo, C.-k., Avramidis, E., Rei, R., Thompson, B., Kocmi, T., Blain, F., Deutsch, D., Stewart, C., et al. Results of wmt23 metrics shared task: Metrics might be guilty but references are not innocent. In Proceedings of the Eighth Conference on Machine Tr...

    2023

  3. [11]

    Are llms breaking mt metrics? results of the wmt24 metrics shared task

    Freitag, M., Mathur, N., Deutsch, D., Lo, C.-K., Avramidis, E., Rei, R., Thompson, B., Blain, F., Kocmi, T., Wang, J., et al. Are llms breaking mt metrics? results of the wmt24 metrics shared task. In Proceedings of the Ninth Conference on Machine Translation, pp.\ 47--81, 2024

    2024

  4. [12]

    Gemini: A family of highly capable multimodal models, 2024

    Gemini Team . Gemini: A family of highly capable multimodal models, 2024. URL https://arxiv.org/abs/2312.11805

    2024 arXiv

  5. [13]

    Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets

    Geva, M., Goldberg, Y., and Berant, J. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. arXiv preprint arXiv:1908.07898, 2019

    1908 arXiv

  6. [14]

    S., Omrani, A., and Dehghani, M

    Golazizian, P., Ziabari, A. S., Omrani, A., and Dehghani, M. Cost-efficient subjective task annotation and modeling through few-shot annotator adaptation. arXiv preprint arXiv:2402.14101, 2024

    2024 arXiv

  7. [15]

    M., Rei, R., van Stigt, D., Coheur, L., Colombo, P., and Martins, A

    Guerreiro, N. M., Rei, R., van Stigt, D., Coheur, L., Colombo, P., and Martins, A. F. xcomet: Transparent machine translation evaluation through fine-grained error detection. arXiv preprint arXiv:2310.10482, 2023

    2023 arXiv

  8. [16]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  9. [17]

    Metricx-24: The google submission to the wmt 2024 metrics shared task

    Juraska, J., Deutsch, D., Finkelstein, M., and Freitag, M. Metricx-24: The google submission to the wmt 2024 metrics shared task. arXiv preprint arXiv:2410.03983, 2024

    2024 arXiv

  10. [18]

    Kamoi, R., Das, S. S. S., Lou, R., Ahn, J. J., Zhao, Y., Lu, X., Zhang, N., Zhang, Y., Zhang, R. H., Vummanthala, S. R., et al. Evaluating llms at detecting errors in llm responses. arXiv preprint arXiv:2404.03602, 2024

    2024 arXiv

  11. [19]

    The perils of using mechanical turk to evaluate open-ended text generation

    Karpinska, M., Akoury, N., and Iyyer, M. The perils of using mechanical turk to evaluate open-ended text generation. arXiv preprint arXiv:2109.06835, 2021

    2021 arXiv

  12. [20]

    Prometheus: Inducing fine-grained evaluation capability in language models

    Kim, S., Shin, J., Cho, Y., Jang, J., Longpre, S., Lee, H., Yun, S., Shin, S., Kim, S., Thorne, J., et al. Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth International Conference on Learning Representations, 2023

    2023

  13. [21]

    Y., Shin, J., Welleck, S., Neubig, G., Lee, M., Lee, K., and Seo, M

    Kim, S., Suk, J., Longpre, S., Lin, B. Y., Shin, J., Welleck, S., Neubig, G., Lee, M., Lee, K., and Seo, M. Prometheus 2: An open source language model specialized in evaluating other language models. arXiv preprint arXiv:2405.01535, 2024

    2024 arXiv

  14. [22]

    and Federmann, C

    Kocmi, T. and Federmann, C. Gemba-mqm: Detecting translation quality error spans with gpt-4. arXiv preprint arXiv:2310.13988, 2023 a

    2023 arXiv

  15. [23]

    and Federmann, C

    Kocmi, T. and Federmann, C. Large language models are state-of-the-art evaluators of translation quality. arXiv preprint arXiv:2302.14520, 2023 b

    2023 arXiv

  16. [24]

    Longeval: Guidelines for human evaluation of faithfulness in long-form summarization

    Krishna, K., Bransom, E., Kuehl, B., Iyyer, M., Dasigi, P., Cohan, A., and Lo, K. Longeval: Guidelines for human evaluation of faithfulness in long-form summarization. arXiv preprint arXiv:2301.13298, 2023

    2023 arXiv

  17. [25]

    Generative judge for evaluating alignment

    Li, J., Sun, S., Yuan, W., Fan, R.-Z., Zhao, H., and Liu, P. Generative judge for evaluating alignment. arXiv preprint arXiv:2310.05470, 2023

    2023 arXiv

  18. [26]

    Holistic evaluation of language models

    Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022

    2022 arXiv

  19. [27]

    Multidimensional quality metrics (mqm): A framework for declaring and describing translation quality metrics

    Lommel, A., Uszkoreit, H., and Burchardt, A. Multidimensional quality metrics (mqm): A framework for declaring and describing translation quality metrics. Tradum \`a tica , 0 (12): 0 0455--463, 2014

    2014

  20. [28]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    2022

  21. [29]

    C., and Lavie, A

    Rei, R., Stewart, C., Farinha, A. C., and Lavie, A. Comet: A neural framework for mt evaluation. arXiv preprint arXiv:2009.09025, 2020

    2009 arXiv

  22. [30]

    Finding replicable human evaluations via stable ranking probability

    Riley, P., Deutsch, D., Foster, G., Ratnakar, V., Dabirmoghaddam, A., and Freitag, M. Finding replicable human evaluations via stable ranking probability. arXiv preprint arXiv:2404.01474, 2024

    2024 arXiv

  23. [31]

    Sellam, T., Das, D., and Parikh, A. P. Bleurt: Learning robust metrics for text generation. arXiv preprint arXiv:2004.04696, 2020

    2004 arXiv

  24. [32]

    A benchmark for learning to translate a new language from one grammar book

    Tanzer, G., Suzgun, M., Visser, E., Jurafsky, D., and Melas-Kyriazi, L. A benchmark for learning to translate a new language from one grammar book. arXiv preprint arXiv:2309.16575, 2023

    2023 arXiv

  25. [33]

    Foundational autoraters: Taming large language models for better automatic evaluation

    Vu, T., Krishna, K., Alzubi, S., Tar, C., Faruqui, M., and Sung, Y.-H. Foundational autoraters: Taming large language models for better automatic evaluation. arXiv preprint arXiv:2407.10817, 2024

    2024 arXiv

  26. [34]

    Y., Li, L., and Freitag, M

    Xu, W., Deutsch, D., Finkelstein, M., Juraska, J., Zhang, B., Liu, Z., Wang, W. Y., Li, L., and Freitag, M. LLMR efine: Pinpointing and refining large language models via fine-grained actionable feedback. In Duh, K., Gomez, H., and Bethard, S. (eds.), Findings of the Associati...

    2024 doi

  27. [35]

    Understanding in-context learning from repetitions

    Yan, J., Xu, J., Song, C., Wu, C., Li, Y., and Zhang, Y. Understanding in-context learning from repetitions. arXiv preprint arXiv:2310.00297, 2023

    2023 arXiv

  28. [36]

    H., Dadkhahi, H., Finkelstein, M., Trabelsi, F., Luo, J., and Freitag, M

    Zhang, L. H., Dadkhahi, H., Finkelstein, M., Trabelsi, F., Luo, J., and Freitag, M. Learning from others' mistakes: Finetuning machine translation models with span-level error annotations, 2024 a . URL https://arxiv.org/abs/2410.16509

    2024 arXiv

  29. [37]

    J., Wang, Z., Hwang, J

    Zhang, M. J., Wang, Z., Hwang, J. D., Dong, Y., Delalleau, O., Choi, Y., Choi, E., Ren, X., and Pyatkin, V. Diverging preferences: When do annotators disagree and do models know? arXiv preprint arXiv:2410.14632, 2024 b

    2024

  30. [38]

    P., et al

    Zheng, L., Chiang, W.-L., Sheng, Y., Li, T., Zhuang, S., Wu, Z., Zhuang, Y., Li, Z., Lin, Z., Xing, E. P., et al. Lmsys-chat-1m: A large-scale real-world llm conversation dataset. arXiv preprint arXiv:2309.11998, 2023 a

    2023 arXiv

  31. [39]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36: 0 46595--46623, 2023 b

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.