Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluation

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read M-MAD claims that an LLM-as-a-judge evaluator for machine translation can reach the accuracy of trained automatic metrics by splitting the MQM rubric into separate dimensions, having agents debate each dimension, and combining the debates…

desk verdict Useful matched-baseline study of dimension-split LLM MT evaluation, but the headline ranking is weakened by test-set configuration selection and a debate component that contributes almost nothing. read the letter →

arxiv 2412.20127 v3 pith:QKY4MEK3 submitted 2024-12-28 cs.CL cs.AI

Zhaopeng Feng , Jiayuan Su , Jiamei Zheng , Jiahan Ren , Yan Zhang , Jian Wu , Hongwei Wang , Zuozhu Liu This is my paper · ORCID
classification cs.CLcs.AI
keywords machinetranslationevaluationLLM-as-a-judgemulti-agentdebateMQMqualityestimationsegment-levelWMT23GPT-4omini
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes M-MAD, a three-stage framework that turns an LLM into a machine-translation evaluator. Instead of asking one model to apply the full MQM error rubric at once, it splits the rubric into four dimensions (accuracy, fluency, terminology, style), has pairs of agents debate the severity of errors in each dimension, and then has a judge merge the viewpoints into a final error list and MQM score. The claim is that this division and debate lets a cheap model like GPT-4o mini beat other LLM-as-a-judge methods and match trained reference-based metrics such as MetricX-23 and XCOMET on the WMT23 benchmark. If true, it would mean high-quality MT evaluation no longer requires training dedicated metric models or human-level annotation budgets, just orchestrated prompting.

What carries the argument

The load-bearing object is the three-stage M-MAD pipeline plus the MQM severity-weighted scoring formula. Stage 1 partitions the MQM rubric into $d=4$ dimensions (accuracy, fluency, style, terminology) with an independent few-shot evaluation agent per dimension. Stage 2 runs a two-agent pro-con debate on the severity of each detected error, with a consensus checker ending the debate early and a default rule that unresolved disagreements keep the supportive side's initial evaluation. Stage 3's judge agent merges dimension viewpoints, removes overlapping spans, and produces the final annotation set, from which the score is computed as $MQM_{score} = -5\,n_{major} - 1\,n_{minor}$. The mechanism that carries the argument is that decoupling removes the coupled-template bias of GEMBA-MQM-style prompts, and the severity-focused consensus debate corrects the over-severity bias that single-agent LLM judges show, as evidenced by the error-span prediction F1 (0.54 vs 0.37 for GEMBA-MQM) and the MQM score distribution that matches gold annotations.

What would settle it

Evaluate M-MAD with the exact ZH-EN-tuned settings on the WMT24 Metrics Shared Task (or any held-out set with MQM labels). If its meta-score no longer exceeds GEMBA-MQM, or falls more than one point below the XCOMET-QE ensemble, the reported margin is configuration-specific. A second, cheaper check: re-run the ZH-EN experiments with the debate topic fixed to "Category" instead of "Severity" and R=2; the paper's Table 6 predicts a drop of more than 0.14 in meta-score, so observing a smaller drop or a gain would refute the severity-debate mechanism.

Watch

Extended reading notes

Core claim

The central discovery is a "neural network in natural language form" — the paper's own analogy — in which the stages act as layers, agents as neurons, and their message exchanges as hidden states. The three stages are: first, decouple the MQM annotation guideline into four independent evaluation dimensions and have an initial agent produce fine-grained error annotations per dimension; second, run a two-agent pro-con debate on error severity within each dimension until consensus or three rounds, with a default toward minor severity; third, have a judge agent merge the dimension viewpoints, removing duplicate or overlapping error spans, and compute the final score with the weighted MQM formula ($w_{major}=5$, $w_{minor}=1$). The paper reports that this pipeline raises segment-level agreement with human judgments substantially over GEMBA-MQM and EAPrompt, and the overall meta-score on WMT23 ZH-EN reaches 0.808, ahead of every compared LLM-as-a-judge baseline and most learned metrics, trailing only the XCOMET-Ensemble among reference-based systems.

Load-bearing premise

The central premise is that the framework configuration tuned to perform best on the WMT23 ZH-EN test set — debating error severity, using consensus, with three rounds — also works for EN-DE and HE-EN and for future test sets; if that choice is noise, the reported improvements over baselines may not persist.

Editorial extensions

If this is right

  • Using the same pipeline, an LLM-as-a-judge evaluator beats all other prompting-only evaluators and several trained metrics without any training data, on three language pairs of WMT23.
  • Because M-MAD is reference-free and training-free, MT evaluation could be run on demand for any language pair an LLM can translate, without collecting human-annotated metric training sets.
  • The configuration findings — severity debates beat category debates, consensus beats review-style strategies, three rounds is optimal — give concrete design rules for multi-agent evaluators beyond MT.
  • The decoupling insight implies that any coupled MQM-style prompt inherits a detectable error-severity bias, which explains the gap between system-level and segment-level performance in prior LLM-as-a-judge methods.
  • If the improvements hold on fresh test sets, the main obstacle to LLM-as-a-judge metrics surpassing learned metrics becomes cost, not capability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The configuration was chosen on the WMT23 ZH-EN test set; a natural next check is whether the same settings (severity topic, consensus, R=3) win on WMT24, which would rule out selection-on-test-set as the source of the gains.
  • The dimension-decoupling idea is general: any MQM-style or rubric-based evaluation (e.g., summarization, dialogue) could be split per criterion with per-criterion debate, and the paper's evidence suggests the gains come from the decoupling itself, not from translation-specific prompts.
  • The paper's own limitation note implies a testable extension: heterogeneous debating groups mixing strong closed models and weak open models might push performance higher than the homogeneous GPT-4o mini groups used here, and would also test whether debate gains scale with total reasoning budget.
  • The high token cost of repeated debates suggests a practical threshold: for deployment, one could measure the cost-quality tradeoff of reducing rounds to 2 or using a cheaper debater in Stage 1 with the expensive model only in the debate stage.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes M-MAD, a three-stage LLM-based framework for machine translation evaluation. Stage 1 decomposes the MQM annotation guideline into four dimensions (accuracy, fluency, style, terminology); Stage 2 runs two-agent 'Pro-Con' debates within each dimension, with a consensus checker; Stage 3 synthesizes dimension-level viewpoints into a final MQM-style score. Using GPT-4o mini at temperature 0, the authors report WMT23 meta-evaluation results on ZH-EN, EN-DE, and HE-EN, claiming that M-MAD outperforms existing LLM-as-a-judge methods and is competitive with state-of-the-art learned automatic metrics, despite being training-free and reference-free. Detailed ablations identify dimension partition as the largest contributor, and appendix analyses discuss error-span prediction and cases of possibly mislabeled gold annotations.

Significance. If the reported results are robust, the paper would make a useful contribution: it shows that a prompt-based, training-free LLM judge can approach the segment-level performance of learned metrics such as MetricX-23 and XCOMET on WMT23, and it provides a systematic decomposition of the MQM task. The strengths include matched GPT-4o mini backends across LLM-as-a-judge baselines, use of the standard MTME evaluation tool, publicly available code and data, and a transparent ablation structure. However, the significance is tempered by the fact that key framework choices (debate topic, debate strategy, number of rounds) were selected by maximizing the same ZH-EN test-set meta-score that is then included in the headline results, and by the very small measured contribution of the multi-agent debate component itself.

major comments (4)
  1. [Section 4.3, Tables 6-7, Figure 3] The debate topic (Severity), debate strategy (Consensus), and maximum debate rounds (R=3) are selected by maximizing meta-evaluation scores on the WMT23 ZH-EN test set, and the same ZH-EN scores are then included in the Table 2 average and in the abstract claims. Table 6 shows large swings across topics (0.808 vs. 0.666 vs. 0.737), Table 7 shows smaller but still positive differences for Consensus, and Figure 3 selects R=3. This makes the reported ZH-EN result a maximum over searched configurations rather than the performance of a pre-specified method. The EN-DE results provide partial evidence of transfer, but the headline average still mixes the tuned ZH-EN result with the untuned EN-DE result. Please provide a held-out validation procedure (for example, tuning on one language pair and reporting only the other pair as the test set, or using WMT22 for configuration selection) and clearly separate tuned from untuned results.
  2. [Tables 3 and 7, Section 3.2] No significance estimates or confidence intervals are reported anywhere, and the margins that carry the framework-level claims are extremely small. The ablation in Table 3 shows that removing Stage 2 (the multi-agent debate) changes the ZH-EN meta score by only -0.006; Table 7 shows the Consensus strategy at 0.808 versus the no-debate baseline at 0.802. Because temperature is 0, these are deterministic outputs, but the underlying test set and configuration interaction are not quantified. A difference of 0.006 in a composite meta-score cannot be distinguished from noise without bootstrap or other uncertainty estimates. Please add such estimates, or explicitly identify which conclusions remain supported without them.
  3. [Appendix D, Table 9] The appendix states that 'M-MAD is also superior among all metrics in the He-En task,' but Table 9 shows M-MAD with a meta score of 0.787, below MetricX-23 (0.807), XCOMET-QE-Ensemble (0.789), and MaTESe (0.792), with segment-level Pearson also below several learned metrics. This factual contradiction undermines the claim of consistent cross-lingual superiority and should be corrected. It also matters for the abstract's 'competes with state-of-the-art reference-based automatic metrics' claim, since the HE-EN results are weaker than the ZH-EN/EN-DE results.
  4. [Table 3 and Section 2.2] The paper's title and central narrative emphasize the multi-agent debate component, but the decoupled multidimensional design (Stage 1) is what drives most of the improvement (-0.041 meta when removed), while removing Stage 2 changes the meta score by only -0.006. This is not a refutation of the empirical headline, but it is a mismatch between the framework's claimed mechanism and the evidence. Please report more granular results for the debate stage (for example, per-dimension accuracy, error-span F1 changes, or case-level agreement) so that the contribution of Stage 2 is characterized substantively rather than through a single near-zero delta.
minor comments (5)
  1. [Section 3.2] The text says 'In the reference-based setting, M-MAD surpasses COMETKiwi by 2.6% and MetricX-23-QE by 0.9%,' but COMETKiwi and MetricX-23-QE are reference-free metrics. The sentence should refer to the reference-free setting.
  2. [Table 1 caption] The caption says 'Metrics with gray background is reference-based.' The verb should agree, and it would help to clarify that the gray background is not visible in the text-only rendering.
  3. [Figure 13] The second Style few-shot example annotates the span 'merchants' in a translation that does not contain the word 'merchants.' This appears to be a copy-paste error and may degrade the few-shot prompt for the Style agent.
  4. [Section 4.4, Figure 3] The claim that system-level performance 'consistently peaks at round 3' is not supported by error bars or quantitative values in the figure. Please state the exact values and, if possible, add variability estimates.
  5. [Table 2 caption] The caption is ambiguous about the first numerical column; it should state explicitly that the META column is the average meta-evaluation score across ZH-EN and EN-DE, while the subsequent columns are per-language component scores.

Circularity Check

1 steps flagged · score 4.0 of 10

Method configuration is selected on the WMT23 ZH-EN test set and the same set is used in the headline meta-score, making the reported gain partly a selection artifact; no held-out validation or significance estimate is provided.

  1. fitted input called prediction [Section 4.4 (Performance Convergence as Debating Rounds Progress), Figure 3; configuration choices also made in Section 4.3, Tables 6-7; headline results in Section 3.2, Table 2.]
    "As shown in Figure 3, system-level performance fluctuates across rounds but consistently peaks at round 3."

    The specific M-MAD configuration—severity debate topic (Table 6), Consensus strategy (Table 7), and R=3 (Figure 3)—is selected by maximizing the WMT23 ZH-EN meta-evaluation score. The same ZH-EN score is then included in the headline average (Table 2) and in the abstract claim that M-MAD 'outperforms all existing LLM-as-a-judge methods.' Consequently, the reported ZH-EN meta score is the maximum over the explored configurations, not the score of a pre-specified method; the 'improvement' over GEMBA-MQM and EAPrompt on ZH-EN is partly a selection artifact. No held-out split, standard error, or significance test is reported.

full rationale

M-MAD's method definition is not circular: the framework's output is compared against external human MQM labels using the standard WMT23 meta-evaluation protocol, and the MQM scoring formula (Eq. 1) is the same weighting used by the baselines, not derived from M-MAD's own outputs. I found no load-bearing self-citations, no imported uniqueness theorems, and no renaming of a known result as a new derivation. The one substantive circularity is empirical rather than definitional. In Sections 4.3-4.4 and Tables 6-7 and Figure 3, the authors select the debate topic (Severity), debating strategy (Consensus), and number of rounds (R=3) by maximizing meta-evaluation scores on the WMT23 ZH-EN test set. Those same ZH-EN numbers then enter Table 2 and the abstract's claim that M-MAD 'outperforms all existing LLM-as-a-judge methods.' Thus the ZH-EN result is the maximum of the explored configurations, not the outcome of a pre-specified system; the margin over GEMBA-MQM and EAPrompt on this pair is inflated by selection on the evaluation set. No held-out validation split, standard error, or significance test is reported, and with temperature fixed at 0 the only unquantified variance is exactly this test-set-and-configuration interaction. The EN-DE and HE-EN results were not used for configuration selection and provide genuinely independent, though weaker, evidence that the framework transfers; this prevents the entire central claim from reducing to a fit. In addition, the ablation in Table 3 shows that removing Stage 2 changes ZH-EN meta by only -0.006, so the named multi-agent debate component is responsible for little of the reported gain, underscoring the fragility of the headline margin. Overall score 4: partial circularity through test-set configuration selection, with independent cross-lingual evidence keeping the central claim from being wholly forced.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No new ontological entities are introduced; M-MAD is a procedural framework. The main ledger entries are hyperparameters (debate topic, strategy, round count, dimension count, agent count) and the domain assumptions of relying on WMT23 human MQM labels and the MTME protocol.

free parameters (5)
  • Debate topic = Severity
    Chosen by comparing Severity, Category, and Entirety on the WMT23 ZH-EN test set (Table 6), i.e., tuned on test data.
  • Debate strategy = Consensus
    Chosen by comparing Consensus, Deliberation, Interactive Review, and Consultancy Review on the WMT23 ZH-EN test set (Table 7), i.e., tuned on test data.
  • Maximum number of debate rounds R = 3
    Selected from a 1 to 5 round sweep on the WMT23 ZH-EN test set (Figure 3).
  • Number of evaluation dimensions d = 4
    Heuristic decomposition of MQM into Accuracy, Fluency, Style, and Terminology; non-translation and locale convention excluded to match GEMBA-MQM. Not tuned but a hand-made modeling choice.
  • Number of debaters per dimension n = 2
    No ablation is reported for the number of agents; this is a fixed design choice.
assumptions (4)
  • domain assumption WMT23 MQM human annotations are an accurate gold standard for translation quality.
    Used as ground truth for meta-evaluation throughout Section 3; the paper itself notes in Section 4.5 and Appendix E that some gold annotations may be mislabeled, which weakens this assumption.
  • domain assumption The MTME meta-evaluation protocol correctly aggregates the four WMT 2023 scenarios into a meaningful comparison.
    Section 3.1 adopts the WMT 2023 shared task meta-evaluation, including system-level pairwise accuracy, system-level Pearson, segment-level Accuracy-t, and segment-level Pearson.
  • domain assumption LLM-written error annotations in the prescribed JSON format can be reliably counted and converted to MQM scores via the weighted formula.
    Stage 3 and Equation 1 compute MQMscore = -5*n_major - 1*n_minor; this assumes LLM error counts match the human notion of distinct errors.
  • domain assumption The string matching of LLM output spans to gold error spans provides a valid precision/recall estimate.
    Section 4.2 and Table 4 use span matching; the paper concedes this may inflate scores for GEMBA-MQM and EAPrompt but does not fully validate the matching procedure.

how reviews work

0 comments
Cite this review

Pith. "Pith review of M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluation." pith.science (2026). https://pith.science/paper/QKY4MEK3

@misc{pith2026241220127,
  author       = {Pith},
  title        = {Pith review of: M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QKY4MEK3}},
  note         = {Machine review of arXiv:2412.20127}
}
read the original abstract

Recent advancements in large language models (LLMs) have given rise to the LLM-as-a-judge paradigm, showcasing their potential to deliver human-like judgments. However, in the field of machine translation (MT) evaluation, current LLM-as-a-judge methods fall short of learned automatic metrics. In this paper, we propose Multidimensional Multi-Agent Debate (M-MAD), a systematic LLM-based multi-agent framework for advanced LLM-as-a-judge MT evaluation. Our findings demonstrate that M-MAD achieves significant advancements by (1) decoupling heuristic MQM criteria into distinct evaluation dimensions for fine-grained assessments; (2) employing multi-agent debates to harness the collaborative reasoning capabilities of LLMs; (3) synthesizing dimension-specific results into a final evaluation judgment to ensure robust and reliable outcomes. Comprehensive experiments show that M-MAD not only outperforms all existing LLM-as-a-judge methods but also competes with state-of-the-art reference-based automatic metrics, even when powered by a suboptimal model like GPT-4o mini. Detailed ablations and analysis highlight the superiority of our framework design, offering a fresh perspective for LLM-as-a-judge paradigm. Our code and data are publicly available at https://github.com/SU-JIAYUAN/M-MAD.

Figures

Figures reproduced from arXiv: 2412.20127 by the authors.

Figure 1
Figure 1. Comparison between Single-Agent (SA) MT Evaluation and Multidimensional Multi-Agent Debate [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. MQM score distribution for WMT 23 Metrics [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Performance with increased multi-agent debating rounds on WMT 23 ZH-EN. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: The general prompt for GEMBA-MQM, with the “(user)” and “(assistant)” section repeated for each [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Three examples used for GEMBA-MQM [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: The prompt for EAPrompt, demonstrated here with an example for EN-DE. [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: The prompt for M-MAD Stage 1, demonstrated here with an example for Accuracy agent. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: The prompt for M-MAD Stage 2, demonstrated here with an example for Accuracy agent. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: The prompt for M-MAD Stage 3. Chinese source: 工厂直销生产，欢迎代理批发！ English translation: Factory direct production, welcome agent wholesale! MQM annotations: {"annotations":[{"error_span": "Factory direct production", "category": "accuracy/mistranslation", "severity": "major"…
Figure 10
Figure 10. Figure 10: Our examples used in Stage 1 for the Accuracy agent in Zh-En few-shot prompting. [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Our examples used in Stage 1 for the Fluency agent in Zh-En few-shot prompting. [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Our examples used in Stage 1 for the Terminology agent in Zh-En few-shot prompting. [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Our examples used in Stage 1 for the Style agent in Zh-En few-shot prompting. [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: Our examples used in Stage 1 for the Accuracy agent in En-De few-shot prompting. [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]
Figure 15
Figure 15. Figure 15: Our examples used in Stage 1 for the Fluency agent in En-De few-shot prompting. [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]
Figure 16
Figure 16. Figure 16: Our examples used in Stage 1 for the Terminology agent in En-De few-shot prompting. [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]
Figure 17
Figure 17. Figure 17: Our examples used in Stage 1 for the Style agent in En-De few-shot prompting. [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TACTIC: Translation Agents with Cognitive-Theoretic Interactive Collaboration

    cs.CL 2025-06 conditional novelty 4.0 of 10

    TACTIC, a cognitive-inspired six-agent workflow, improves LLM translation quality over direct prompting on FLORES-200 and WMT24, with the best DeepSeek-V3 setup reaching 96.19 XCOMET on English-to-X.

Reference graph

Works this paper leans on

43 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Sweta Agrawal, Ant \'o nio Farinhas, Ricardo Rei, and Andre Martins. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.802 Can automatic metrics assess high-quality translations? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14491--14502, Miami, Florida, USA. Association for Computational Linguistics

    doi:10.18653/v1/2024.emnlp-main.802 2024

  4. [4]

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv preprint arXiv:2305.10403

    arXiv 2023

  5. [5]

    Aljoscha Burchardt. 2013. https://aclanthology.org/2013.tc-1.6 Multidimensional quality metrics: a flexible system for assessing translation quality . In Proceedings of Translating and the Computer 35, London, UK. Aslib

    2013

  6. [6]

    Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201

    arXiv 2023

  7. [7]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1--45

    2024

  8. [8]

    Cheng-Han Chiang and Hung-Yi Lee. 2023. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607--15631

    work page 2023

Show all 43 references
  1. [9]

    Daniel Deutsch, George Foster, and Markus Freitag. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.798 Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Proc...

    2023 doi

  2. [10]

    S \"o ren Dreano, Derek Molloy, and Noel Murphy. 2023. https://doi.org/10.18653/v1/2023.wmt-1.60 E mbed \_ L lama: Using LLM embeddings for the metrics shared task . In Proceedings of the Eighth Conference on Machine Translation, pages 738--745, Singapore. Association for Comp...

    2023 doi

  3. [11]

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. 2024. Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning

    2024

  4. [12]

    Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, Andr \'e Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat. 2023. https://doi.org/10.18653/v1/2023.wmt-1.100 The devil is in the errors: Leveraging large language models for f...

    2023 doi

  5. [13]

    Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. https://doi.org/10.1162/tacl_a_00437 Experts, errors, and context: A large-scale study of human evaluation for machine translation . Transactions of the Association for Comp...

    2021 doi

  6. [14]

    Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023. https://doi.org/10.18653/v1/2023.wmt-1.51 Results of ...

    2023 doi

  7. [15]

    Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and Andr \'e F. T. Martins. 2022. https://aclanthology.org/2022.wmt-1.2 Results of WMT 22 metrics shared task: Stop using BLEU -- neural metrics ...

    2022

  8. [16]

    Thamme Gowda, Tom Kocmi, and Marcin Junczys-Dowmunt. 2023. https://doi.org/10.18653/v1/2023.wmt-1.62 Cometoid: Distilling strong reference-based machine translation metrics into E ven stronger quality estimation metrics . In Proceedings of the Eighth Conference on Machine Tran...

    2023 doi

  9. [17]

    Nuno M Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and Andr \'e FT Martins. 2024. xcomet: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12:979--995

    2024

  10. [18]

    David W Johnson and Roger Johnson. 1985. Classroom conflict: Controversy versus debate in learning groups. American educational research journal, 22(2):237--256

    1985

  11. [19]

    Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, and Markus Freitag. 2023. https://doi.org/10.18653/v1/2023.wmt-1.63 M etric X -23: The G oogle submission to the WMT 2023 metrics shared task . In Proceedings of the Eighth Conference on Machin...

    2023 doi

  12. [20]

    Marzena Karpinska, Nader Akoury, and Mohit Iyyer. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.97 The perils of using M echanical T urk to evaluate open-ended text generation . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, page...

    2021 doi

  13. [21]

    Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R Bowman, Tim Rockt \"a schel, and Ethan Perez. 2024. Debating with more persuasive llms leads to more truthful answers. In Forty-first International Conference ...

    2024

  14. [22]

    Tom Kocmi and Christian Federmann. 2023 a . Gemba-mqm: Detecting translation quality error spans with gpt-4. In Proceedings of the Eighth Conference on Machine Translation, pages 768--775

    2023

  15. [23]

    Tom Kocmi and Christian Federmann. 2023 b . https://aclanthology.org/2023.eamt-1.19 Large language models are state-of-the-art evaluators of translation quality . In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193--203, ...

    2023

  16. [24]

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Ren Lu, Thomas Mesnard, Johan Ferret, Colton Bishop, Ethan Hall, Victor Carbune, and Abhinav Rastogi. 2023. Rlaif: Scaling reinforcement learning from human feedback with ai feedback

    2023

  17. [25]

    Christoph Leiter and Steffen Eger. 2024. Prexme! large scale prompt exploration of open source llms for machine translation and summarization evaluation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11481--11506

    2024

  18. [26]

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. 2024. From generation to judgment: Opportunities and challenges of llm-as-a-judge. arXiv preprint arXiv:2411.16594

    2024

  19. [27]

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.992 Encouraging divergent thinking in large language models through multi-agent debate . In Proceedings of the 2024 ...

    2024 doi

  20. [28]

    Qingyu Lu, Baopu Qiu, Liang Ding, Kanjian Zhang, Tom Kocmi, and Dacheng Tao. 2024. https://doi.org/10.18653/v1/2024.findings-acl.520 Error analysis prompting enables human-like translation evaluation in large language models . In Findings of the Association for Computational L...

    2024 doi

  21. [29]

    Subhajit Naskar, Daniel Deutsch, and Markus Freitag. 2023. https://doi.org/10.18653/v1/2023.wmt-1.67 Quality estimation using minimum B ayes risk . In Proceedings of the Eighth Conference on Machine Translation, pages 806--811, Singapore. Association for Computational Linguistics

    2023 doi

  22. [30]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...

    2002

  23. [31]

    Maja Pavlovic and Massimo Poesio. 2024. The effectiveness of llms as annotators: A comparative overview and empirical analysis of direct representation. arXiv preprint arXiv:2405.01299

    2024 arXiv

  24. [32]

    Stefano Perrella, Lorenzo Proietti, Alessandro Scir \`e , Niccol \`o Campolungo, and Roberto Navigli. 2022. https://aclanthology.org/2022.wmt-1.51 M a TES e: Machine translation evaluation as a sequence tagging problem . In Proceedings of the Seventh Conference on Machine Tran...

    2022

  25. [33]

    Maja Popovi \'c . 2015. https://doi.org/10.18653/v1/W15-3049 chr F : character n-gram F -score for automatic MT evaluation . In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392--395, Lisbon, Portugal. Association for Computational Linguistics

    2015 doi

  26. [34]

    Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. Comet: A neural framework for mt evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685--2702

    2020

  27. [35]

    Ricardo Rei, Marcos Treviso, Nuno M Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, Jos \'e GC De Souza, Taisiya Glushkova, Duarte Alves, Lu \' sa Coheur, et al. 2022. Cometkiwi: Ist-unbabel 2022 submission for the quality estimation shared task. In Proceedings of...

    2022

  28. [36]

    Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text generation. In Proceedings of ACL

    2020

  29. [37]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  30. [38]

    Chris van der Lee, Albert Gatt, Emiel van Miltenburg, Sander Wubben, and Emiel Krahmer. 2019. https://doi.org/10.18653/v1/W19-8643 Best practices for the human evaluation of automatically generated text . In Proceedings of the 12th International Conference on Natural Language ...

    2019 doi

  31. [39]

    Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. 2024. Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge. arXiv preprint arXiv:2407.19594

    2024 arXiv

  32. [40]

    Wenda Xu, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Wang, and Lei Li. 2023. Instructscore: Towards explainable text generation evaluation with automatic feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processin...

    2023

  33. [41]

    Ruoyu Zhang, Yanzeng Li, Yongliang Ma, Ming Zhou, and Lei Zou. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.872 LLM a AA : Making large language models as active annotators . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13088--13103,...

    2023 doi

  34. [42]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675

    2019 arXiv

  35. [43]

    Haofei Zhao, Yilun Liu, Shimin Tao, Weibin Meng, Yimeng Chen, Xiang Geng, Chang Su, Min Zhang, and Hao Yang. 2024. From handcrafted features to llms: A brief survey for machine translation quality estimation. arXiv preprint arXiv:2403.14118

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.