{"id":"c614efac-eef2-48eb-9eb2-2028a9c13862","arxiv_id":"2411.15387","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Using per-example in-context demonstrations built from historical same-source human MQM ratings makes an LLM judge dramatically better at fine-grained MT evaluation on WMT'23 and WMT'24.","lead":"This paper shows that prompting an LLM evaluator with historical human error ratings of other translations of the same source sharply improves fine-grained machine translation evaluation. The method, called Specialist, beats the XCOMET metric by 54% and 119% on the WMT'23 and WMT'24 test sets, which matters because automatic evaluation on fixed benchmarks is a practical bottleneck.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 'state-of-the-art' comparison is against a zero-shot XCOMET that never sees the historical pseudo-SxS ratings; without a same-data adapted baseline, the 54%/119% gain conflates same-source ICL with mere access to test-set labels.","rationale":"The paper's internal ablations are well-designed: Shuffled sources, Fixed different source, Parrot, and Round2-rater controls collectively show that same-source, same-rater ICL examples carry the gain within the prompted-LLM setup, and the paper is transparent about the pseudo-SxS data requirement. I do not see an internal inconsistency that invalidates the mechanism claim. The load-bearing gap is external to that mechanism: the headline numbers compare a method that is handed the test set's historical labels against XCOMET, which is not. That makes the 'state-of-the-art' claim a comparison across different information regimes, not across methods in the same regime. The concrete check—fine-tuning XCOMET on the same held-out labels—would settle whether the ICL mechanism is necessary. Note also that the reader's reported Table 1 vs Table 13 discrepancy is mostly resolved: Table 1 WMT'24 35.59 is exactly the average of Table 13's Specialist rows (43.04, 26.58, 37.16); the actual inconsistency is in Table 13's shuffled-sources row for WMT'24 en-es, which appears to duplicate the en-de values and does not match Table 1's 24.46. That row does not affect the Specialist-vs-XCOMET comparison, but it should be corrected before publication.","tokens_in":20734,"tokens_out":16808,"duration_ms":163617,"concrete_test":"Use the exact WMT'23 and WMT'24 MQM ratings that Algorithm 1 uses as ICL examples. For each held-out system M*, fine-tune XCOMET-XXL (or MetricX-24) on the same-source, same-rater annotations of all other systems, then evaluate on M* with the same character-level F1 and partial-credit scoring as Tables 11/13. If this adapted XCOMET reaches or exceeds 51.59 (WMT'23) and 35.59 (WMT'24), the Specialist gain is attributable to test-set adaptation rather than to the same-source ICL prompting mechanism, and the headline should be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1 (Section 5) reports Specialist AutoMQM (row 2a) outperforming XCOMET-XXL-QE (row 1a) by 54% and 119% in character-level F1. But Algorithm 1 gives the Specialist prompt, for every test example, the human MQM annotations of all other systems on the same source and the same rater, while XCOMET is applied zero-shot with no such labels. The reported gain therefore conflates the paper's causal claim (same-source ICL demonstrations are the active ingredient) with the much weaker statement that having historical pseudo-SxS ratings helps. The paper's controls—Shuffled sources, Fixed different source, and Round2 ICL (Table 5)—show that ICL format and rater matching matter within the prompted-LLM family, but they do not test a strong non-ICL method that consumes the same held-out ratings. A supervised span tagger or a fine-tuned XCOMET trained on the same held-out other-system ratings is the relevant state-of-the-art baseline for a test-set-specialized metric; its absence leaves the headline SOTA claim undersupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Specialist, a method that specializes a prompted LLM-based autorater to a fixed test set by constructing per-example in-context learning (ICL) demonstrations from historical human MQM ratings of other translation systems on the same source segment, with the pseudo-SxS condition that a single rater annotated all translations of that source. The LLM then evaluates a held-out system's translation on that segment using those demonstrations. On WMT'23 and WMT'24, the authors report 54% and 119% relative character-level F1 improvements over XCOMET-XXL-QE. They support the central claim with extensive controls: shuffled-source ICL, fixed-different-source ICL, ICL-size scaling, a Parrot baseline that copies spans from demonstrations, filtering of exact-match demonstration errors, cross-rater rounds, multiple LLM backbones, and a transfer experiment to scalar score prediction.","tokens_in":20897,"tokens_out":7138,"duration_ms":69493,"significance":"If properly framed, the paper is a useful and carefully controlled contribution to transductive, test-set-specialized evaluation. The hold-one-out prompting protocol in Algorithm 1 is sound: the evaluated system's own ratings are never used as demonstrations, so the reported gains are not a hidden circularity. The controlled comparisons—Shuffled sources, Fixed different source, Parrot model, filtered ICL, and Round2-rater ICL—are strong evidence that same-source, same-rater demonstrations are the active ingredient rather than simply the number of ICL examples or naive copying. The paper's explicit accounting of rater variability is a further strength. However, the headline comparison to XCOMET-XXL-QE compares a transductive method that sees historical test-set labels with an inductive metric that does not; the 'state-of-the-art' claim is therefore overstated unless a baseline that consumes the same held-out ratings is added. The strict pseudo-SxS data requirement is also openly acknowledged in the Limitations, and it substantially narrows the practical applicability claimed in the introduction.","major_comments":[{"comment":"The headline 54%/119% F1 improvement over XCOMET-XXL-QE is not a same-setting comparison. Under Algorithm 1, Specialist receives, for every test example, the human MQM annotations of all other systems on the same source and from the same rater, whereas XCOMET-XXL-QE is applied without any of those test-set labels. The reported gain therefore conflates the paper's actual contribution—same-source ICL demonstrations, as supported by the Shuffled and Fixed-different-source controls—with the much weaker observation that access to historical pseudo-SxS ratings helps span prediction. To support the 'state-of-the-art' claim, the paper should add a strong baseline that consumes the same held-out ratings, for example XCOMET fine-tuned on the other systems' WMT'23/24 MQM labels, or a supervised span tagger trained on those labels. Without such a baseline, the abstract and Section 5 should be rephrased as improvements over zero-shot XCOMET in a transductive setting.","section":"Section 5, Table 1 (rows 1a vs 2a); Abstract"},{"comment":"The method requires that the exact test set already have MQM ratings from at least three other translation systems, collected pseudo-SxS so that a single rater rated all translations of each source. The paper's own Table 5 shows that using ICL examples from a different rating round—even for the same source—drops F1 to approximately the shuffled baseline level, and the Limitations section states that no commonly used public benchmark follows the pseudo-SxS collection procedure and that an MT-Bench attempt showed no gain. This is a load-bearing scope limitation: it narrows the introductory claim that evaluation in practice is performed on fixed canonical test sets to the much more specific situation of WMT-style pseudo-SxS MQM collections. The paper should state precisely which existing test sets satisfy the prerequisites and, given that the method is being proposed as a general approach, discuss explicitly whether the contribution is also a recommendation for how future test sets should be collected.","section":"Algorithm 1, Section 4.1, Limitations"},{"comment":"The Shuffled-sources row for en→es reports F1=26.12, precision=19.67, recall=38.84, which are exactly the same values as the en→de row above it. Because the table's note and the text in Section 5 use the en→es result to conclude that gains are smaller when pseudo-SxS is not enforced, this duplicate row must be corrected or explained. If the en→es values are a typographical error, the corrected numbers could affect the interpretation of the non-pseudo-SxS condition.","section":"Table 13, WMT'24 en→es row"}],"minor_comments":[{"comment":"The sentence 'also note that the Shuffled sources baseline already performs on par ... and and outperforms XCOMET' contains a duplicated 'and'.","section":"Section 5, first paragraph"},{"comment":"The direct-assessment prompt says 'Perfect meaning nd grammar'; this should read 'and grammar'.","section":"Appendix A, Figure 6 prompt"},{"comment":"The text says 'difference-source ICL examples'; this should be 'different-source'.","section":"Appendix B.1.2"},{"comment":"The notation is slightly inconsistent: line 5 initializes RM* to an empty list, line 14 says 'Append Ri M* to RM*', but the final 'Return RM*' is fine; consider adding an index to the append for clarity, e.g., RM*[i].","section":"Algorithm 1"},{"comment":"The cross-rater matrices would be easier to read with a colorbar and a short caption explaining that higher values on the diagonal indicate rater specialization; the current caption describes the axes but not the color scale.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The core experimental design is sound and the paper appears to be free of circular evaluation. The main issue is the SOTA framing: the comparison to XCOMET mixes settings, and the pseudo-SxS data requirement severely limits the setting in which the method can be used. I would be willing to accept a revised version that adds same-data baselines (or explicitly reframes the contribution as transductive) and that corrects the Table 13 duplication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this if you care about LLM-as-judge evaluation or MT metrics. The actual contribution is a prompt-only method that specializes an autorater to a fixed test set by building per-example ICL demonstrations from historical pseudo-SxS ratings of the same source and same rater. The controlled ablations are the strongest part: shuffled sources and fixed-different-source baselines isolate the same-source effect, and the Parrot baseline rules out simple copying. The gains over a zero-shot XCOMET are large (54%/119% F1), but that comparison is apples-to-oranges—XCOMET never sees the historical ratings. A fine-tuned XCOMET (or any span tagger) trained on the same held-out ratings is the missing baseline for the 'state-of-the-art' claim.\n\nWhat's new is the per-example, same-source ICL construction and the demonstration that it specializes to a single rater. The rater-round experiments are nice: using Round2 ICL drops to shuffled-baseline level, and the cross-rater matrix shows diagonal dominance. The scaling and filtering ablations are thorough. The en→es WMT'24 result is a useful negative control, confirming that rater matching matters.\n\nThe main soft spot is the headline comparison. The stress-test note is right: the Specialist consumes test-set labels, while XCOMET is zero-shot. The paper's internal controls show same-source ICL is the active ingredient among prompted variants, but they don't tell you whether a supervised model given the same labels would do better. That's a moderate omission, not a fatal one—the method is still new and useful—but the abstract's 'state-of-the-art' wording should be qualified. Minor issues: no code released, no contamination checks for the commercial LLMs, and the method's applicability is limited to test sets with pseudo-SxS ratings, which the paper acknowledges.\n\nI'd send it to review. The mechanism is well-supported, and the missing baseline is addressable in revision. A serious referee will ask for the same-data adapted baseline before accepting the SOTA claim, but the core result is likely to stand.","headline":"Strong controlled study of test-set-specific ICL for LLM-based MT evaluation, but the headline SOTA claim over zero-shot XCOMET needs a same-data baseline to be fair.","tokens_in":21508,"tokens_out":3120,"would_cite":true,"duration_ms":28930,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Specialist LLM raters beat the top MT metric by 54% to 119%.","keywords":["LLM-as-a-Judge","Autorater","machine translation evaluation","MQM","in-context learning","test-set specialization","pseudo-SxS rating collection","quality estimation"],"falsifier":"A decisive check: on a pseudo-SxS-rated test set, compare Specialist AutoMQM with the fixed-different-source baseline (same ICL examples and same rater, but demonstrations drawn from a different source segment). The paper reports 51.59 vs 27.06 F1 on WMT'23; if on another such set this gap collapses to near zero, the same-source mechanism is not the driver of the result. The same test applies in reverse: if a non-pseudo-SxS test set still yields the full Specialist gain, then the pseudo-SxS requirement is not load-bearing.","tokens_in":20487,"feed_emoji":"🧐","tokens_out":7787,"duration_ms":65529,"temperature":0.7,"pith_summary":"The paper argues that, when evaluating a machine translation system on a fixed canonical test set, the most effective evaluator is not a general-purpose LLM judge but one specialized to that test set through in-context demonstrations built from historical human ratings. For each source segment, the prompt shows the model the human MQM error annotations of other systems' translations of that same segment, collected by a single rater; the resulting Specialist AutoMQM metric improves character-level F1 by 54% on WMT'23 and 119% on WMT'24 over the previous state-of-the-art XCOMET. Ablations attribute the gains to the same-source demonstrations rather than to extra demonstrations or rater matching alone, and the method is robust across three LLM backbones and extends to direct score prediction. The practical appeal is that it upgrades automatic evaluation without finetuning or new per-system human annotation, at the cost of a one-off pseudo-SxS rating effort per benchmark.","feed_headline":"Specialist LLM raters beat the top MT metric by 54% to 119%","feed_subtitle":"Showing a judge the same source's past human ratings turns it into a state-of-the-art evaluator, no finetuning.","key_machinery":"The central mechanism is the Specialist method's per-example construction of in-context learning (ICL) examples: for each source segment $x_i$ in the test set, the prompt for evaluating a system $M^*$ includes the historical MQM ratings (error spans with severities and categories) for $N$ other systems' translations of the same $x_i$, with the pseudo-SxS constraint that those ratings were produced by a single fixed rater (Algorithm 1). Each test example therefore gets its own unique demonstration set, which distinguishes the method from fixed-demonstration baselines such as GEMBA-MQM. The demonstrations teach the model both which errors to flag and which spans to leave alone; the authors verify this with a 'Parrot' baseline that copies matching spans from ICL examples, which reaches only about half the Specialist's F1, and with an abstention analysis showing the Specialist predicts fewer copied errors than the shuffled baseline does.","core_discovery":"The central claim is that an LLM-based quality metric, which the paper calls AutoMQM, can be made state of the art for a fixed MT test set by prompting the LLM with historical human MQM error annotations of other systems' translations of the same source sentence, provided all those annotations come from a single rater per source (the pseudo-SxS constraint). Under hold-one-out evaluation on WMT'23 and WMT'24, this Specialist AutoMQM reaches character-level F1 of 51.59 and 35.59, respectively, versus 33.50 and 16.23 for XCOMET-XXL-QE. The paper shows that the same-source ingredient is decisive: using the same ICL examples but shuffled across sources, or drawn from a fixed different source with the same rater, does not beat XCOMET. It also claims the metric specializes to the rater as well as the test set, because replacing same-rater ICL examples with a different rater's ratings for the same source drops performance to the shuffled-baseline level, and the metric matches human inter-annotator agreement when the ICL rater and test-set rater differ.","pith_inferences":["If the Specialist route is applied to other NLG evaluation benchmarks (summarization, code generation, instruction following), the pseudo-SxS rating protocol is likely the make-or-break requirement: the paper's own MT-Bench experiment failed to beat its shuffled baseline precisely because ratings were not collected that way.","The rater-specialization result suggests that a large share of LLM-as-a-Judge performance may be implicit rater imitation; meta-evaluating against a single rater's ground truth can therefore understate a metric that genuinely generalizes across raters.","A practical testable extension: on a new test set with pseudo-SxS ratings, only three historical systems should suffice for Specialist AutoMQM to beat XCOMET; if the margin is instead within shuffle noise, the protocol has not been satisfied.","The method suggests a route toward personalized judges: metrics calibrated to one rater's error behavior, which could predict how that particular rater would judge new systems."],"forward_implications":["On the WMT'23 and WMT'24 MT test sets, a prompted LLM with no finetuning can outperform the previous state-of-the-art finetuned span-level quality-estimation metric XCOMET by 54% and 119% relative F1.","A one-time investment of pseudo-SxS human MQM ratings for a few systems makes a test set reusable: new systems can then be evaluated accurately without additional human annotation.","The method generalizes across LLM backbones (Gemini 1.5 Pro, GPT-4o, Claude 3.5 Sonnet), which suggests the gains are not an artifact of a single model.","The same-source demonstrations are the decisive ingredient: the 'Fixed, different source' and 'Shuffled sources' baselines do not reproduce the gains, so the effect is not merely extra ICL examples or rater matching.","The pattern extends to direct assessment (a 0-100 score prediction task), where the Specialist Scorer beats or matches the state-of-the-art MetricX-24 on most tested language pairs."],"supporting_citations":[{"why":"Supplies XCOMET, the state-of-the-art span-level baseline that Specialist AutoMQM must beat and is compared against in the headline F1 numbers.","marker":"Guerreiro et al. (2023)"},{"why":"Provides GEMBA-MQM, the closest prior prompted-LLM MQM metric; the Specialist prompt is adapted from its instructions and GEMBA serves as a baseline.","marker":"Kocmi & Federmann (2023a)"},{"why":"Provides the WMT'23 MQM test set with historical ratings from many systems, which is the data source for Specialist ICL examples and the meta-evaluation ground truth.","marker":"Freitag et al. (2023)"},{"why":"Provides the WMT'24 MQM test set, including the en→es pair where ratings were not pseudo-SxS and the Specialist gains shrink.","marker":"Freitag et al. (2024)"},{"why":"Defines the pseudo-SxS rating collection protocol that the Specialist method requires and that the paper shows is necessary for its gains.","marker":"Riley et al. (2024)"},{"why":"Defines the character-level F1 span-tagging metrics used to meta-evaluate Specialist AutoMQM against human MQM annotations.","marker":"Blain et al. (2023)"},{"why":"Provides the Acc23 segment-level pairwise accuracy with tie calibration used to meta-evaluate the Specialist Scorer in the score-prediction experiments.","marker":"Deutsch et al. (2023)"},{"why":"Defines the MQM error-annotation framework that supplies the human ratings used both as ICL demonstrations and as ground truth.","marker":"Lommel et al. (2014)"}],"fun_headline_variants":["Same-source history makes LLM rater beat XCOMET by up to 119%","Specialist LLM rater: 119% better on WMT24, no finetuning","Master-of-one LLM rater: historical ratings, big MT gains","One-trick LLM judge: specialize to test set, beat XCOMET","LLM rater specialized to test set beats XCOMET by 119%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method presupposes that the test set already carries human MQM ratings for at least three other translation systems, and that every translation of a given source segment was rated by one fixed rater; where that collection protocol is missing, such as WMT'24 en→es, the Specialist gain shrinks to a few F1 points.","fun_headline_variants_meta":{"raw":{"variants":["Same-source history makes LLM rater beat XCOMET by up to 119%","Specialist LLM rater: 119% better on WMT24, no finetuning","Master-of-one LLM rater: historical ratings, big MT gains","One-trick LLM judge: specialize to test set, beat XCOMET","LLM rater specialized to test set beats XCOMET by 119%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000211,"raw_usage":{"total_tokens":1461,"prompt_tokens":1036,"completion_tokens":425,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":652,"completion_tokens_details":{"reasoning_tokens":315}},"tokens_in":652,"tokens_out":425,"duration_ms":4265,"temperature":1.0,"reasoning_tokens":315,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:21:43.770430+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check: on a pseudo-SxS-rated test set, compare Specialist AutoMQM with the fixed-different-source baseline (same ICL examples and same rater, but demonstrations drawn from a different source segment). The paper reports 51.59 vs 27.06 F1 on WMT'23; if on another such set this gap collapses to near zero, the same-source mechanism is not the driver of the result. The same test applies in reverse: if a non-pseudo-SxS test set still yields the full Specialist gain, then the pseudo-SxS requirement is not load-bearing.","supporting_citations":[{"cited_title":"Results of wmt23 metrics shared task: Metrics might be guilty but references are not innocent","cited_arxiv_id":null,"evidence_quote":"Provides the WMT'23 MQM test set with historical ratings from many systems, which is the data source for Specialist ICL examples and the meta-evaluation ground truth."},{"cited_title":"Are llms breaking mt metrics? results of the wmt24 metrics shared task","cited_arxiv_id":null,"evidence_quote":"Provides the WMT'24 MQM test set, including the en→es pair where ratings were not pseudo-SxS and the Specialist gains shrink."},{"cited_title":"Finding Replicable Human Evaluations via Stable Ranking Probability","cited_arxiv_id":"2404.01474","evidence_quote":"Defines the pseudo-SxS rating collection protocol that the Specialist method requires and that the paper shows is necessary for its gains."},{"cited_title":"M., Kanojia, D., de Souza, J","cited_arxiv_id":null,"evidence_quote":"Defines the character-level F1 span-tagging metrics used to meta-evaluate Specialist AutoMQM against human MQM annotations."},{"cited_title":"Multidimensional quality metrics (mqm): A framework for declaring and describing translation quality metrics","cited_arxiv_id":null,"evidence_quote":"Defines the MQM error-annotation framework that supplies the human ratings used both as ICL demonstrations and as ground truth."}],"review_version":1}