{"id":"92527b6c-9349-40e5-92dd-89041554a02d","arxiv_id":"2412.03152","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A new metric, SysDep, quantifies how much an MT metric depends on the system being scored, and XCOMET's system dependence is large enough to change system rankings.","lead":"This paper introduces SysDep, a measure of how consistently automated machine translation metrics score different translation systems. Using WMT23 data it shows that XCOMET, despite high correlation with human ratings, overrates and underrates specific systems enough to change their ranking.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Out-of-bounds 'nan' handling in Appendix E can bias ED/SysDep: unrated extreme metric scores are dropped from remapped averages, inflating the ED range and the claimed system dependence.","rationale":"The reader's verdict was CONDITIONAL, with the weakest assumption centered on the reliability of per-system calibration functions and the out_of_bounds='nan' choice. My stress-test agrees that the out-of-bounds handling is a major risk, but it sharpens the issue: ED in Table 1 does not actually use the per-system f_k estimates—it compares a remapped average over 1976 metric ratings with a human average over 1177 ratings. Thus the accuracy of f_k is not the primary load-bearing assumption; the primary issue is the mismatch between the two segment sets combined with silent dropping of out-of-support scores. This can bias ED in a direction that artificially increases SysDep, exactly matching the observed pattern. The proposed test—recomputing with clipping and on the shared 1177 segments—would settle whether the SysDep magnitude survives. Since the conditional verdict already requires additional robustness checks, the verdict remains CONDITIONAL (no change needed).","tokens_in":14246,"tokens_out":11613,"duration_ms":114654,"concrete_test":"Rerun the provided code on the WMT23 zh-en XCOMET data with out_of_bounds='clip' instead of 'nan' (or equivalent bounded extrapolation), and recompute ED and SysDep for all systems; also report the number and metric-score distribution of dropped segments per system under the original 'nan' setting. If SysDep drops below the intra-system maximum in Table 9, or the sign pattern of ED changes, the paper's conclusion that XCOMET exhibits high system dependence is not robust. Additionally, recompute the human-vs-metric ranking comparison using only the shared 1177 human-rated segments for both mu_H_k and mu_G_k to separate segment-set mismatch from calibration effects.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—SysDep = 2.816 for XCOMET on zh-en—rests on ED values computed as mu_G_k - mu_H_k, where mu_G_k averages the pooled isotonic calibration function over NM = 1976 metric ratings per system, while mu_H_k averages human MQM ratings over only NH = 1177 segments. This is only valid if the 1177 human-rated segments are a random subset of the 1976 metric-rated segments and if f_G can be evaluated on all 1976 scores. The code in Appendix E (Listing 1, line 43) sets out_of_bounds='nan', so any metric score outside the pooled human-rated support is silently dropped from mu_G_k. Because the 1976-set includes 799 unrated segments that are more likely to contain extreme XCOMET scores, the dropped segments are not random: top systems lose high scores (pulling mu_G_k down, making ED more negative), while bottom systems lose low scores (pushing mu_G_k up, making ED more positive). This mechanical effect produces exactly the observed ED pattern—negative for top systems, positive for bottom systems—and inflates SysDep. The paper reports no count of dropped segments and no comparison to a clipping or extrapolation variant, so the magnitude and even the sign pattern of ED may be an artifact of the implementation rather than genuine metric bias.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a new evaluation dimension for automatic MT metrics: how consistently a metric maps metric scores to human quality across different MT systems. It formalizes the system-specific calibration function f_k and a global function f_G, defines the Expected Deviation ED(k) as the difference between the global predicted mean and the system-specific human mean, and defines SysDep as the range of ED across systems. On WMT23 zh-en data with XCOMET, the paper reports SysDep = 2.816, with top systems underrated and bottom systems overrated, and argues this explains ranking inconsistencies despite high segment-level correlation. Additional results cover other language pairs and metrics, plus an intra-system variability control.","tokens_in":14531,"tokens_out":9849,"duration_ms":93220,"significance":"The core idea is timely and practically relevant: correlation with human judgments alone does not guarantee consistent system-level rankings, and the proposed measure directly quantifies a form of metric bias. The derivation of Eq. (1) is sound, the estimation via isotonic regression with bootstrap is standard, and the code is provided, which supports reproducibility. The intra-system control in Appendix D is a valuable sanity check. However, the empirical support for the central claim is currently weakened by a silent data-dropping mechanism in the implementation and by the absence of uncertainty quantification; these issues are addressable within the scope of the paper.","major_comments":[{"comment":"The code in Appendix E (Listing 1, line 43) sets out_of_bounds='nan' for each isotonic regression model, and Listing 2 computes remapped expectations with np.nanmean. Any metric rating outside the pooled human-rated support is therefore silently discarded when computing mu_G_k. Since mu_G_k is averaged over NM=1976 metric ratings while mu_H_k is averaged over NH=1177 human ratings, and the 799 unrated segments are likely to contain extreme metric scores, the dropped segments are not random: top systems lose high scores and bottom systems lose low scores. This mechanically produces the observed ED pattern of negative values for top systems and positive values for bottom systems, thereby inflating SysDep. The paper does not report the number of dropped segments per system, and the text in Section 3 says the ranking discrepancy is 'attributed to the uncertainty introduced by bootstrapping and extrapolating to the unpaired metric ratings,' although the code does not extrapolate. Please report drop counts and rerun the analysis with clipping or extrapolation to show that the main claim is robust to the choice of handling out-of-range metric scores.","section":"Section 3, Appendix E, Table 1"},{"comment":"ED and SysDep are reported without any uncertainty estimates. Given that each system's f_k is fitted on only 1177 human-rated segments, the ED point estimates have substantial sampling error, and SysDep, being a max-min over 15 noisy EDs, is particularly sensitive to extreme observations. The bootstrap machinery already used for the calibration curves could be extended to produce bootstrap or permutation confidence intervals for the per-system ED values and for SysDep, together with a test of whether the observed between-system SysDep exceeds the intra-system baseline in Appendix D. Without such intervals, the reader cannot tell whether SysDep = 2.816 is statistically distinguishable from zero or from the intra-system control. Please provide these intervals and a formal comparison.","section":"Section 3, Table 1"}],"minor_comments":[{"comment":"The illustrative example with f1(m) = f2(m) + C claims that equal metric means imply 1/N sum f2(m1) > 1/N sum f2(m2). This inequality does not follow from monotonicity alone; it requires an additional condition such as convexity of f2 or a stochastic dominance assumption. Please revise the example to state the needed assumption or provide explicit distributions.","section":"Section 2"},{"comment":"The phrase 'extrapolating to the unpaired metric ratings' is inconsistent with the code, which uses out_of_bounds='nan' and therefore drops rather than extrapolates. Either implement a bounded extrapolation (e.g., constant or linear extension) or reword the sentence to describe dropping.","section":"Section 3"},{"comment":"The exact XCOMET variant used in the main experiment is not specified. Appendix C lists XCOMET-Ensemble, XCOMET-XL, and XCOMET-XXL, but Table 1 labels the metric simply as 'XCOMET.' Please identify which checkpoint produced the reported numbers.","section":"Section 3, Table 1"},{"comment":"The Lommel et al. (2014) reference is malformed: it reads 'Arle. Language Technology Lab) Lommel, Hans. Language Technology Lab) Uszkoreit, and Aljoscha. Language Technology Lab) Burchardt.' and needs to be corrected to a standard author list.","section":"References"},{"comment":"Equation (2) uses a single N for both sums, but Section 3 distinguishes NH and NM. Please clarify whether the definition assumes equal sample sizes or whether the estimator should explicitly account for the different numbers of human-rated and metric-rated segments.","section":"Section 2, Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript makes a useful contribution to MT metric evaluation, and the central methodological issue (the NaN dropping in Appendix E) is fixable with additional experiments. The lack of confidence intervals is also straightforward to address given the existing bootstrap code. I recommend major revision rather than rejection, because the proposed measure and the general approach are sound and within the scope of the journal; the empirical claims need to be substantiated with robustness checks and uncertainty quantification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: SysDep is a genuinely useful addition to MT metric evaluation. The formalism is straightforward, the code ships, and the paper demonstrates a real phenomenon—a metric can have high segment-level correlation yet disagree with human system rankings because the human-to-metric mapping differs across systems. That is worth saying clearly.\n\nWhat is new: the SysDep summary and its application to WMT23. The building blocks are properly attributed to Wu and Resnick (2024) and Posocco and Bonnefoy (2021), and the derivation of Eq. 1 is sound. The definition of ED as the difference between global and per-system remapped means matches the intuition behind system-dependent calibration. The limitations section is honest, and the intra-system control in Appendix D is a good attempt to address noise. Shipping the code in Appendix E helps a lot.\n\nThe soft spots are empirical, not conceptual. First, there are no confidence intervals on ED or SysDep; the bootstrap is used only for the f curves in Figure 1. That makes it hard to know whether the 2.8-point SysDep is distinguishable from sampling noise. Second, the causal phrasing in the introduction (\"the cause... is due to\") overstates what a correlational study can show. Third, and most important, the code evaluates f_G on all NM metric ratings with out_of_bounds='nan', so any score outside the pooled training range is silently dropped from mu_G_k. Because mu_G_k averages over NM while mu_H_k averages over NH, the dropped segments are exactly the unrated ones—and they are not necessarily a random subset. The paper does not report how many scores are dropped per system or check a clipping/extrapolation variant. Without that, the sign pattern of ED (negative for top systems, positive for bottom systems) could in principle be an artifact of dropping extreme scores. I would not call it a proven fatal flaw—the pooled global training set is large, so the dropped fraction may be small—but the authors need to show that.\n\nThe intra-system control in Appendix D does not settle this, because it inherits the same code path.\n\nThis paper is for MT evaluation researchers and anyone choosing among metrics at the system level. The conceptual contribution should survive; the empirical claims need revision. I would send it to peer review, with the expectation that the authors add sensitivity analyses, report dropped counts, and temper the causal language.","headline":"A genuinely useful measure of metric/system dependence, but the empirical ED/SysDep numbers rest on a NaN-handling choice that could bias the sign pattern; worth careful revision, not a desk reject.","tokens_in":15026,"tokens_out":3535,"would_cite":true,"duration_ms":34143,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A translation metric can show high segment-level agreement with humans and still rank systems wrongly, because the mapping from metric score to human quality changes from system to system.","keywords":["machine translation evaluation","automated metrics","system dependence","Expected Deviation","SysDep score","metric calibration","MQM ratings","XCOMET"],"falsifier":"Re-run the SysDep computation on a fresh, larger set of human MQM ratings for the same WMT23 systems, or on held-out halves of the existing ratings: if the expected deviations shrink to near zero, flip sign, or vary as much within a single system as between systems, the paper's claim of systematic system dependence would be contradicted.","tokens_in":14065,"feed_emoji":"📏","tokens_out":6822,"duration_ms":62647,"temperature":0.7,"pith_summary":"This paper argues that a translation metric can pass the standard test, namely high correlation with human judgments at the segment level, and still be unfair when used to rank entire systems. The reason is formalized as a per-system calibration curve: for each system, the expected human rating given a metric score can differ, even when the pooled curve across all systems is monotonic. The authors introduce Expected Deviation, how much the global curve over- or under-predicts a system's human quality, and SysDep, the spread of those deviations across systems, as measures of this system dependence. On WMT23 Chinese-to-English data, XCOMET shows a SysDep of 2.816, with individual deviations ranging from $-0.82$ to $1.996$, enough to invert the order of systems compared with human rankings. The paper concludes that metrics should be evaluated for consistency across systems, not only for correlation.","feed_headline":"Top translation metric matches humans yet misranks systems","feed_subtitle":"New SysDep score shows when a metric’s score-to-quality mapping changes with the system being measured.","key_machinery":"The load-bearing object is the system-specific conditional expectation $f_k(m) = \\mathbb{E}[h \\mid m]$, estimated by bootstrap-averaged isotonic regression on paired human and metric ratings for each system. It converts metric scores to the human scale for one system, and the analogous pooled function $f_G$ converts them on average. From these, Expected Deviation is $ED(k) = \\frac{1}{N}\\sum_j f_G(m_k^{(j)}) - \\frac{1}{N}\\sum_j f_k(m_k^{(j)})$, and the SysDep score is the range $\\max_k ED(k) - \\min_k ED(k)$; the range captures the worst-case pair of systems that could be misordered.","core_discovery":"The paper's central claim is that a metric's per-system mapping $f_k(m) = \\mathbb{E}[h \\mid m]$, the expected human rating for a given metric score on system $k$, is not the same across systems, and that this heterogeneity, not noise, explains why metric and human system rankings disagree. Using isotonic regression with bootstrap averaging on WMT23 data, the authors estimate each $f_k$ and a global $f_G$; the difference between remapping system scores through $f_G$ and through $f_k$ defines the Expected Deviation. For XCOMET on zh-en, the deviations span 2.816 points on the MQM scale, which moves NLLB-Greedy from 15th to 12th place and leaves the metric ranking inconsistent with the human ranking despite XCOMET's high global segment-level correlation of 0.65. The authors therefore state that monotonic correlation to humans is insufficient to certify a metric for use in ranking systems.","pith_inferences":["The same calibration-curve argument should apply to any automated evaluator that ranks systems by averaged scores, such as summarization or image captioning, so SysDep could become a generic fairness diagnostic for automatic evaluation.","With only about 1,177 human-rated segments per system, the differences between ED values for neighboring systems may be within bootstrap noise; reporting confidence intervals on ED and testing pairwise ED differences would sharpen the paper's central claim.","A direct extension would be to decompose SysDep by score range to identify where a metric is unfair and to use that information to recalibrate per system, although the paper explicitly stops at measurement."],"forward_implications":["A metric can report a high segment-level correlation with humans and still produce system rankings that contradict human rankings, as XCOMET does on WMT23 zh-en.","SysDep gives a single number for comparing metrics on fairness: on zh-en, GEMBA-MQM has the lowest range among WMT23 metrics while XCOMET's range is 2.816.","System-level evaluation should report remapped scores through the global calibration function, not raw metric averages, because raw averages implicitly assume all per-system curves coincide.","Adding system dependence to standard metric evaluation would change which metrics are considered best: metrics that win on correlation will not necessarily be the most consistent across systems."],"supporting_citations":[{"why":"Supplies the WMT23 Metrics shared task data and the human MQM ratings used in every experiment.","marker":"Freitag et al. (2023)"},{"why":"Defines XCOMET, the metric whose system dependence is measured in the main experiment.","marker":"Guerreiro et al. (2023)"},{"why":"Provides isotonic regression, the method used to estimate the monotonic calibration curves.","marker":"Barlow and Brunk (1972)"},{"why":"Gives the conditional-expectation derivation that the paper adapts from binary prevalence estimation to metric calibration.","marker":"Wu and Resnick (2024)"},{"why":"Is the basis for treating Expected Deviation as an analogue of Expected Calibration Error.","marker":"Posocco and Bonnefoy (2021)"},{"why":"Prior evidence that metrics favor some systems over humans; SysDep is offered as its explanation.","marker":"von Däniken et al. (2024)"},{"why":"Decomposes ranking sign errors into bias and variance, the bias that SysDep quantifies.","marker":"Wei and Jia (2021)"},{"why":"Defines the WMT23 general MT task whose participating systems are ranked.","marker":"Kocmi et al. (2023)"},{"why":"Defines MQM, the human rating scheme used as ground truth.","marker":"Lommel et al. (2014)"}],"fun_headline_variants":["High metric-human correlation hides system misranking","System-specific metric curves explain ranking errors","Metrics need system-fairness checks, not just correlation","SysDep reveals when metrics play favorites among systems","Even top metrics can misrank translation systems"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The per-system calibration curves, fitted from only 1,177 human-rated segments per system, truly reflect how the metric maps to human quality instead of random noise.","fun_headline_variants_meta":{"raw":{"variants":["High metric-human correlation hides system misranking","System-specific metric curves explain ranking errors","Metrics need system-fairness checks, not just correlation","SysDep reveals when metrics play favorites among systems","Even top metrics can misrank translation systems"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000183,"raw_usage":{"total_tokens":1237,"prompt_tokens":791,"completion_tokens":446,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":407,"completion_tokens_details":{"reasoning_tokens":377}},"tokens_in":407,"tokens_out":446,"duration_ms":4992,"temperature":1.0,"reasoning_tokens":377,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:42:06.238446+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the SysDep computation on a fresh, larger set of human MQM ratings for the same WMT23 systems, or on held-out halves of the existing ratings: if the expected deviations shrink to near zero, flip sign, or vary as much within a single system as between systems, the paper's claim of systematic system dependence would be contradicted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the conditional-expectation derivation that the paper adapts from binary prevalence estimation to metric calibration."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the basis for treating Expected Deviation as an analogue of Expected Calibration Error."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Decomposes ranking sign errors into bias and variance, the bias that SysDep quantifies."}],"review_version":1}