{"id":"99b7b75a-8685-4ff2-8cc8-5c247a06d053","arxiv_id":"2412.11615","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MT-LENS is an open-source extension of LM-eval-harness that bundles MT quality, gender bias, added toxicity, and perturbation-robustness evaluations into one command-line and web-based toolkit.","lead":"This paper presents MT-LENS, an open-source toolkit that bundles many machine-translation evaluation checks into one framework, including translation quality, gender bias, added toxicity, and robustness to typos. It matters because it lowers the effort for researchers and engineers to run these checks on any translation model and inspect the results in a web interface.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No reference-implementation validation: the bias/toxicity pipelines and metric wrappers are asserted, not checked, so MT-LENS can report incorrect scores while running cleanly.","rationale":"I read the paper in good faith as a systems/demonstration contribution: it does not claim a new metric or a new empirical finding, but rather an integrated, reusable evaluation toolkit built on LM-eval-harness. The code release on GitHub is genuine supporting evidence, and the task catalog is concrete and well-motivated. The most load-bearing condition for the central claim is that the toolkit actually implements the advertised metrics and datasets correctly. The paper provides no evidence for this beyond its own description. The reader's weakest assumption pointed at the validity of third-party toxicity and gender classifiers; I agree but would sharpen and broaden it: even if those classifiers are valid, MT-LENS could still misapply them through incorrect filtering, subsetting, or output parsing. The same applies to the standard metrics, where a wrapper bug in SacreBLEU or COMET invocation would silently corrupt every downstream comparison. No internal inconsistency is apparent, but the absence of validation against reference implementations means 'it runs' is not enough to establish 'it evaluates correctly.' The broken demo links are a smaller, addressable issue that does not change the verdict. My recommendation therefore maintains the reader's CONDITIONAL verdict: the toolkit is plausibly useful, and the authors can resolve the concern with a small validation suite and version-pinned reproducibility instructions.","tokens_in":11238,"tokens_out":3525,"duration_ms":36652,"concrete_test":"Run the paper's example on a pinned commit of the repository: evaluate a small model on en_ca_flores_devtest, then recompute every metric from the generated JSON files by calling the upstream libraries directly (SacreBLEU with identical arguments, unbabel-comet for COMET/COMET-KIWI/XCOMET, metricx, etc.) and compare segment-level scores within floating-point tolerance. Additionally, run 100 MUST-SHE sentences through MT-LENS and through the Mash et al. (2024) revised script on the same translations, and compare the resulting gender-accuracy scores. Any mismatch identifies a wrapper or pipeline bug; passing both checks would eliminate the central correctness concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MT-LENS provides a reliable, unified MT evaluation framework. Reliability rests on every implemented task computing exactly what its source dataset or metric defines. The paper never shows that MT-LENS outputs match the official or reference implementations it wraps. Section 3.2 describes three gender-bias tasks (MUST-SHE with the 'revised script of Mash et al. (2024)', MMHB via CHRF subset scores, MT-GENEVAL) and an added-toxicity pipeline that filters with MUTOX and scores with ETOX, MUTOX, and DETOXIFY. These are composite pipelines with many places for off-by-one errors, wrong morphological forms, or incorrect subsetting; the paper reports no validation against the original task scripts, no smoke-test numbers, and no comparison against human judgments. The same gap holds for the core metrics: Table 2 lists wrappers around SacreBLEU, unbabel-comet, metricx, and transformers, but no experiment shows the wrapper reproduces the underlying library outputs on a shared test set. A user who follows the example command and gets a score has no way to know whether that score is correct. The limitations section only mentions customization effort, and the ethical statement concedes the proxies are binary/context-independent; neither addresses implementation correctness. Broken 'this link' placeholders for the demo and video (Sections 1 and 4) further block independent verification of the UI claim. Thus the load-bearing unverified assumption is not merely that the third-party classifiers are valid, but that MT-LENS integrates them and the base metrics correctly at all.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"MT-LENS is a proposed open-source toolkit that extends LM-eval-harness to machine translation evaluation, supporting translation quality metrics (BLEU, TER, CHRF, COMET, BLEURT, MetricX, XCOMET), three gender-bias tasks (MUST-SHE, MMHB, MT-GENEVAL), an added-toxicity pipeline (HOLISTIC BIAS with MUTOX/ETOX/DETOXIFY), and character-noise robustness perturbations. It also provides a Streamlit-based user interface for segment- and system-level analysis, including error-span visualization, segment-length scatter plots, and bootstrapped significance tests. The paper describes the architecture, lists supported datasets and metrics in tables, shows an example command, and reports qualitative UI observations from a Catalan-to-English comparison of madlad-400-3B and NLLB-3.3B.","tokens_in":11456,"tokens_out":4581,"duration_ms":39792,"significance":"If the toolkit works as described, it would be a useful contribution to the MT evaluation ecosystem: it builds on the widely adopted LM-eval-harness, unifies several evaluation tasks beyond translation quality, offers a user-friendly UI, and releases code on GitHub. The inclusion of bootstrapped significance tests and support for error-span visualization are concrete strengths. However, the paper's central claim—that MT-LENS provides reliable evaluation across these tasks—is not yet supported by any validation evidence; the described pipelines are asserted rather than checked against reference implementations or human judgments. Because the tool produces numeric scores that users will trust, the lack of correctness evidence is a load-bearing gap that must be addressed before the claims can be accepted.","major_comments":[{"comment":"The gender-bias and added-toxicity pipelines (MUST-SHE with the 'revised script of Mash et al. (2024)', MMHB via CHRF subset scores, MT-GENEVAL, and the MUTOX-filtered ETOX/MUTOX/DETOXIFY toxicity pipeline) are described but never validated against the original task scripts, an independent reference implementation, or human judgments. These are composite pipelines with many opportunities for off-by-one errors, morphological mismatches, incorrect subsetting, or classifier-threshold mistakes, so MT-LENS can report misleading bias and toxicity scores while running without errors. Please add a validation experiment that reproduces published dataset statistics or reference outputs on a small shared set (e.g., recompute MUST-SHE accuracy with the official script and compare, or verify toxicity labels against the original ETOX/MUTOX releases).","section":"Section 3.2"},{"comment":"The metric wrappers listed in Table 2 (SacreBLEU, unbabel-comet, metricx, transformers for BLEURT) are claimed to support 'state-of-the-art metrics', but the paper provides no evidence that MT-LENS reproduces the outputs of the underlying libraries. BLEU depends heavily on tokenization and smoothing choices, COMET versions differ in model weights and normalization, and MetricX has multiple variants, so a wrapper can silently produce different scores. Include a reproducibility check, such as running the same sentences through MT-LENS and the official implementations on a standard set (e.g., FLORES-200 devtest) and reporting score differences; this is essential for any evaluation tool.","section":"Section 3 and Table 2"},{"comment":"The demo and demo-video links are placeholder text ('this link' in Section 1), and the UI demonstration in Section 4 reports only qualitative findings (e.g., 'madlad-400-3B exhibits greater robustness') without providing the underlying score tables, evaluation JSON, or a reproducible command sequence. This blocks independent verification of the central usability claim. Populate the links and include a small reproducible example with actual computed scores (e.g., BLEU/COMET values for a few segments) so readers can compare against their own runs.","section":"Sections 1 and 4"}],"minor_comments":[{"comment":"There are numerous spacing artifacts in key terms: the title uses 'MT-L ENS', Table 1 has 'H OLISTIC BIAS', 'M UST-SHE', and 'MT-G ENEVAL', and the example command in Section 3.1 contains '-- tr a n sl a tio n _ kw a r gs'; these should be corrected to 'MT-LENS', 'HOLISTIC BIAS', 'MUST-SHE', 'MT-GENEVAL', and '--translation_kwargs'.","section":"Throughout"},{"comment":"The example usage is not valid shell syntax; it mixes variable assignment with a JSON-like block and uses spaces within option names. Provide an actual command-line example that users can copy and run.","section":"Section 3.1"},{"comment":"The related work says 'MT-C OMPARE EVAL' and the CTranslate2 footnote reads 'CTranslate22'; these should be 'MT-ComparEval' and 'CTranslate2'.","section":"Section 2"},{"comment":"The gender-bias UI is described as having tabs for MUST-SHE and MMHB, but the paper does not explain how MT-GENEVAL results are visualized or integrated into the UI; clarify.","section":"Section 4.3"},{"comment":"The paper does not specify which versions or splits of the datasets are used (e.g., HOLISTIC BIAS release, FLORES-200 devtest, NTREX-128 version); pinning these versions is important for reproducibility.","section":"Tables 1 and 2"}],"recommendation":"major_revision","confidential_remarks":"The central contribution is a software release, and the manuscript would be considerably strengthened by adding a validation section that compares MT-LENS outputs against the official implementations of the wrapped metrics and the original task scripts. As it stands, the paper is more of a system description than a verified research contribution; if the authors can demonstrate correctness on a small shared test set, the paper would be suitable for publication. The broken demo links and lack of numeric results in the UI section are also fixable and should be addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a systems paper about a new wrapper around LM-eval-harness, and that is fine. What's actually new is the integration: translation quality, gender bias, added toxicity, and robustness to character noise in one tool, with a Streamlit UI, plus bootstrapped significance tests. The code is on GitHub and the authors build on a framework the community already uses. If the tool works as described, it lowers the barrier to running these evaluations, and that is a real contribution.\n\nWhat I think is missing is validation. The paper lists wrappers around SacreBLEU, unbabel-comet, metricx, and transformers, but never shows that MT-LENS scores match the reference libraries on a shared test set. The gender-bias and toxicity pipelines are composites of third-party datasets and classifiers, and there are plenty of places for off-by-one errors, wrong morphological forms, or incorrect subsetting. The stress-test note is right: a user can get a number with no way to know if it's the right number. That is the load-bearing issue. Also the demo and video links are literally \"this link\" placeholders in Sections 1 and 4; that blocks independent verification of the UI claim. And the robustness comparison in Section 4.4 reports a difference without any error bars or significance test, even though the paper claims significance testing as a feature.\n\nThese are addressable, not deal-breaking. The central claim—that the toolkit exists and runs—is plausible, and the authors are honest about the proxy limitations in the ethical statement. But the limitations section only mentions customization effort, not implementation correctness, so the gap is real.\n\nFor a systems/demonstration paper, the fix is straightforward: add smoke tests that reproduce a handful of scores from reference implementations, pin package versions, fix the links, and report the illustrative comparison with confidence intervals. If that validation is added, this is a solid tool. I'd accept it for peer review now, but I'd send it back for those additions.","headline":"A useful MT evaluation toolkit that needs reference-implementation checks before I'd trust its numbers.","tokens_in":12126,"tokens_out":1873,"would_cite":false,"duration_ms":17587,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MT-LENS is an open-source toolkit that extends the LM-eval-harness to cover translation quality, gender bias, added toxicity, and robustness to character noise in one evaluation workflow.","keywords":["machine translation evaluation","gender bias","added toxicity","character noise robustness","LM-eval-harness","Streamlit UI","bootstrapping significance tests","COMET"],"falsifier":"Take a held-out set of English sentences with known correct and incorrect gendered translations (for example, nurse and doctor templates with the referent swapped), run the MUST-SHE and MMHB tasks through MT-LENS, and compare the reported accuracy against the known labels; if a deliberately wrong-gender translation is not flagged as incorrect, the gender-bias pipeline is not measuring what it claims.","tokens_in":10978,"feed_emoji":"🧰","tokens_out":7996,"duration_ms":62790,"temperature":0.7,"pith_summary":"MT-LENS is an open-source toolkit that extends the LM-eval-harness library so that the same evaluation workflow covers machine translation quality, gender bias, added toxicity, and robustness to character noise. The authors claim that no existing MT evaluation tool brings all of these tasks together with an interface that lets users inspect both individual segments and whole systems. If the paper is right, researchers and engineers can run a broader, more interpretable battery of MT evaluations from one command-line tool, including statistical significance tests, without stitching together separate scripts. The paper builds on established datasets and metrics rather than proposing a new metric, so its contribution is infrastructure: making these evaluations accessible and comparable.","feed_headline":"MT-LENS runs quality, bias, toxicity, and noise checks on MT systems","feed_subtitle":"Four evaluation tasks in one interface, so comparing NMT systems goes beyond quality scores alone.","key_machinery":"The central mechanism is the task abstraction inherited and extended from LM-eval-harness: each evaluation is a named task {src}_{tgt}_{dataset} whose dataset, prompt template, and metric configuration are declared in YAML, and whose results are emitted as JSON with segment-level scores. This lets MT-LENS treat quality, bias, toxicity, and noise robustness uniformly, and the UI layer then renders those JSON files as interactive comparisons. The bootstrapped t-test over BLEU, COMET, and COMET-KIWI is the significance machinery for system comparison.","core_discovery":"The paper introduces MT-LENS as a unified evaluation framework built on LM-eval-harness. It defines MT tasks by dataset plus language pair and supports five blocks: model backends (fairseq, CTranslate2, transformers, vllm, plus pre-generated translations), task definitions, prompt formatting, metrics, and JSON results. For quality, it supports BLEU, TER, CHRF, COMET, BLEURT, MetricX, XCOMET, COMET-KIWI, and quality-estimation variants; for added toxicity, it filters HOLISTIC BIAS source sentences with MUTOX and scores translations with ETOX, MUTOX, and DETOXIFY; for gender bias, it runs MUST-SHE, MMHB, and MT-GenEval tasks out of English; for robustness, it injects swap, character-duplication, and character-drop noise into FLORES-200 at a controllable level. The Streamlit interface shows error spans from XCOMET, segment-length scatter plots, bootstrapped significance tests, and per-task dashboards.","pith_inferences":["If MT-LENS gains adoption, evaluation of MT systems could standardize around the same harness used for LLM benchmarks, making it easier to compare bias and toxicity results across papers—but only if the community agrees on which classifiers and thresholds to use.","The paper's added-toxicity pipeline could be extended to non-English source languages by combining MUTOX's multilingual coverage with source-side toxicity classifiers, something the current HOLISTIC-BIAS-based setup only partially supports.","The same perturbation framework could be used to test robustness to real-world OCR or keyboard noise if natural noise corpora were swapped in for the synthetic swap, chardupe, and chardrop operations.","The UI's error-span visualization depends on XCOMET; a natural extension would be to let users click through to the underlying token-level scores or to compare spans produced by different error-detection models."],"forward_implications":["Users can run translation quality, gender bias, added toxicity, and character-noise robustness evaluations on the same generative model through a single command-line interface.","The JSON output format with segment-level scores makes it possible to inspect exactly which sentences drive quality, bias, or toxicity differences between systems.","Bootstrapped significance tests on BLEU, COMET, and COMET-KIWI let practitioners see whether observed system differences are statistically meaningful.","Because it is built on LM-eval-harness, new MT datasets can be added by declaring tasks in YAML without changing evaluation code, and non-MT NLU tasks remain available in the same harness."],"supporting_citations":[{"why":"Supplies the LM-eval-harness base that MT-LENS extends.","marker":"Gao et al., 2024"},{"why":"Provides FLORES-200, used for general MT and for the character-noise robustness task.","marker":"Costa-jussà et al., 2022"},{"why":"Provides the MUST-SHE dataset used for gender bias evaluation.","marker":"Bentivogli et al., 2020"},{"why":"Provides the MMHB dataset and the MUTOX filtering pattern used for added toxicity.","marker":"Tan et al., 2024"},{"why":"Provides MT-GenEval, a counterfactual and contextual dataset for gender accuracy.","marker":"Currey et al., 2022"},{"why":"Provides the HOLISTIC BIAS dataset used to detect added toxicity.","marker":"Smith et al., 2022"},{"why":"Provides the COMET metric used for translation quality and system comparison.","marker":"Rei et al., 2022a"},{"why":"Provides XCOMET, the error-span metric that powers the UI's segment-level visualization.","marker":"Guerreiro et al., 2023b"}],"fun_headline_variants":["MT-LENS: one toolkit for MT quality, bias, toxicity, and noise","All-in-one MT evaluation: quality, bias, toxicity, robustness","MT-LENS unifies four MT evaluation tasks in one interface","Evaluate MT quality, bias, toxicity, and noise with MT-LENS"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reliability of the gender-bias and added-toxicity scores rests entirely on the external classifiers and datasets that label toxicity and gender, and the paper does not check those labels against human judgments, so a mislabeling proxy would make the toolkit's outputs look valid while being wrong.","fun_headline_variants_meta":{"raw":{"variants":["MT-LENS: one toolkit for MT quality, bias, toxicity, and noise","All-in-one MT evaluation: quality, bias, toxicity, robustness","MT-LENS unifies four MT evaluation tasks in one interface","Evaluate MT quality, bias, toxicity, and noise with MT-LENS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000543,"raw_usage":{"total_tokens":2579,"prompt_tokens":906,"completion_tokens":1673,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":1595}},"tokens_in":522,"tokens_out":1673,"duration_ms":9460,"temperature":1.0,"reasoning_tokens":1595,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:45:28.811888+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of English sentences with known correct and incorrect gendered translations (for example, nurse and doctor templates with the referent swapped), run the MUST-SHE and MMHB tasks through MT-LENS, and compare the reported accuracy against the known labels; if a deliberately wrong-gender translation is not flagged as incorrect, the gender-bias pipeline is not measuring what it claims.","supporting_citations":[],"review_version":1}