{"id":"2f3a332b-3e4d-4c2b-88c6-16dc9ec6d8ca","arxiv_id":"2412.01340","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A rubric-plus-question-answering LLM framework for literary translation evaluation beats traditional MT metrics but still trails human agreement, especially on Korean honorifics.","lead":"This paper proposes a two-step framework to automatically evaluate literary machine translations from English to Korean, combining a rubric-based scorer with story-specific question answering. It finds the framework matches expert human judgments better than standard MT metrics, but still falls short of inter-human agreement and shows LLM judges favor LLM-made translations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline comparison is not apples-to-apples; sentence-level metrics applied to paragraphs may be systematically depressed, so the claimed superiority over traditional MT metrics is not yet established.","rationale":"The reader isolated the same self-admitted baseline mismatch as the weakest assumption; I concur. The strongest claim is explicitly comparative ('higher correlation with human judgment than traditional machine translation metrics'), so the fairness of the baseline comparison is load-bearing. The paper acknowledges the mismatch in Section 5 and Appendix B.1, yet still presents Table 4 as the headline evidence for superiority. The absence of confidence intervals compounds the problem: with 200 paragraphs and three annotators, a tau gap of 0.1-0.2 could easily be noise. I do not see a more serious internal inconsistency. The LLM-prefers-LLM finding in Section 6 is a documented pitfall and a threat to absolute-score trustworthiness, but not directly to the comparative correlation claim on the machine-translated evaluation set. The small question-validity check (one expert, 50 questions) is a real limitation but already folded into the conditional verdict. Thus the conditional accept stands, with the baseline recomputation as the decisive check.","tokens_in":19037,"tokens_out":5347,"duration_ms":52424,"concrete_test":"Recompute the 'Baseline' rows of Table 4 by scoring the same 200 human-annotated paragraphs at the sentence level and aggregating to paragraph scores (mean and max), handling BLEURT/COMET length limits by chunking or using long-text checkpoints, and report bootstrap 95% confidence intervals for Kendall's tau (or jackknife by story). If the RULER/VERSE tau advantage over BERTScore, BLEURT, COMET, and GEMBA-MQM falls inside overlapping confidence intervals, the claimed 'higher correlation than traditional machine translation metrics' is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5 ('Baseline') concedes that the comparison to COMET, BERTScore, BLEURT, and GEMBA-MQM 'may not be entirely appropriate' because those metrics target sentence-level evaluation while RULER/VERSE score paragraphs. The central claim that the framework 'obtains a higher correlation with human judgment than traditional machine translation metrics' rests on Table 4, yet that table gives no details on how the sentence-level baselines were applied to the 200 annotated paragraphs. BLEURT-20 and COMET have input-length limits; if 6.8-sentence paragraphs were fed directly, truncation or padding could depress baseline scores. GEMBA-MQM uses a different MQM error typology, and Appendix B.1 states its categories are 'very different' from RULER's. Without matched evaluation units, a stated aggregation rule, and confidence intervals, the reported tau advantage (e.g., RULER Lexical tau=0.68 vs. BLEURT tau=0.23) could reflect a procedural handicap rather than genuine metric quality. This is the main load-bearing weakness in the headline superiority claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-step framework for evaluating English-to-Korean literary machine translation. Step 1 (RULER) uses a Likert-scale rubric with four criteria (lexical choice, honorifics, syntax/grammar, content accuracy), and Step 2 (VERSE) uses LLM-generated story-specific verification questions that a second LLM grades. The authors validate the framework against expert human annotations on 200 paragraphs from 10 translated short stories, compare it with COMET, BERTScore, BLEURT, and GEMBA-MQM, and report Kendall's tau, Spearman's rho, and MSE. They find that GPT-4o as an evaluator outperforms the traditional metrics but still falls short of inter-human agreement, particularly on Korean honorifics. They also observe that the evaluator LLM tends to rate LLM-generated translations higher than human reference translations. The paper includes ablations (chain-of-thought, few-shot, rubric removal, reference removal), exact model versions, and public code and data.","tokens_in":19183,"tokens_out":6005,"duration_ms":54910,"significance":"If the central comparison were properly controlled, the framework would be a useful contribution to literary MT evaluation: it offers fine-grained, interpretable dimensions (honorifics, style, imagery, etc.) and its negative results about LLM judges' limitations on culturally specific features are valuable to the community. The paper's strengths include reproducible model versions, released data and code, human checks of generated questions, and a concrete A/B test illustrating LLM bias against human translations. However, the headline claim of superiority over traditional MT metrics rests on a baseline comparison that the authors themselves concede is 'not entirely appropriate,' and the exact rubric prompts needed to reproduce RULER are not included. These issues are fixable within the manuscript's scope, so a major revision is appropriate.","major_comments":[{"comment":"The comparison that supports the abstract's claim of 'higher correlation with human judgment than traditional machine translation metrics' is not apples-to-apples. The authors state that the baselines 'may not be entirely appropriate' because they are designed for sentence-level evaluation while RULER and VERSE operate on paragraphs, and Appendix B.1 notes that GEMBA-MQM and the framework's numbers are 'not really comparable' due to different error categories. The manuscript does not specify how BERTScore, BLEURT, COMET, and GEMBA-MQM were applied to the 200 annotated paragraphs, whether input-length limits caused truncation, how segment scores were aggregated to paragraph level, or whether the baselines were evaluated on the same 200 paragraphs. Table 4 also reports no confidence intervals or significance tests for the tau/rho differences, so the reported advantage (e.g., RULER Lexical tau=0.68 vs. BLEURT tau=0.23) could reflect a procedural handicap rather than genuine metric quality. Please provide the missing implementation details and matched evaluation units, or weaken the superiority claim accordingly.","section":"Section 5, Baseline; Appendix B.1; Table 4"},{"comment":"The full RULER rubric is not included in the manuscript; the text says 'The detailed rubric for each criterion will be released with our code and data after acceptance.' Since RULER scores are obtained by prompting an LLM with this rubric, the reported correlations in Table 4 cannot be independently reproduced from the paper alone. The exact rubric wording, the VERSE question-generation prompt, and the grading prompts should be included in an appendix or in the released repository at the time of submission.","section":"Section 4.1"},{"comment":"The abstract's claim that 'LLMs tend to favor translations produced by other LLMs' is supported by only one external A/B test: a single human annotator on 50 paragraphs from one story. The aggregate scores in Table 6 are produced by the same GPT-4o evaluator whose bias is being demonstrated, so they do not independently establish the claim. Please strengthen this analysis with multiple stories, multiple annotators, and inter-annotator reliability for the A/B judgments, or present the finding as a preliminary observation rather than a general result.","section":"Section 6.3, Table 6, and the 'What is Remembered' A/B test"}],"minor_comments":[{"comment":"MSE is reported for the proposed framework and LLM judges but omitted for COMET, BERTScore, BLEURT, and GEMBA-MQM; if these metrics produce scores on different scales, please state this explicitly so the reader knows the dash entries are not missing values.","section":"Table 4"},{"comment":"The sentence 'language models are much better at correlating with human preference... than the traditional MT metrics' should be supported with a significance test or confidence interval; the raw tau differences in Table 4 may not be statistically distinguishable given 200 paragraphs.","section":"Section 5.2"},{"comment":"Please clarify whether 'Claude-3.5 Sonnet' in the prose refers to 'claude-3-5-sonnet-20240620' in Table 9, since the translated model names in Tables 6 and 7 are abbreviated inconsistently.","section":"Section 6.3"},{"comment":"There is a typo: 'specially with honorifics' should be 'especially with honorifics'.","section":"Section 5.3, ablation bullets"},{"comment":"The radial axes in Figures 2, 4, and 6 lack numerical tick labels, making it difficult to read exact percentages; consider adding gridlines or a legend with values.","section":"Section 4.2 and Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper's main contribution is the two-step framework and its careful ablation study, and the authors are appropriately candid about the baseline mismatch. The biggest risk is that a reader will take the abstract's superiority claim at face value despite the self-admitted procedural confound. With the baseline details and rubric prompts supplied, the paper would be acceptable; without them, the central quantitative claim is not yet established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a careful, honest empirical study, and the new thing is real: a two-step pipeline for evaluating English-to-Korean literary translation, combining a rubric-based stage (RULER) with a QA-based story-specific stage (VERSE). Neither component is brand new, but the combination, applied to literary MT with culture-specific rubrics like Korean honorifics, is a legitimate contribution. The paper does the right reproducibility things—code and data are public, exact model versions are listed, ablations cover CoT, few-shot, and rubric removal, and inter-annotator agreement is reported. I trust the core numbers.\n\nThe softest spot is the baseline comparison. COMET, BERTScore, BLEURT, and GEMBA-MQM are sentence-level metrics; RULER and VERSE score whole paragraphs. The authors admit this in Section 5 and Appendix B.1, which is good, but it means the headline claim—'higher correlation with human judgment than traditional MT metrics'—rests on a comparison that is, at best, rough. If the sentence-level metrics were fed 6.8-sentence paragraphs without a stated aggregation or truncation rule, their scores could be artificially depressed. The stress-test concern is legitimate. That said, the authors flag it themselves, and the magnitude of the gap (e.g., Lexical tau 0.68 vs 0.23) is large enough that I suspect the direction would survive a fair comparison, though not necessarily the exact margins.\n\nOther soft spots are minor: no confidence intervals or significance tests on the correlations; the honorifics analysis is dominated by the default score of 5 for non-dialogue paragraphs; and the VERSE question quality check was one expert on 50 questions. The mild circularity—rubric categories induced from observed LLM errors, and the same model generating and grading VERSE questions—is a design limitation, not a fatal flaw, and the authors note it.\n\nThe LLM self-preference finding (the judge rates LLM translations above human gold, with a small A/B check where a human picked gold 39/50 times) is interesting and worth following up.\n\nWho this is for: people building LLM-as-a-judge metrics for long-form or literary text, and MT evaluation researchers generally. This is not a paper with a fatal flaw; it is a solid empirical contribution with an overly strong abstract given the baseline caveat. I would send it to peer review with a request for matched evaluation units, significance testing, and a toned-down claim. Conditional accept.","headline":"A careful, honest empirical study on LLM-based literary translation evaluation; the baseline comparison is weaker than the headline claim, but the core findings hold up and deserve a serious referee.","tokens_in":19806,"tokens_out":4034,"would_cite":true,"duration_ms":33983,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-step framework beats standard MT metrics for literary translation but trails humans on Korean honorifics.","keywords":["literary translation","machine translation evaluation","LLM-as-a-judge","rubric-based evaluation","question-answering evaluation","Korean honorifics","English-to-Korean","interpretable metrics"],"falsifier":"Evaluate the same translations at matching granularity—for example, compute baseline metrics on full paragraphs by averaging or concatenating sentences, or run RULER/VERSE on single sentences—and check whether the reported correlation advantage over COMET, BERTScore, BLEURT, and GEMBA-MQM persists; if the advantage disappears or reverses, the central claim fails.","tokens_in":18770,"feed_emoji":"📖","tokens_out":6698,"duration_ms":48059,"temperature":0.7,"pith_summary":"To automate the evaluation of literary machine translation, this paper proposes a two-step framework that scores English-to-Korean translations on both general quality and story-specific literary fidelity. The first step, RULER, rates paragraphs with a human-crafted rubric covering lexical choice, honorifics, syntax/grammar, and content accuracy. The second step, VERSE, has one LLM generate story-specific literary questions and another verify whether the translation satisfies them. The paper reports that these fine-grained scores correlate with expert human judgments more strongly than traditional machine-translation metrics such as COMET, BERTScore, BLEURT, and GEMBA-MQM, though not as well as human annotators agree with each other. The authors present this as evidence that automated literary translation evaluation is feasible but not yet a replacement for human judgment, particularly on culturally loaded features like Korean honorifics.","feed_headline":"Two-step evaluator beats standard MT metrics for literary translation","feed_subtitle":"A rubric-and-question-answering pipeline gives fine-grained quality scores, but still misses human-level judgment on Korean honorifics.","key_machinery":"RULER is a four-criteria Likert-scale rubric (1–5) for lexical choice, proper use of Korean honorifics in dialogue, syntax and grammar, and content accuracy, based on the Multidimensional Quality Metrics (MQM) framework. VERSE is a two-agent verification module: an LLM generates story-specific literary questions from a short story summary, and a second LLM grades each question on a 1–3 scale. Together they turn 'quality of a literary translation' into a vector of interpretable scores, and the paper's key evidence is the correlation of those scores with expert human annotations.","core_discovery":"The central claim is that an automated two-stage pipeline—rubric-based rating (RULER) followed by story-specific question-answering verification (VERSE)—can evaluate literary machine translation in a fine-grained and interpretable way, and that it correlates with expert human judgment better than standard machine-translation metrics. Using 15 English short stories translated to Korean, with 725 aligned paragraphs and three literary experts as annotators, the authors find that the framework outperforms COMET, BERTScore, BLEURT, and GEMBA-MQM on correlation with human scores across all RULER criteria and on VERSE. The claim is qualified: the framework does not reach inter-human agreement, and the honorific category is the clearest failure mode, where LLM judges miss serious mistakes. The paper also reports that LLM evaluators tend to rate translations produced by other LLMs higher than human translations, and that chain-of-thought prompting does not improve evaluation accuracy while removing the rubric hurts more than removing the reference translation.","pith_inferences":["The honorific failure mode likely extends to other register-rich languages (Japanese, Javanese, Thai); a testable prediction is that the gap between LLM judges and humans widens with the complexity of the target language's honorific system.","If the LLM-prefers-LLM bias is model-family-specific rather than a general quality confusion, then pairing translator and evaluator from the same model family could further inflate scores; this could be tested by cross-family evaluations.","The finding that reasoning prompts do not help suggests the bottleneck is missing cultural knowledge, not reasoning capability; injecting explicit honorific rules into the rubric might reduce the gap and is a concrete extension of RULER."],"forward_implications":["Literary translation quality can be scored as a profile of interpretable dimensions—lexical choice, honorifics, syntax, content, and story-specific literary criteria—rather than a single opaque number, giving translators and pipeline developers actionable feedback.","The framework can be used as an automated signal in agentic translation systems, letting developers test multiple translation attempts without running a human evaluation each time, though final screening for culturally sensitive features should remain human.","Rubrics are the stronger lever: removing the rubric degrades scores more than removing the reference translation, so designing explicit, language-specific evaluation guidelines matters more than supplying examples.","Chain-of-thought prompting and few-shot examples do not consistently help LLM evaluators in this domain, so simply giving the judge more reasoning time or examples is not the path to closing the gap with human agreement."],"supporting_citations":[{"why":"Supplies the Multidimensional Quality Metrics (MQM) framework that RULER's four rubric categories are modeled on.","marker":"(Mariana, 2014)"},{"why":"Provides GEMBA-MQM, the GPT-4-based baseline metric the framework is compared against and outperforms.","marker":"(Kocmi and Federmann, 2023)"},{"why":"Provides the COMET-22 baseline used in the comparison; a standard neural MT metric the framework claims to beat.","marker":"(Rei et al., 2022)"},{"why":"Provides BLEURT, a learned reference-based metric used as a baseline.","marker":"(Sellam et al., 2020)"},{"why":"Provides BERTScore, a token-overlap semantic metric used as a baseline.","marker":"(Zhang* et al., 2020)"},{"why":"Motivates the literary translation setting and the few-shot, document-level translation prompt setup used to generate the evaluated translations.","marker":"(Karpinska and Iyyer, 2023)"},{"why":"Establishes the question-answering paradigm for MT evaluation that VERSE extends to literary texts.","marker":"(Krubiński et al., 2021)"},{"why":"Documents that LLMs are biased evaluators, supporting the paper's observed LLM-prefers-LLM-translation bias.","marker":"(Wang et al., 2023b)"}],"fun_headline_variants":["Two-step literary MT evaluator beats standard metrics, but not humans","Rubric+QA eval outranks COMET for fiction, stumbles on honorifics","LLM-based literary translation checker: fine-grained yet biased","Automated literary translation scoring: outpaces MT metrics, lags experts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The fair-comparison assumption: the baselines (COMET, BERTScore, BLEURT, GEMBA-MQM) are sentence-level metrics while RULER and VERSE evaluate paragraphs, and the paper itself says the comparison 'may not be entirely appropriate'; if the baseline numbers are not fairly obtained, the headline correlation advantage is not established.","fun_headline_variants_meta":{"raw":{"variants":["Two-step literary MT evaluator beats standard metrics, but not humans","Rubric+QA eval outranks COMET for fiction, stumbles on honorifics","LLM-based literary translation checker: fine-grained yet biased","Automated literary translation scoring: outpaces MT metrics, lags experts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000771,"raw_usage":{"total_tokens":3367,"prompt_tokens":850,"completion_tokens":2517,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":466,"completion_tokens_details":{"reasoning_tokens":2437}},"tokens_in":466,"tokens_out":2517,"duration_ms":19165,"temperature":1.0,"reasoning_tokens":2437,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:27:30.208343+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the same translations at matching granularity—for example, compute baseline metrics on full paragraphs by averaging or concatenating sentences, or run RULER/VERSE on single sentences—and check whether the reported correlation advantage over COMET, BERTScore, BLEURT, and GEMBA-MQM persists; if the advantage disappears or reverses, the central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Multidimensional Quality Metrics (MQM) framework that RULER's four rubric categories are modeled on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the COMET-22 baseline used in the comparison; a standard neural MT metric the framework claims to beat."}],"review_version":1}