{"id":"3a08cd16-ee1d-4497-b56f-09a290a384ed","arxiv_id":"2505.13554","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A learned source-feature decider routes each sentence to either an NMT model or an LLM, improving average translation quality over both single systems and a QE-based baseline while using the LLM for only about 20-30% of sentences.","lead":"This paper introduces a small source-sentence classifier that decides, per sentence, whether a fast NMT model or a large language model should translate it. On multilingual tests the hybrid improves average quality over either model alone while calling the LLM for roughly a quarter of sentences, but the evidence rests on in-house data and small test sets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"JDM is trained on wmt22-comet-da labels and evaluated with the same metric, with the oracle ceiling also defined by it; if COMET diverges from human judgment, the reported gains and 'minimal LLM usage' claim are biased in one direction, and BLEURT is only a partially independent proxy.","rationale":"The reader's weakest_assumption identifies the same point: the training labels and the evaluation metric are both wmt22-comet-da. I agree this is the single most load-bearing concern because it threatens every reported number and the oracle ceiling, not just one table. The paper does include BLEURT, which is not used in training, and this is real independent evidence; the consistent JDM advantage across both metrics weakens the concern somewhat. However, BLEURT is itself a neural metric trained on human judgments and can share systematic biases with COMET, especially on informal and idiomatic content where the Literary test set is designed to show LLM advantages. A human evaluation would settle whether the COMET-based labels and evaluation are faithful. I also note that Tables 2 and 3 show JDM slightly below NMT on En2Zh Tech and Flores, which contradicts the abstract's 'optimal' wording, but the core verdict remains conditional: accept if the authors add human evaluation or an independent metric not used in training, plus significance tests. Since the reader already reached CONDITIONAL, I recommend no change to the verdict.","tokens_in":11724,"tokens_out":8924,"duration_ms":94516,"concrete_test":"Select a stratified sample of roughly 200 sentences per test set from News, Literary, Tech, Subtitle, and Travel for Zh2En and En2Zh (about 2,000 sentences total). Collect NMT, LLM, QET, and JDM outputs and have bilingual annotators perform pairwise preference or MQM judgments. Compare the human-judged average quality of JDM against NMT, LLM, and QET, and compute a human-judged oracle by choosing the better of NMT and LLM per sentence. If JDM's human-judged gains over NMT and QET do not mirror the wmt22-comet-da gains, or if the COMET oracle disagrees with the human oracle on a substantial fraction of sentences, the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that JDM achieves optimal translation quality with minimal LLM usage depends on wmt22-comet-da being a faithful proxy for translation quality, because the metric plays three connected roles. In Section 3.2, wmt22-comet-da scores NMT and LLM outputs and defines the positive/negative training labels for the JDM decider via Equation 1. In Section 4, the same metric is the primary DA evaluation. In Tables 2 and 3, the oracle upper bound is defined as the better of NMT and LLM according to wmt22-comet-da. If COMET's ranking diverges from human preferences on Literary, Tech, Subtitle, or Travel sentences, then the decider is trained to optimize the wrong objective, the headline gains are biased in the same direction as the labels, and the oracle is not an upper bound on human-judged quality. BLEURT is not used in training and is therefore partially independent, but it is also a learned metric trained on human judgments and can share COMET's systematic biases. The paper reports no human evaluation, so this circularity is untested. The reported numbers also contain a smaller inconsistency with the 'optimal' wording: on En2Zh Tech and Flores, JDM is slightly below NMT alone (Table 3), so the integration can hurt on individual domains. That is secondary; the metric-overlap issue is the more fundamental risk to the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper addresses the problem of combining NMT and LLM translation to improve quality while limiting LLM use. It proposes two routing methods: PPLT, which uses source-sentence perplexity from a small LM to decide when to invoke the LLM, and JDM, a binary classifier trained on source-sentence features to predict when the LLM output is better than the NMT output. The JDM training labels are derived from wmt22-comet-da scores using the conditions in Equation (1). The paper compares these methods against QET (based on wmt22-cometkiwi-da quality estimation) and single-system baselines on four language pairs (Zh2En, En2Zh, Ja2En, De2En) and several test sets (News, Flores, self-constructed Literary and Tech, plus Subtitle and Travel). The main finding is that JDM achieves the best average DA and BLEURT among the compared integration methods, with LLM usage around 22-30%, and outperforms QET on average across language pairs. The paper also contributes manually constructed literary and technical test sets.","tokens_in":12005,"tokens_out":7270,"duration_ms":68963,"significance":"The contribution is practically relevant: the JDM router uses only the source sentence, so it avoids running a QE model on every NMT output and can skip NMT when the LLM is chosen, reducing inference cost relative to QET. The experimental coverage is broad for a paper of this type (four directions, multiple domains, two metrics), and the reported pattern that JDM wins on average while using comparable LLM calls is consistent across language pairs. The analysis in Appendix A, showing that the router preferentially sends informal expressions to the LLM, gives some qualitative insight. However, the core quantitative claim is currently supported only by automatic metrics, and the overlap between the training metric and the evaluation metric is a genuine concern; the paper's significance would be materially strengthened by a human evaluation and by error bars on the headline comparisons.","major_comments":[{"comment":"The JDM decider is trained on positive/negative labels computed from wmt22-comet-da scores via Equation (1), and the same metric is used as the primary evaluation (DA) and as the oracle upper bound in Tables 2 and 3. If wmt22-comet-da's ranking of translations diverges from human preferences on Literary, Tech, Subtitle, or Travel sentences, then the training labels, the reported quality gains, and the oracle ceiling are all biased in the same direction. The paper reports no human evaluation. BLEURT is a partially independent proxy because it is not used in training, but it is also a learned metric and the paper does not report whether the JDM gains on BLEURT are statistically reliable. Please add a human evaluation on a sample of sentences, or at least a sensitivity analysis that varies the evaluation metric, so that the central quality claim does not rest on a single metric that also generated the training signal.","section":"§3.2, §4"},{"comment":"The abstract's claim of 'optimal translation performance with minimal LLM usage' is too strong. In Table 3, JDM's DA is below NMT alone on En2Zh Tech (85.39 vs 86.30) and Flores (87.76 vs 87.88). In Tables 7 and 8, JDM's LLM usage on Ja2En and De2En is slightly higher than QET (22.42% vs 21.95% and 23.32% vs 22.06%). Thus JDM does not dominate all baselines on every domain or use the fewest LLM calls in all settings. Please revise the abstract and conclusion to say that JDM achieves the best average performance among the compared policies with LLM usage comparable to or slightly higher than the QET baseline, rather than claiming optimality and minimality in an unqualified sense.","section":"Abstract and §4 (Tables 2-3, 7-8)"},{"comment":"The paper does not report confidence intervals or significance tests for the DA/BLEURT differences. For example, in Table 2 the Zh2En average DA difference between JDM (78.81) and NMT (77.29) is 1.52 points, and the Literary test set has only 500 sentences. Without a bootstrap or other significance test, it is unclear whether the observed differences are within noise. This matters for the central claim that JDM improves over both single systems and QET. Please add significance tests or confidence intervals for the main comparisons, especially for the average scores across domains.","section":"§4, Tables 2, 3, 7, 8"}],"minor_comments":[{"comment":"The LLM-SFT row appears to be missing the BLEURT score on the Tech test set; the entry reads '78.20 100.00%' with no BLEURT value, whereas the other columns in that row show DA, BLEURT, and LLMp.","section":"Table 9"},{"comment":"The citation 'A Conneau. 2019. Unsupervised cross-lingual representation learning at scale' is incomplete; the XLM-R paper is by Conneau et al. and was published at ACL 2020. Please correct the reference.","section":"References"},{"comment":"The phrase 'obtain best in results for MT' in the introduction is grammatically awkward and should be rephrased.","section":"§1 and §4"},{"comment":"The threshold sensitivity discussion is qualitative; please clarify how the thresholds were validated across domains and whether the same thresholds are used for all test sets within a language pair, since Table 5 lists only one threshold per language pair.","section":"Appendix F"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a practical, useful paper on routing between NMT and LLM using a source-only classifier. The new piece is JDM, a small xlm-roberta-based decider trained on COMET-derived labels from source sentences alone, and it consistently beats the QET baseline on average across four language pairs and several domains, with BLEURT pointing the same direction. That is a real result for someone running translation services: a roughly 25% LLM call rate with quality at or above either single system. The PPLT baseline is also a nice sanity check that source complexity alone gets you part of the way.\n\nThe main thing to know: the abstract's \"optimal translation performance with minimal LLM usage\" is not what the tables show. LLM usage is set in advance to about 25% by threshold, so \"minimal\" is a chosen operating point, not an optimum. On En2Zh, JDM actually uses slightly more LLM than QET, and on En2Zh Tech and Flores it lands slightly below NMT alone. Those are secondary issues, but the wording should be tempered.\n\nThe more fundamental soft spot is the COMET circularity. The same wmt22-comet-da metric creates the training labels, sets the oracle upper bound, and reports the main DA scores. If COMET's ranking diverges from human judgment on literary, tech, subtitle, or travel sentences, the decider is optimized for the wrong objective and the gains are biased in that same direction. BLEURT is partially independent and does corroborate the direction, but it is also a learned metric. The paper reports no human evaluation, so the assumption that COMET is a faithful proxy is untested. That is a genuine limitation, not a manufactured one. It does not kill the paper, but it caps how strongly the central claim can be stated.\n\nThe other gaps are smaller: 500-sentence test sets with no significance tests or confidence intervals, thresholds chosen per language pair and tuned to hit the same LLM rate, and no code or NMT models released. The authors do promise to open-source the constructed test sets, which is good.\n\nWho this is for: anyone working on hybrid MT deployment, LLM cost reduction, or model routing. It is not a theoretical contribution and does not change how we think about translation quality, but it is evidence that a cheap source-side decider can work in practice.\n\nMy recommendation: send it to peer review. It deserves referee time. The authors should be asked to temper the abstract, add uncertainty estimates, and break the metric overlap with a human evaluation or at least an independent metric not used in training. With those revisions, the practical claim would be solid.","headline":"Practical routing paper with consistent average gains; the headline overstates the tables and the COMET label/eval overlap is a real, untested weakness, but it deserves a serious referee.","tokens_in":12607,"tokens_out":1548,"would_cite":true,"duration_ms":18796,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One small classifier picks NMT or LLM per sentence, beats both systems","keywords":["machine translation","LLM","NMT","hybrid translation","source-sentence decider","translation routing","quality estimation","COMET"],"falsifier":"Run a human preference study on the Literary, Tech, Subtitle, and Travel test sets: if human raters do not consistently prefer JDM outputs over the better single system, or if human preferences disagree with wmt22-comet-da on the sentences JDM routes to the LLM, the central claim of optimal translation quality at minimal LLM usage is not supported.","tokens_in":11492,"feed_emoji":"🌐","tokens_out":11649,"duration_ms":99818,"temperature":0.7,"pith_summary":"The paper claims that a translation system does not need to score every output of its neural machine translation (NMT) model to decide when a large language model (LLM) should take over: a small classifier that looks only at the source sentence can route each sentence to either model. The authors compare their Joint Decision-making (JDM) decider with QE-threshold routing (QET), which calls the LLM whenever an automatic quality estimate of the NMT output is low, and with a perplexity threshold on the source text (PPLT). On Zh2En, En2Zh, De2En, and Ja2En test suites, JDM reports the best average COMET-DA and BLEURT scores among the integrated methods while using the LLM on roughly 22% to 30% of sentences. The practical payoff is cost: most traffic stays on a fast NMT model, and the expensive LLM is reserved for the sentences where it actually helps.","feed_headline":"One small classifier picks NMT or LLM per sentence, beats both systems","feed_subtitle":"It uses only source text, cutting LLM calls to about 25% while beating quality-scoring routing on four language pairs.","key_machinery":"JDM (Joint Decision-making) is the central object: a binary classifier trained on positive samples where the NMT output is bad and the LLM is clearly better, meaning the NMT's COMET-DA score is below a first threshold and the LLM-minus-NMT DA gap is above a second threshold, plus negative samples drawn from the rest of the data. At inference the decider consumes only source-sentence features, so it replaces a quality-estimation model over the NMT output with a small classifier over the input text. The same metric, wmt22-comet-da, supplies the training labels, the reported DA scores, and the oracle ceiling, which is what makes the paper's quality claims internally coherent.","core_discovery":"The central claim is that a binary classifier trained offline on triples of source sentences with NMT and LLM quality scores can learn to predict, from the source sentence alone, when the LLM will beat the NMT model. At inference time the JDM decider runs before translation: if it says LLM, the sentence goes to the LLM, otherwise to NMT. The paper reports that this routing policy achieves the best average DA and BLEURT among all compared hybrid methods on four language-pair test suites with an average LLM usage between 22% and 30%, and that the call rate adapts by domain, rising on Literary and News where the LLM is strong and falling on Tech and Flores where the NMT model is stronger. The oracle upper bound, defined by choosing the better of the two outputs with wmt22-comet-da, shows that the decider leaves some headroom and is not selecting optimally in every case.","pith_inferences":["A natural next test is replacing the COMET-based training labels with human preference judgments; if human rankings disagree with COMET on the routed sentences, JDM's reported gains may not transfer to how readers perceive quality.","The paper's own category-3 failures, sentences that need context to translate, are a hard limit of sentence-level routing; a document-level decider that looks beyond the current sentence would be a direct extension.","Because the decider's signal appears to be largely about sentence complexity and domain, one could train it on one language pair and test it zero-shot on another to see how much of the routing signal is language-independent.","The threshold-setting procedure is a fixed quantile rule; learning the T1 and T2 thresholds per domain or per language pair from small labeled sets would likely improve the trade-off between quality and LLM usage."],"forward_implications":["Production MT services can capture most of the quality benefit of LLM translation while sending only about a quarter of sentences to the LLM, without running a per-sentence quality estimator.","A single fixed decider can serve mixed traffic because its LLM call rate rises in domains where the LLM is strong and falls where the NMT model is strong.","The generalization experiment suggests the decider can be reused with fine-tuned NMT and LLM models without retraining, as long as the two systems keep their complementary strengths.","The gap between JDM and the oracle upper bound implies that better routing or a confidence-aware fallback could improve hybrid quality further without increasing LLM usage."],"supporting_citations":[{"why":"Defines the QET baseline that invokes the LLM when a quality estimator scores the NMT output below a threshold, and provides the finding that LLMs excel in particular domains.","marker":"Hendy et al. (2023)"},{"why":"Supplies wmt22-comet-da, the metric used for JDM training labels, reported DA scores, and the oracle upper bound.","marker":"(Rei et al., 2022a)"},{"why":"Supplies wmt22-cometkiwi-da, the quality-estimation model that QET depends on and that is used to set QET's threshold.","marker":"(Rei et al., 2022b)"},{"why":"Provides the pretrained multilingual model that the JDM decider is fine-tuned from.","marker":"(Conneau, 2019)"},{"why":"Defines the instruction-tuned LLM used as the LLM translator in all experiments.","marker":"(Touvron et al., 2023)"},{"why":"Supplies the BLEURT metric used as the second automatic evaluation score alongside DA.","marker":"(Sellam et al., 2020)"}],"fun_headline_variants":["Small decider picks NMT or LLM per sentence, cuts LLM calls","Hybrid MT: source-only decider achieves best quality with less LLM","Classifier routes translation to NMT or LLM, beats alternatives","Source-based routing for hybrid NMT-LLM, minimal LLM usage","One decider for NMT vs LLM, optimal quality at 25% LLM calls"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that wmt22-comet-da, the metric used to label the decider's training data, to score all reported results, and to define the oracle ceiling, agrees with what human judges would prefer; if COMET's rankings diverge from human preferences, the training labels, the reported quality gains, and the oracle are all biased in the same direction.","fun_headline_variants_meta":{"raw":{"variants":["Small decider picks NMT or LLM per sentence, cuts LLM calls","Hybrid MT: source-only decider achieves best quality with less LLM","Classifier routes translation to NMT or LLM, beats alternatives","Source-based routing for hybrid NMT-LLM, minimal LLM usage","One decider for NMT vs LLM, optimal quality at 25% LLM calls"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000576,"raw_usage":{"total_tokens":2696,"prompt_tokens":902,"completion_tokens":1794,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":1691}},"tokens_in":518,"tokens_out":1794,"duration_ms":13453,"temperature":1.0,"reasoning_tokens":1691,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:26:54.309203+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a human preference study on the Literary, Tech, Subtitle, and Travel test sets: if human raters do not consistently prefer JDM outputs over the better single system, or if human preferences disagree with wmt22-comet-da on the sentences JDM routes to the LLM, the central claim of optimal translation quality at minimal LLM usage is not supported.","supporting_citations":[],"review_version":1}