{"id":"9818eed8-0d97-43c1-8ea2-22c893959c9a","arxiv_id":"2509.03809","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An evaluation framework for document-level MT that aligns source and target sentences via dynamic programming and averages 1-4 chunk sliding-window scores, reporting 0.929 Pearson correlation with human MQM and enabling CPO/GRPO training.","lead":"The paper introduces a two-stage evaluation framework for document-level machine translation that aligns source and translated sentences automatically, then scores the document in sliding windows of 1 to 4 sentences. This matters because current metrics assume one-to-one sentence correspondence, which whole-document LLM translations routinely violate, and a reliable document-level score could improve both evaluation and training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Many-to-one mappings are not resolved; sliding windows only partially recover the target content, so the abstract's central claim overstates what Align-then-Slide does.","rationale":"The central claim hinges on the alignment being faithful enough that sliding-window scores reflect translation quality. The paper explicitly acknowledges that Stage 1 cannot handle many-to-one mappings, yet the abstract claims they are resolved. Our analysis shows Stage 2 cannot fully repair this: the reconstructed target sequence has empty placeholders for non-matched source sentences, so the sliding window only recovers the correct target if the window covers the entire merged group and includes the assigned source. Averaging 1–4 chunk scores, the 1-chunk scores always penalize the placeholder, and partial windows distort scores. This is a systematic bias against systems that legitimately merge sentences, which is exactly the kind of divergence doc-mt evaluation must handle. Since the correlation studies use only 7 and 6 systems and do not measure the frequency of many-to-one mappings, the reported high correlations do not establish that the bias is harmless. A controlled test can determine if the method truly handles many-to-one. This aligns with the reader's weakest assumption, and the verdict remains CONDITIONAL pending correction of the abstract, code/data release, and the proposed verification.","tokens_in":9386,"tokens_out":10327,"duration_ms":98899,"concrete_test":"Construct a controlled document pair where the reference translation merges two or more source sentences into one target sentence (a many-to-one mapping), and create a hypothesis that is an exact copy of that reference. Run Align-then-Slide and compare its score against a baseline where the same content is translated with a one-to-one sentence split. If the merged hypothesis scores significantly lower than the split hypothesis, the framework fails to neutralize many-to-one mappings, confirming the abstract overclaim. Also, independently re-derive the DP recurrence from Section 2.1 and re-run the WMT2020 correlation; if the recurrence described is implemented literally, it violates the stated path constraints, and the corrected DP may change the reported 0.929 Pearson correlation.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that Align-then-Slide resolves many-to-one mappings is contradicted by Section 2.2, which admits 'Stage 1 Align cannot handle many-to-one mappings.' The DP path constraint (y must increase by exactly 1 each step, x non-negative) forces each target sentence to be assigned to at most one source sentence; when several source sentences are translated as a single target, the unmatched source sentences are replaced by empty placeholders (Algorithm 1, lines 17–18). The paper argues that n-chunk sliding windows 'neutralize' this conflict (Figure 4), but this only works when a window happens to contain exactly the assigned source sentence and all its merged siblings. For a group of L source sentences, the 1-chunk setting always scores the empty placeholder, and windows that include only a subset of the group receive either the full group translation (if they contain the assigned source) or only placeholder content (if not), injecting systematic bias. The abstract's claim of resolving 'many-to-one mappings' is therefore unsupported; documents with frequent merging will be penalized, which could distort system rankings depending on how often systems merge. The reported correlations (0.929, 0.943) come from only 7 and 6 systems and do not report the prevalence of many-to-one mappings in the test sets, so the impact of this bias is unknown.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Align-then-Slide, a two-stage evaluation framework for document-level MT. Stage 1 automatically infers sentence-level source–target correspondences using a similarity matrix (COMETKiwi or LaBSE) and a dynamic-programming path, then reconstructs the target to have the same number of sentences as the source, padding omissions and concatenating one-to-many matches. Stage 2 evaluates the aligned document by averaging COMET scores over sliding windows of 1, 2, 3, and 4 consecutive sentences. The authors report Pearson correlations of 0.929 with expert MQM rankings on WMT2020 Chinese→English (7 systems) and 0.943 with professional human rankings on a newly constructed 6-model real-world test set. They further use the metric to select training data and as a reward model for CPO/GRPO, reporting that the resulting models are preferred by humans over an SFT baseline. The central claim is that the framework is an accurate, robust, and actionable evaluation tool for ultra-long document-level MT.","tokens_in":9685,"tokens_out":6457,"duration_ms":70340,"significance":"If the claimed correlations hold, Align-then-Slide would be a practical, fully automatic evaluation method for long document-level MT, with the additional useful property of supplying preference signals for RL training. The paper has notable strengths: it uses external human judgments (MQM and professional pairwise rankings) as ground truth rather than tuning the metric to a target; it includes ablations showing stability to sentence segmentation tools and alignment models; and the training study is verified by independent human preference, not just metric self-agreement. The method is simple enough to be reproducible. However, the central empirical support rests on very small numbers of systems, the few evaluation settings omit the most directly relevant prior document-level metrics, and the handling of many-to-one mappings is overstated relative to what the algorithm actually does.","major_comments":[{"comment":"The abstract claims that the Align stage resolves 'many-to-one/one-to-many mappings', but §2.2 explicitly states that Stage 1 cannot handle many-to-one mappings, and Algorithm 1 assigns an empty placeholder to each source sentence sharing a target (matched set empty). The argument that 2–4-chunk sliding windows 'neutralize' this conflict is only partially supported: for a group of L source sentences merged into one target, the 1-chunk setting always scores the placeholder for unassigned sources, and windows of size smaller than L necessarily miss part of the group. Thus the averaged 1–4-chunk score still carries a systematic bias whose magnitude depends on the prevalence of many-to-one mappings, a quantity not reported for either test set. This directly undermines the abstract's central claim and the framework's claimed generality. The authors should either revise the claim or provide qu","section":"Abstract; §2.2; Algorithm 1; Figures 3–4"},{"comment":"The central correlation results rest on only 7 systems (WMT2020) and 6 systems (real-world test set). No confidence intervals, p-values, or alternative rank-based measures are reported for the real-world set. With n=6, a correlation of 0.943 can be sensitive to a single pairwise swap, and the absence of uncertainty quantification makes it impossible to judge whether the apparent superiority over COMET20 is statistically meaningful. Please report CIs/significance, scatterplots, and per-system agreement, and note the small-sample caveat in the text.","section":"§4.1; Tables 1–3"},{"comment":"The paper motivates Align-then-Slide as an advance over SLIDE (Raunak et al., 2024) and doc-metrics (Vernikos et al., 2022), and explicitly distinguishes the fixed-stride hierarchical chunking from SLIDE's dynamic window. Yet the correlation study compares only against sentence-level COMET20; no result for SLIDE, doc-metrics, or any other document-level metric is reported. Without these baselines, the claimed advantage of the n-chunk sliding component is not demonstrated. The authors should add these comparisons under the same document reassembly and alignment conditions, or justify their omission.","section":"§4.1.1–4.1.2; §1; §2.2"},{"comment":"The dynamic-programming definition is internally inconsistent. The text states that the y-position must increase by exactly 1 each move and the x-position by a non-negative amount, which implies that the predecessor of (i,j) should be (i', j−1) for any i' ≤ i. However, the recurrence is written as taking the maximum over dp[i−1][j−1], dp[i−1][j−2], ..., dp[i−1][0], which corresponds to a different constraint on the x-increment. This makes the algorithm under-specified and hard to reproduce. Please correct the recurrence and ensure that Algorithm 1 and the example path match the stated constraints.","section":"§2.1, DP description"}],"minor_comments":[{"comment":"The human evaluation protocol for the training study is not described: number of documents/segments judged, number of translators, judgment scale, tie handling, and whether the comparison was blind. These details are needed to interpret the claimed superiority of CPO/GRPO over SFT.","section":"§4.2; Table 4"},{"comment":"The WMT2020 test set is described as sentence-aligned and free of omissions. This should be stated as a limitation: the primary benchmark does not exercise the alignment difficulties that motivate the framework; only the real-world test set does, and its mapping statistics are not reported.","section":"§4.1.1"},{"comment":"The real-world test set contains only 50 document pairs selected from CommonCrawl. Please report document lengths, domain distribution, and the selection criteria in more detail, and indicate whether the 50 pairs are sufficient to support system-level correlation claims.","section":"§3.1.1; Appendix A"},{"comment":"The phrase 'the y-position must increase by 1 each move ... resulting in n points' is confusing because the path has n points while the reconstructed target has m sentences. Clarify that the path length is n and the reconstruction maps these to m source positions.","section":"§2.2"},{"comment":"The paper refers to 'Pearson correlation between rankings'. Since ranks are used, this is effectively Spearman's rho; please report it explicitly or use raw scores, whichever is intended.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant and timely problem, and the training-utility results are a useful addition. The main reasons for major revision are the unresolved mismatch between the abstract and the algorithm's actual handling of many-to-one mappings, the very small system-level samples without uncertainty quantification, and the absence of comparisons to the two most directly relevant prior document-level metrics (SLIDE and doc-metrics). These are addressable in revision. I also suggest the authors consider whether 'ultra-long' is justified given the reported test sizes and document lengths."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the core idea is worth taking seriously, but the paper's own Section 2.2 contradicts the abstract. It says 'Stage 1 Align cannot handle many-to-one mappings,' while the abstract claims those mappings are resolved by the Align stage. The sliding window only partly patches this. So the framing is over-claimed, but the underlying method is a sensible, useful contribution.\n\nWhat's new: the combination of source-anchored DP sentence alignment with 1- to 4-chunk sliding windows (stride 1) is a specific configuration I haven't seen in SLIDE or doc-metrics. Using the metric as a CPO/GRPO reward and for data selection is a nice extension. The reported correlations — 0.929 on WMT20 Zh-En with seven systems, 0.943 on their real-world test with six — are high. The ablation shows the method is robust to segmentation tool and alignment model, which is good evidence that the pipeline isn't fragile.\n\nSoft spots, in proportion. First and most important: the many-to-one issue. The DP path forces the target index to increase by exactly 1 each step, so a source sentence that shares a target with a neighbor gets a placeholder. The 2- to 4-chunk windows can recover the content when the window happens to contain the assigned source and its merged siblings, but for a group of L merged source sentences, the 1-chunk scoring always sees the placeholder, and partial windows get distorted source-target pairs. If systems differ in how often they merge sentences, this injects bias into the ranking. The test sets' prevalence of many-to-one mappings isn't reported, so the impact is unknown. The abstract should say 'mitigates,' not 'resolves.'\n\nSecond, the correlation evidence is thin: 7 and 6 data points, no error bars or significance tests. High Pearson values on that sample size are easy to move. Third, the paper never compares ASD to SLIDE or doc-metrics on the correlation task, even though those are the closest prior methods. That's a notable omission for a paper claiming a complete framework. Fourth, the DP description is internally inconsistent (the text says y must increase by 1, but the recurrence implies something different); the reader can infer the intended algorithm from Figure 2, but the formalism should be fixed. Fifth, the training study's human evaluation protocol is under-specified — no details on number of annotators, pairwise procedures, or agreement.\n\nBottom line: this is a solid working paper with a real problem and a plausible solution. It deserves a serious referee, but the authors need to correct the abstract, add baselines, and report uncertainty before I'd trust the rankings. I'd send it to peer review, not desk reject.","headline":"A practical doc-MT evaluation pipeline with high reported correlations, undermined by an abstract that promises more than the aligner delivers.","tokens_in":10200,"tokens_out":3817,"would_cite":true,"duration_ms":36449,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rebuilding the sentence correspondence before scoring is what lets document-level MT evaluation rank systems the way expert human judges do.","keywords":["document-level machine translation","evaluation metric","dynamic programming alignment","sliding window","quality estimation","preference optimization","large language models","translation ranking"],"falsifier":"Build a test set in which most translation divergence is many-to-one (two or more source sentences merged into one target sentence), have professional translators rank the systems, and compare with Align-then-Slide. If the Pearson correlation between the metric's ranking and the human ranking falls below what an unaligned, whole-document score already achieves, then the sliding-window patch does not actually neutralize the many-to-one mappings the DP cannot represent.","tokens_in":9267,"feed_emoji":"📄","tokens_out":11682,"duration_ms":106999,"temperature":0.7,"pith_summary":"The paper argues that you cannot reliably score a whole-document translation until you know which source sentences correspond to which target sentences. It introduces Align-then-Slide: a dynamic-programming step that reconstructs a one-to-one source–target sentence alignment (patching omissions with placeholders and merging repeated target sentences), followed by a sliding-window pass that scores the aligned document at 1-, 2-, 3-, and 4-sentence granularity and averages the results. On a standard Chinese–English benchmark the system rankings correlate at 0.929 with expert human quality judgments, and on a newly built real-world test set at 0.943. The same scores can be converted into preference pairs that improve translation models under both offline preference optimization (CPO) and online reinforcement learning (GRPO) relative to a supervised baseline. If correct, this gives document-level MT an accurate evaluation signal that works even when systems omit, merge, or split sentences.","feed_headline":"Rebuild the alignment, then slide: document-level MT metric hits 0.929","feed_subtitle":"On a standard Chinese-English benchmark, the framework's rankings correlate at 0.929 with expert human quality scores.","key_machinery":"The load-bearing object is the dynamic-programming alignment path (Algorithm 1). Given an m by n similarity matrix, the path starts at (0,0), ends at (m-1, n-1), increases the target index by exactly 1 at every move, and increases the source index by any non-negative amount. That constraint encodes omissions (an unmatched source sentence is skipped) and one-to-many mappings (one source sentence gathers several target sentences); the reconstruction step anchors on the source, so the rebuilt target always has the same number of sentences as the source. The second object is the n-chunk sliding window: with stride 1, windows of 1, 2, 3, and 4 sentences are scored by an off-the-shelf quality esti","core_discovery":"At the heart of the paper is the claim that the hardest part of document-level MT evaluation is not scoring but alignment. The authors propose to solve that explicitly: segment source and translation, build an m by n matrix of sentence-pair similarity scores, and then use dynamic programming to find an optimal path in which every target sentence is assigned to exactly one source sentence (or to a placeholder when omitted), while several target sentences may attach to the same source sentence. The reconstructed target now has the same sentence count as the source, and it is scored with sliding windows of 1 through 4 sentences, averaged. This two-stage procedure yields system-level rankings th","pith_inferences":["The same reconstruct-then-slide recipe should transfer to other long-document generation tasks—summarization, long-form QA, subtitling—where system outputs disagree in segmentation, as long as a sentence-level similarity signal exists.","Relaxing the DP so the target index may also stay fixed would let the alignment stage represent many-to-one merges directly, potentially removing the sliding-window patch and simplifying evaluation to a single pass.","The n-chunk score profile could be used as a feedback tool for translators: a deep 1-chunk dip identifies skipped sentences, while a 2–4 chunk rebound flags merging artifacts, giving targeted revision advice instead of just one number.","Because the metric is cheap to run per sample, it could be used at inference time to rerank multiple candidate document translations, a straightforward extension of its GRPO reward-model role."],"forward_implications":["Document-level MT evaluation no longer requires the test set to be pre-aligned sentence by sentence; any pair of documents can be evaluated after automatic re-alignment.","Whole-sentence omissions, one-to-many splits, and variable target sentence counts stop being an error condition and become part of what the metric measures.","Metric scores can be turned into preference pairs and reward signals, so document-level reinforcement learning of translators no longer waits for a working sentence-level evaluator.","The multi-granularity window gives a diagnostic signature: 1-chunk scores expose omissions while 2-, 3-, and 4-chunk scores moderate merge artifacts, making failure modes visible in the score profile.","Because the alignment step is a module, swapping the similarity model or the segmenter changes scores only slightly, making the framework robust to its components."],"supporting_citations":[{"why":"Defines the sliding-window document evaluation strategy that the n-chunk pass extends to multiple fixed chunk sizes.","marker":"(Raunak et al., 2024)"},{"why":"Establishes a prior document-level metric approach that Align-then-Slide is positioned against.","marker":"(Vernikos et al., 2022)"},{"why":"Supplies the expert human quality judgments used as the gold-standard ranking for the benchmark comparison.","marker":"(Freitag et al., 2021)"},{"why":"Defines the neural evaluation metric family from which the scoring backbones used in the sliding-window pass are drawn.","marker":"(Rei et al., 2020a)"},{"why":"Provides the reference-free quality estimator used to populate the alignment similarity matrix before DP search.","marker":"(Rei et al., 2020b)"},{"why":"Provides the cross-lingual sentence embedding model used as an alternative alignment scorer in the robustness ablation.","marker":"(Feng et al., 2022)"},{"why":"Provides the alternative sentence segmenter used in the ablation of the alignment stage.","marker":"(Wicks and Post, 2021)"},{"why":"Defines CPO, the preference-optimization method that consumes Align-then-Slide preference pairs in the training study.","marker":"(Xu et al., 2024)"},{"why":"Defines GRPO, the online reinforcement-learning algorithm in which Align-then-Slide serves directly as the reward model.","marker":"(Shao et al., 2024)"}],"fun_headline_variants":["Align first, then slide: doc-MT metric tops at 0.929","New evaluation framework fixes alignment for long-document MT","Slide after aligning: 0.929 correlation with human MQM","Better doc-MT scoring: align-then-slide achieves 0.929","Evaluation that realigns: doc-MT metric hits 0.929"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The whole score rests on the DP reconstruction being correct, and that reconstruction only allows a source sentence to be skipped or several target sentences to answer for one source; when a single target sentence covers multiple source sentences, the extra sources are labeled as missing and every later chunk score is computed on a distorted input.","fun_headline_variants_meta":{"raw":{"variants":["Align first, then slide: doc-MT metric tops at 0.929","New evaluation framework fixes alignment for long-document MT","Slide after aligning: 0.929 correlation with human MQM","Better doc-MT scoring: align-then-slide achieves 0.929","Evaluation that realigns: doc-MT metric hits 0.929"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000209,"raw_usage":{"total_tokens":1244,"prompt_tokens":747,"completion_tokens":497,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":401}},"tokens_in":491,"tokens_out":497,"duration_ms":5428,"temperature":1.0,"reasoning_tokens":401,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:39:07.473700+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a test set in which most translation divergence is many-to-one (two or more source sentences merged into one target sentence), have professional translators rank the systems, and compare with Align-then-Slide. If the Pearson correlation between the metric's ranking and the human ranking falls below what an unaligned, whole-document score already achieves, then the sliding-window patch does not actually neutralize the many-to-one mappings the DP cannot represent.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes a prior document-level metric approach that Align-then-Slide is positioned against."}],"review_version":1}