{"id":"616bf234-bb4e-4fd2-85de-9e7826296704","arxiv_id":"2412.18707","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"For literary machine translation, filtering multiple human references by medium-to-high semantic similarity improves BLEU, COMET, and chrF++ over using an unfiltered reference set.","lead":"This paper tests whether training machine translation models on multiple human translations with medium or high semantic similarity beats training on an unfiltered set. It finds that filtering references by semantic similarity improves literary translation quality, and that multiple references can nearly match having more source texts.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Medium+High advantage may be an artifact of evaluating only on medium-similarity references; an unfiltered held-out test set is needed.","rationale":"I agree with the reader's weakest_assumption: the central Section 6.2 claim is conditioned on a test set whose reference similarity distribution is the same property used to construct the training filters. This is the most load-bearing point because the entire practical recommendation is a filtering rule, and the only evidence for it comes from an evaluation that has already applied that rule. The paper has real strengths—public code, three model families, significance testing, and a clear experimental grid—so the appropriate response is not rejection but a required validation step. If the unfiltered test set confirms the advantage, the conditional can be lifted; if not, the claim must be narrowed to 'optimal when downstream evaluation uses medium-similarity references.' The reader's conditional verdict already captures this, so I recommend no change.","tokens_in":14519,"tokens_out":6574,"duration_ms":59497,"concrete_test":"Using the public Par3 data and the authors' code, construct a held-out test set from the source paragraphs in the existing 20% split without applying the sim_p filter, scoring each source against its complete reference set. Retrain or reuse the Table 3 configurations: Unfiltered (77k sources) and Medium+High (68k sources) for mT5-large, LLaMA-2-7B, and Opus-MT, and recompute BLEU, COMETDA22, and chrF++ on this unfiltered test set, optionally also stratified by test-reference sim_p. If Medium+High no longer beats Unfiltered, the recommendation is an artifact of medium-only evaluation; if it still wins, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the evaluation distribution is representative of general use. Section 4 states: 'The test dataset is used in all experiments, and it has multiple human references per source text for evaluation. It is composed of medium semantic similarity only.' The Section 6.2 recommendation—filter multi-reference training by sim_p and discard low-similarity references—is validated only against a test set that has already been filtered by the same criterion. Since every test reference is medium-similarity, a model trained on Medium+High references is distribution-matched to the test references, whereas Unfiltered training exposes the model to low- and high-similarity references that may shift its output style away from the medium-only test references. The observed BLEU/COMET/chrF++ gains are therefore consistent with a train/eval distribution match rather than with the causal claim that low-similarity references are generally harmful. The paper's Limitations section does not mention this test-set confound. A secondary overstatement: Table 3 shows Opus-MT does not consistently improve under Medium+High (e.g., BLEU 25.1 vs 25.3 at #SRC=68k), so the 'across models and metrics' phrasing in Section 6.2 is stronger than the data support; the abstract's 'when fine-tuning an LLM' is the defensible scope.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates how to best use multiple human reference translations when fine-tuning literary machine translation models. Using the Par3 dataset, it scores the semantic similarity between English references with PARAGRAM-SP and partitions source texts into Low, Medium, and High similarity groups. It constructs Single, Medium, Unfiltered, Medium+Low, and Medium+High training sets, and fine-tunes mT5-large, LLaMA-2-7B, and Opus-MT. Two main findings are reported: (i) holding total training instances constant, single-reference training with more source texts is only marginally better than multiple-reference training with half the sources; (ii) filtering to Medium+High similarity references outperforms an unfiltered dataset, with gains of 0.3–0.5 BLEU, 0.1–0.9 COMET, and 0.17–0.32 chrF++ for the LLMs. The paper provides public code and uses paired bootstrap significance tests.","tokens_in":14794,"tokens_out":7558,"duration_ms":57967,"significance":"If the findings hold, they offer practical guidance for multi-reference MT training and demonstrate the value of semantic-similarity filtering. The study has notable strengths: it uses real expert literary translations, controls for total training instances in the single-versus-multiple comparison, evaluates three diverse model architectures, and applies bootstrap significance testing. The code is publicly available, which supports reproducibility. However, the central recommendation about filtering low-similarity references is currently validated only against a test set that shares the same similarity filter, which limits its generalizability until that confound is addressed.","major_comments":[{"comment":"The evaluation set is constructed to contain only medium-similarity references, which is the same property used to build the Medium and Medium+High training sets. Section 4 states: 'The test dataset is used in all experiments, and it has multiple human references per source text for evaluation. It is composed of medium semantic similarity only.' Consequently, the comparison between Unfiltered and Medium+High training in Table 3 confounds the effect of reference similarity with train/eval distribution matching: a model trained on Medium+High sees training targets that resemble the test references, while Unfiltered training includes low- and high-similarity targets that may shift output style away from the test distribution. The observed gains are therefore consistent with an artifact of test-set construction. The paper should either evaluate on an unfiltered test set covering all similarity levels, or report results separately for low- and high-similarity test references. The Limitations section does not mention this confound.","section":"Section 4 and Section 6.2"},{"comment":"The claim that 'Medium+High (#SOURCE = 66k, 68k) outperforms Unfiltered across different MT metrics and models (p-value < 0.01)' is not supported by the data in Table 3. For Opus-MT, BLEU at #SRC=66k is tied at 25.3 (marked †, not significant) and at #SRC=68k is lower than Unfiltered (25.1 vs 25.3, marked †). The significant improvements are limited to mT5 and LLaMA, and for Opus only on chrF++ at #SRC=66k. The abstract's phrasing 'when fine-tuning an LLM' is appropriate; the broader 'MT model' phrasing in Section 6.2 and the bullet points in the Introduction overstate the results. Please restrict the claim accordingly or provide justification for excluding Opus-MT's BLEU scores.","section":"Section 6.2, Table 3"},{"comment":"The text reports that 'Single on average only marginally outperforms Medium ... ranging 0.0–0.7 in BLEU.' However, Table 2 shows LLaMA-2-7B at #INSTANCE=77k has a BLEU difference of 1.2 (29.4 vs 28.2), and Opus-MT at #INSTANCE=58k has a difference of 0.8 (23.9 vs 23.1). The stated range is therefore incorrect and should be 0.0–1.2. While the qualitative conclusion of marginal superiority may still hold, the misreported range should be corrected.","section":"Section 6.1, Table 2"},{"comment":"The threshold for 'meaningful variations' (sim_p in [0.45, 0.85]) is selected by manual inspection, and the same threshold is used to define both the training conditions and the test set. No sensitivity analysis is provided for this choice. Because the main recommendation is to filter by this threshold, the paper should demonstrate that the results are stable across nearby threshold values or at least discuss how the threshold was validated. This is particularly important given that the similarity scorer (PARAGRAM-SP) is co-developed by one of the authors, which increases the burden of showing the results are not an artifact of the specific scoring choice.","section":"Section 3 and Section 4"}],"minor_comments":[{"comment":"The description of the Single dataset should state whether its references are drawn from all similarity levels in Par3 or only from a particular subset. This affects the interpretation of the Single vs. Medium comparison in Table 2.","section":"Section 4"},{"comment":"The paper does not report the size or language breakdown of the test set; this information should be added for reproducibility.","section":"Section 4"},{"comment":"The Medium+1.0 condition is constructed by repeating references from other categories, which may not be equivalent to naturally occurring identical paraphrases; this limitation should be acknowledged.","section":"Section 6.5, Table 4"},{"comment":"There is a typo in Section 5.3: 'randolmized' should be 'randomized'.","section":"Section 5.3"},{"comment":"The notation #SOURCE and #INSTANCE is used throughout; an explicit definition table in Section 4 would improve readability.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The primary risk to this paper is the train/eval distribution-match confound in the test set construction. I would encourage the editor to ask the authors to add an unfiltered evaluation set or to analyze results by test-reference similarity level. Also note that the paper's third author is a co-author of both Par3 and PARAGRAM-SP; this is not a conflict of interest but may warrant transparency about the evaluation tool's provenance. The paper is otherwise well-executed and could be suitable after a major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the paper's most solid contribution is the experiment holding total training instances constant and showing single-reference training only marginally beats multiple-reference with half the source texts. That's a new result and it holds up. The second headline claim, that filtering multi-reference training to medium+high semantic similarity outperforms unfiltered data, is not yet established: the test set is composed only of medium-similarity references (Section 4), which is the same property that defines the winning training condition. So the observed BLEU/COMET/chrF++ gains are consistent with a train/eval distribution match rather than with the causal claim that low-similarity references are generally harmful.\n\nCredit where due: using expert human translations from Par3 rather than synthetic paraphrases is a real step forward. The three-model setup (mT5-large, LLaMA-2-7B, Opus-MT), the public code, and the bootstrap significance testing are all to the good. The paper is clearly written and the dataset splits by book are sensible.\n\nThe soft spots are, in order of size. First, the test set confound is load-bearing. The authors should evaluate on an unfiltered test set, or at least a held-out set stratified across similarity levels, before claiming filtering is generally beneficial. The Limitations section does not mention this. Second, the Section 6.2 phrasing \"across models and metrics\" is stronger than Table 3 supports: Opus-MT does not consistently improve under Medium+High (BLEU 25.1 vs 25.3 at #SRC=68k, for instance). The abstract's \"when fine-tuning an LLM\" is the defensible scope. Third, the medium range [0.45,0.85) is set by manual inspection of the sim_p distribution. That's fine for an exploratory study, but it makes the thresholds free parameters; the paper should at least acknowledge sensitivity or validate on a held-out range.\n\nThe single-vs-multi-reference finding is robust to the test-set concern because both conditions draw from the same medium-similarity pool; the comparison there is fair. So the paper is a credible empirical contribution with one important validation step missing.\n\nRecommendation: send it to peer review, but the reviewers should require an additional experiment on an unfiltered or similarity-stratified test set. As it stands, the filtering recommendation is an artifact waiting to be ruled out.","headline":"Worth reading for the constant-instance-count single-vs-multi result, but the headline filtering claim is confounded by a test set made of the same medium-similarity references used to build the winning training set.","tokens_in":15288,"tokens_out":2148,"would_cite":true,"duration_ms":18604,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Filtering multiple reference translations by semantic similarity, keeping only medium- and high-similarity paraphrases, improves literary machine translation over unfiltered training data.","keywords":["machine translation","multiple references","paraphrase semantic similarity","literary translation","data filtering","fine-tuning","PARAGRAM-SP","large language models"],"falsifier":"Construct held-out test sets whose references are low-similarity, high-similarity, or a mixture matching the unfiltered distribution, and re-run the Medium+High versus Unfiltered comparison on them; if the Medium+High advantage disappears or reverses, the paper's recommendation is an artifact of test-set construction.","tokens_in":14308,"feed_emoji":"📖","tokens_out":12311,"duration_ms":100673,"temperature":0.7,"pith_summary":"Most machine-translation models are trained with a single reference translation per source, even though a sentence can be translated many ways. This paper argues that when several expert translations of the same source exist, the best recipe is not to use them all, but to filter them by semantic similarity and keep only medium- and high-similarity paraphrases. The paper reports that, across mT5-large, LLaMA-2-7B, and Opus-MT, fine-tuning on this filtered data outperforms an unfiltered multi-reference set on BLEU, COMET, and chrF++, even though the unfiltered set has more training instances. The paper also reports that when total training instances are held fixed, single-reference training with many more source texts is only marginally better than multiple-reference training with fewer source texts. If the findings generalize, data selection by similarity is a cheap way to improve literary machine translation, and multiple human translations become a valuable training resource rather than redundant text.","feed_headline":"Similarity-filtered references improve literary machine translation","feed_subtitle":"Keeping only medium- and high-similarity references lifts BLEU, COMET, and chrF++ across three models.","key_machinery":"The load-bearing mechanism is a semantic-similarity filter built on PARAGRAM-SP, a text-scoring model that returns a value from -1 to 1 for the semantic similarity of two English passages. For each source text, the paper computes $\\mathrm{sim}_p$ as the average pairwise PARAGRAM-SP score over all its reference translations, and thresholds at 0.45 and 0.85 define Low, Medium, and High similarity. This filter is what turns the raw Par3 translations into the Medium, Medium+Low, and Medium+High training sets, so the paper's central intervention is training on similarity-filtered references rather than on all references together.","core_discovery":"The central claim is that the similarity structure inside a multi-reference training set matters as much as its size. Using PARAGRAM-SP to score pairwise similarity among English translations of each source text, the paper defines $\\mathrm{sim}_p$ as the average over all reference pairs and partitions references into Low (below 0.45), Medium (0.45–0.85), and High (above 0.85). Across mT5-large, LLaMA-2-7B, and Opus-MT, fine-tuning on the Medium+High filtered set yields the best overall literary MT performance, with the two LLMs showing the clearest gains over Unfiltered: 0.3–0.5 BLEU, 0.1–0.9 COMET, and 0.17–0.32 chrF++. Adding Low-similarity references to Medium makes performance worse than Unfiltered, which the paper attributes to confusing or misaligned paraphrases. Separate from filtering, the paper finds that holding #INSTANCE constant, Single-reference training with at least twice the source texts only marginally outperforms Medium multi-reference training.","pith_inferences":["A natural extension is to apply the same $\\mathrm{sim}_p$ filter to synthetic paraphrases generated by LLMs; if the threshold transfers, it would give a low-cost cleaning step for domains where multiple human translations do not exist.","The near-parity between single- and multi-reference training at fixed instance count suggests that when expert multi-references already exist, they are almost as valuable as new source texts; a cost comparison of annotation effort could make this concrete.","Because the test set contains only medium-similarity references, the optimal threshold is likely tuned to that evaluation condition; a deployment evaluating on low- or high-similarity references may need to re-select the band.","The finding that Low-similarity references actively hurt performance implies that simply collecting more translations is not always safer than collecting fewer, which may extend to other multilingual data-selection tasks."],"forward_implications":["Fine-tuning on Medium+High references gives the best literary MT performance across the three models and, for mT5-large and LLaMA-2-7B, beats an Unfiltered training set with more instances by 0.3–0.5 BLEU, 0.1–0.9 COMET, and 0.17–0.32 chrF++.","At a fixed total instance count, Single-reference training with at least twice as many source texts differs from Medium multi-reference training by at most 0.7 BLEU, 2.5 COMET, and 1.98 chrF++, so multiple references can nearly substitute for additional source texts.","Adding Low-similarity references to a Medium training set makes performance worse than Unfiltered, indicating that noisy or misaligned paraphrases hurt rather than help.","The filtering benefit is clearer for the two LLMs than for the smaller Opus-MT model, suggesting that model capacity shapes how much varied references can be exploited."],"supporting_citations":[{"why":"Supplies Par3, the expert literary translations with multiple English references that all training and test sets are drawn from.","marker":"Thai et al., 2022"},{"why":"Supplies PARAGRAM-SP, the scorer used to compute $\\mathrm{sim}_p$ and partition references into Low, Medium, and High similarity.","marker":"Wieting et al., 2022"},{"why":"Shows simulated multiple-reference training helps low-resource MT; this paper extends the idea to naturally occurring expert paraphrases with similarity filtering.","marker":"Khayrallah et al., 2020"},{"why":"Prior multi-reference training with pseudo-references; this paper instead uses all available expert references.","marker":"Zheng et al., 2018"},{"why":"Earlier evidence that increasing synthetic reference count helps SMT tuning, providing the baseline question this paper refines.","marker":"Madnani et al., 2008"},{"why":"Provides the paired bootstrap resampling test used to report statistical significance for the headline comparisons.","marker":"Koehn, 2004"},{"why":"Motivates the use of chrF++ alongside COMET because learned metrics can degrade in literary domains.","marker":"Zouhar et al., 2024"}],"fun_headline_variants":["Filter out low-similarity refs for literary MT boost","Similarity-sorted refs lift literary MT scores","Medium-high similarity refs beat unfiltered literary MT","Diverse refs only help if they're similar enough"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the test set's medium-similarity references are the right evaluation condition, since the best training set is selected to contain that same similarity band and the reported gains could vanish if test references were low- or high-similarity.","fun_headline_variants_meta":{"raw":{"variants":["Filter out low-similarity refs for literary MT boost","Similarity-sorted refs lift literary MT scores","Medium-high similarity refs beat unfiltered literary MT","Diverse refs only help if they're similar enough"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000654,"raw_usage":{"total_tokens":3009,"prompt_tokens":967,"completion_tokens":2042,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":1975}},"tokens_in":583,"tokens_out":2042,"duration_ms":15166,"temperature":1.0,"reasoning_tokens":1975,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:32:26.595991+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct held-out test sets whose references are low-similarity, high-similarity, or a mixture matching the unfiltered distribution, and re-run the Medium+High versus Unfiltered comparison on them; if the Medium+High advantage disappears or reverses, the paper's recommendation is an artifact of test-set construction.","supporting_citations":[{"cited_title":"Dorr, and Richard Schwartz","cited_arxiv_id":null,"evidence_quote":"Earlier evidence that increasing synthetic reference count helps SMT tuning, providing the baseline question this paper refines."}],"review_version":1}