{"id":"6dc32b97-bf01-43e7-a702-1434a6605d22","arxiv_id":"2505.12983","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Instruction-tuned open LLMs beat zero-shot GPT-4 on automatic many-to-many summarization scores without lowering MMLU, but human evaluation shows instruction tuning can increase factual errors.","lead":"This paper tests how well large language models summarize documents in one language and produce the summary in another, across 18 models and six languages. It finds that fine-tuning open models on a new multilingual benchmark improves their scores substantially, but can increase factual errors.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test set may share source documents with training set across language pairs, which can inflate instruction-tuned LLM scores in the headline GPT-4 comparison.","rationale":"The reader's weakest assumption already targeted test-set fairness, and I agree with that identification. The specific failure mode I add is cross-language-pair source-document leakage in the train/test split: the paper reports no document-level deduplication, and the shared source datasets make it plausible. This is a correctness risk rather than a claim about consensus, and it is checkable. I keep the reader's CONDITIONAL verdict rather than moving to ACCEPT or REJECT because the existence and magnitude of the overlap are unverified; the proposed audit would decide. Given the authors have not released data or code, the current evidence is insufficient to accept the headline comparison as clean, but the finding could survive the audit, so rejecting outright would be premature.","tokens_in":28871,"tokens_out":9744,"duration_ms":101627,"concrete_test":"Compute exact or near-duplicate source-document overlap between training-plus-validation and testing, using document IDs from the eight source datasets or normalized source text, counting any test sample whose source document appears in a training sample even under a different target language. If the count is nonzero, rebuild document-disjoint splits (all language pairs for one source document in the same split), retrain the instruction-tuned models on the reduced training set, and recompute Table 3 R1/RL/BS for tuned LLMs and zero-shot GPT-4 on the clean test set. If the tuned-versus-GPT-4 margins shrink materially or reverse, the headline claim is unsupported in its current form.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The fairness of the M2MS test set is the load-bearing assumption for the paper's central automatic-metric claim (Table 3: instruction-tuned open LLMs outperform zero-shot GPT-4). To be fair, no test source document should appear in the instruction-tuning training set. The paper does not establish this. Section 3 randomly splits samples within each source-target language pair from the same eight datasets, and Table 8 shows the same source datasets supply many pairs (e.g., En→De, En→Fr, En→Zh all use XWikis and WikiLingua). Multilingual datasets such as XWikis, WikiLingua, and CrossSum align multiple summaries to the same source document, so a single document can be in training for one target language and in testing for another. Because the M2MS input is the source document alone, an instruction-tuned model can then memorize that document and score high ROUGE/BERTScore, while zero-shot GPT-4 has no such exposure. Appendix B's contamination check targets LLM pretraining contamination (BLEURT/ROUGE-L) and caps it below 1% of the test set; it does not check duplication between the reorganized training and test splits. The authors' Limitations section does not mention this either. If such overlap exists, the main comparison is inflated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a systematic empirical study of many-to-many summarization (M2MS) with large language models. The authors reorganize samples from eight existing multilingual summarization datasets into a 47.8K-sample M2MS benchmark spanning six languages and five domains, with train/validation/test splits. They benchmark 18 LLMs in zero-shot and instruction-tuned settings, alongside fine-tuned traditional models mBART-50 and PISCES, using ROUGE, BERTScore, GPT-4o-based quality scores, and a fine-grained human factuality annotation. The headline findings are that instruction-tuned open-source LLMs substantially improve their M2MS performance and can outperform zero-shot GPT-4 on automatic metrics, that this task-specific improvement does not degrade MMLU performance, and that instruction tuning may intensify factual errors in generated summaries.","tokens_in":29126,"tokens_out":2803,"duration_ms":29317,"significance":"If the headline claims hold, the paper is a valuable empirical contribution: it is, to my knowledge, the first systematic study of LLMs on the M2MS task, it assembles a reusable multi-domain, six-language benchmark, and it compares a wide array of open- and closed-source LLMs under controlled prompting and tuning conditions. The paper also merits credit for attempting contamination control for the test set, for including fine-tuned traditional baselines, for checking general ability retention via MMLU, and for conducting human factuality evaluation with reported inter-annotator agreement. The finding that instruction tuning on roughly 19.5K samples yields large automatic-metric gains while preserving MMLU is practically important, and the caution that tuning may intensify hallucination is a useful counterpoint. However, the significance is contingent on two things: whether the test set is actually disjoint from the instruction-tuning training data at the document level, and whether the headline 'significantly outperform' claims are supported by appropriate statistical evidence.","major_comments":[{"comment":"The paper does not establish that the test split is disjoint from the instruction-tuning training split at the document level, which is a load-bearing assumption for the central automatic-metric claim in Table 3. Section 3 states that samples are randomly selected within each dataset and each source-target language pair, and Table 8 shows that the same source datasets (e.g., XWikis, WikiLingua, CrossSum) supply many different language pairs. These multilingual datasets are constructed by aligning multiple summaries to the same source document, so a single source document can appear in the training split for one target language and in the test split for another target language. Because the M2MS input to the model is the source document alone, an instruction-tuned model can memorize that document and obtain inflated ROUGE/BERTScore, while zero-shot GPT-4 has no such exposure. Appendix B's contamination check addresses LLM pretraining contamination using BLEURT/ROUGE-L against the LLMs' pretraining corpora; it does not check duplication between the reorganized training and test splits. The Limitations section also does not mention this issue. The authors should quantify the extent of any document-level overlap between training and test sets (e.g., by exact or near-duplicate source-document matching across all language pairs) and either re-run the headline comparisons on a document-disjoint split or provide evidence that no overlap exists.","section":"§3 Data; Appendix B; Table 8"},{"comment":"The paper repeatedly uses 'significantly' and 'by a large margin' to describe differences between instruction-tuned LLMs, zero-shot LLMs, and traditional models, but it reports only single point estimates without significance tests, confidence intervals, or error bars. For example, the claim that 'instruction-tuned LLMs significantly outperform the best traditional language model' and the claim that some tuned LLMs 'outperform zero-shot GPT-4' rely on differences in ROUGE/BERTScore means that could fall within sampling variation across the 14,150 test samples. The authors should report paired bootstrap confidence intervals or permutation-test p-values for the headline comparisons (at minimum for the overall scores in Table 3 and for the GPT-4o evaluation in Table 4), and should adjust the language of 'significant' to match the statistical evidence actually provided.","section":"§5 Results and Analyses; Tables 3, 4, 11"},{"comment":"The human factuality evaluation is too narrow to support the broad conclusion that 'instruction tuning might intensify the issue' of factual errors. The evaluation in Appendix H uses only 100 English-source documents with Chinese summaries, for five systems, and the counts in Table 7 are small (e.g., hallucination counts of 8, 12, 17 for GPT-4, zero-shot Vicuna, tuned Vicuna). No significance testing or confidence intervals are reported for these differences, and the evaluation does not cover the full language and domain diversity of the benchmark. I recommend treating the factuality finding as suggestive rather than conclusive, and either expanding the evaluation or adding explicit statistical caution in the abstract and conclusion.","section":"§6 Discussion; Table 7; Appendix H"}],"minor_comments":[{"comment":"There are several typos and inconsistencies: 'Appenidx D' should be 'Appendix D'; 'Unversity' in the author affiliation should be 'University'; 'MSMS' in Section 5 should be 'M2MS'; and the terms 'instruct-tuning', 'instruction-tuning', and 'instruction tuning' are used inconsistently.","section":"§1 and throughout"},{"comment":"Liang et al. 2022a and Liang et al. 2022b appear to be the same paper (same title, same venue, same pages) listed twice; the duplicate reference should be removed or corrected.","section":"References"},{"comment":"The main text reports GPT-4o-based evaluation scores in Table 4 without stating that only 500 randomly selected test samples were used for this evaluation; the 500-sample subset is mentioned only in Appendix E.1. The main text should state this clearly so readers can calibrate the precision of those scores.","section":"§4.2 and Appendix E.1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is the first systematic study of many-to-many summarization with LLMs, and it contributes a reusable 47.8K-sample benchmark spanning five domains and six languages. The central empirical claim—that small-scale instruction tuning pushes open LLMs well past zero-shot GPT-4 on ROUGE and BERTScore—is plausible but not fully established, because the split may share source documents between training and test across language pairs.\n\nWhat is genuinely new: the benchmark assembly from eight existing datasets, the comparison of 18 LLMs in zero-shot and instruction-tuned settings, the fine-tuned mBART/PISCES baselines, and the MMLU retention check. The human factuality evaluation is also valuable: instruction tuning appears to increase hallucination and particulars errors, a finding worth taking seriously even though the sample is small.\n\nThe stress-test concern is real. Section 3 randomly splits samples within each source-target pair, not at the source-document level. Table 8 shows the same datasets supply multiple language pairs, and datasets like XWikis, WikiLingua, and CrossSum align multiple summaries to one source document. So the same document can appear in training for En→De and in testing for En→Fr. Since M2MS inputs are source documents alone, an instruction-tuned model can benefit from prior exposure to that document. Appendix B checks LLM pretraining contamination, not train/test duplication, and the Limitations section does not mention the issue. That is a genuine hole, not a manufactured one.\n\nStill, the effect size is large: instruction-tuned Vicuna-13B-16k reaches 38.0 R1 overall versus 25.7 for zero-shot GPT-4. I doubt document overlap alone explains a 12-point gap. But the paper repeatedly says instruction-tuned LLMs \"significantly improve\" and \"outperform\" without any significance tests, confidence intervals, or error bars. That wording needs to change. The GPT-4o evaluation on a random 500 samples is fine as a cost-saving measure, but it could use an uncertainty estimate. The human factuality evaluation on 100 English documents with Chinese summaries is narrow; it supports a qualitative warning, not a strong quantitative claim. No data or code release is provided, which makes verification harder. The citation pattern is fine; the PISCES self-citation is a baseline and not load-bearing.\n\nThis paper is for people building multilingual summarization systems and for benchmark designers. It deserves a serious referee, but the referee should require a document-level deduplicated split or at least a quantification of cross-pair overlap, plus significance testing. I would not desk-reject it.","headline":"A useful new benchmark and the first broad LLM comparison for many-to-many summarization, but the train/test split may let the same source document appear in both sides across language pairs, so the headline instruction-tuning result is not fully trustworthy until that overlap is ruled out.","tokens_in":29676,"tokens_out":2533,"would_cite":true,"duration_ms":29055,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Instruction-tuned open-source LLMs can outperform zero-shot GPT-4 on automatic many-to-many summarization metrics while preserving general ability, but instruction tuning can intensify factual errors in generated summaries.","keywords":["many-to-many summarization","cross-lingual summarization","instruction tuning","large language models","multilingual benchmark","data contamination","factual consistency","MMLU evaluation"],"falsifier":"Take the same instruction-tuned models and evaluate them on a newly collected M2MS test set of the same five domains and six languages that was not assembled from the eight source datasets and was screened with a stronger contamination method; if the ROUGE and BERTScore advantages over zero-shot GPT-4 shrink or vanish, the claimed improvement is an artifact of test-set construction rather than genuine M2MS ability. A simpler check: count the actual contaminated fraction of the released test set for each of the six LLMs measured in Appendix B; if it exceeds the reported 1%, the GPT-4 comparison is suspect.","tokens_in":28678,"feed_emoji":"🌐","tokens_out":6083,"duration_ms":51564,"temperature":0.7,"pith_summary":"This paper tries to establish how well large language models perform many-to-many summarization—summarizing a document in any source language into any target language—when used off the shelf and when instruction-tuned on task data. To do this, the authors reorganize 47.8K samples drawn from eight existing multilingual summarization datasets, covering five domains and six languages, with intrinsic-quality filtering and contamination screening on the test set. They benchmark 18 LLMs in zero-shot and instruction-tuned settings against fine-tuned traditional multilingual models (mBART-50 and PISCES). The central empirical claim is that instruction-tuned open-source LLMs substantially improve on their zero-shot versions and outperform zero-shot GPT-4 and GPT-4o on automatic ROUGE and BERTScore metrics, while preserving MMLU performance. The paper also argues, on the basis of human evaluation, that LLMs still produce factual errors and that instruction tuning can intensify them, making factual control the key obstacle for real applications.","feed_headline":"Tuned open LLMs beat GPT-4 on many-to-many summarization","feed_subtitle":"A 47.8K-sample benchmark across six languages and five domains shows large gains, yet factual errors grow with tuning.","key_machinery":"The central object is the reorganized M2MS benchmark: 47.8K document–summary pairs in five domains (news, encyclopedia, dialogue, how-to guides, technology) across six languages (English, Czech, German, French, Chinese, Ukrainian), built from eight existing datasets, with samples filtered by coverage, redundancy, and coherence and the test set screened so that fewer than 1% of instances are flagged as contaminated for the evaluated LLMs. The mechanism that carries the argument is instruction tuning: the same task prompt used for zero-shot evaluation (system instruction, domain tag, three in-context example summaries) is used to format the 19.5K training samples, and the open-source LLMs are tuned for two epochs. The comparison against fine-tuned traditional models (mBART-50 and PISCES, with a language tag on the decoder) and against zero-shot prompting isolates the effect of parameter updating on M2MS ability. The prompt itself, with in-context examples, is the instrument that elicits M2MS behavior from zero-shot LLMs.","core_discovery":"On the paper's own terms, the discovery is that task-specific instruction tuning on a relatively small, multi-domain M2MS training set (19.5K samples) turns open-source LLMs into many-to-many summarizers that outperform zero-shot closed-source LLMs, including GPT-4, on ROUGE-1, ROUGE-L, and BERTScore, and do so without lowering MMLU scores. For instance, instruction-tuned Vicuna-13B-16k reaches 38.0 ROUGE-1 and 74.1 BERTScore overall, compared with 26.0 and 66.7 for zero-shot GPT-4o; tuned LLaMa, Qwen, Baichuan, and InternLM models all clear the zero-shot GPT-4 marks. The same evaluation shows a caution: a fine-grained human annotation of 100 English-to-Chinese summaries finds that GPT-4 has the fewest factual errors, while tuned LLaMa and Vicuna models hallucinate more than their zero-shot versions, so the automatic-metric gains do not translate into factuality gains.","pith_inferences":["A natural testable extension would be to apply the same instruction-tuning recipe to LLMs that have explicit multilingual alignment or factual-consistency training, since the paper's own results suggest hallucination, not overlap quality, is the bottleneck.","The contamination-screening protocol could be adopted as a standard guardrail for multilingual LLM benchmarks, because the paper shows test-set leakage can be controlled but not fully eliminated for low-resource language pairs.","The factuality regression under instruction tuning suggests that cleaning or filtering hallucinated references in training data may be a more direct lever than better prompts, since the paper attributes intensified errors partly to information gaps in human-written references.","The benchmark's language-pair coverage leaves out Czech–Ukrainian and German–Ukrainian pairs, so a targeted experiment on those missing pairs would delimit whether the gains extend to all language pairs."],"forward_implications":["Instruction-tuned open-source LLMs can serve as practical many-to-many summarizers across six languages and five domains, outperforming fine-tuned traditional multilingual models by wide margins (e.g., about 7 ROUGE-1 points over PISCES).","Task-specific M2MS instruction tuning does not degrade general problem-solving ability as measured by MMLU, so specialized tuning is a viable route to deployable summarizers.","Automatic metrics (ROUGE, BERTScore) can improve substantially while factual error rates worsen, so evaluations of LLM summarizers should include factuality checks, not just overlap metrics.","Zero-shot LLMs already rival fine-tuned traditional models, so the marginal value of traditional encoder-decoder specialization is reduced when an LLM backbone is available.","Training-scale experiments show LLMs retain more performance than traditional models when training data is reduced (about 12.4 versus roughly 22 ROUGE-1 points lost from 19.5K to 2K samples), implying LLMs are more data-efficient for this task."],"supporting_citations":[{"why":"Provides the CrossSum dataset and the earlier M2MS framing that this study extends to multi-domain, multi-LLM evaluation.","marker":"Bhattacharjee et al., 2023"},{"why":"Supplies the PISCES baseline and the M2MS training setup for traditional models, including the decoder language-tag convention.","marker":"Wang et al., 2023c"},{"why":"Supplies the instance-level contamination measurement used to screen the test set.","marker":"Golchin and Surdeanu, 2024"},{"why":"Defines the coverage metric used to filter low-quality document-summary samples.","marker":"Grusky et al., 2018"},{"why":"Supplies the redundancy and coherence metrics and the threshold-based filtering approach adopted for sample selection.","marker":"Bommasani and Cardie, 2020"},{"why":"Supplies the MMLU benchmark used to test whether M2MS instruction tuning preserves general task-solving ability.","marker":"Hendrycks et al., 2021"},{"why":"Supplies the factual-error taxonomy and the human-evaluation procedure for cross-lingual summarization factuality.","marker":"Gao et al., 2023"},{"why":"Supplies the LLaMa-2 backbone and the instruction-tuning recipe, including epoch count and data-scale guidance.","marker":"Touvron et al., 2023b"},{"why":"Supplies the Vicuna models, including the 16K-context variants that achieve the best instruction-tuned results.","marker":"Chiang et al., 2023"}],"fun_headline_variants":["Tuned open LLMs overtake GPT-4 in many-to-many summarization","Task tuning pushes open LLMs past GPT-4 on multilingual summarization","Open LLMs top GPT-4 on summarization metrics, but facts falter","Instruction tuning lifts open models above GPT-4, yet adds errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reorganized test set measures true M2MS ability: the intrinsic-metric filtering and the below-1% contamination screen are assumed to prevent inflated scores, and the train and test splits drawn from the same eight source datasets are assumed not to overlap in a way that favors instruction-tuned models. If contamination or document overlap is higher than measured, the zero-shot versus instruction-tuned comparisons, and particularly the GPT-4 comparisons, could be biased.","fun_headline_variants_meta":{"raw":{"variants":["Tuned open LLMs overtake GPT-4 in many-to-many summarization","Task tuning pushes open LLMs past GPT-4 on multilingual summarization","Open LLMs top GPT-4 on summarization metrics, but facts falter","Instruction tuning lifts open models above GPT-4, yet adds errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000286,"raw_usage":{"total_tokens":1728,"prompt_tokens":1040,"completion_tokens":688,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":656,"completion_tokens_details":{"reasoning_tokens":605}},"tokens_in":656,"tokens_out":688,"duration_ms":6815,"temperature":1.0,"reasoning_tokens":605,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:21:47.478092+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same instruction-tuned models and evaluate them on a newly collected M2MS test set of the same five domains and six languages that was not assembled from the eight source datasets and was screened with a stronger contamination method; if the ROUGE and BERTScore advantages over zero-shot GPT-4 shrink or vanish, the claimed improvement is an artifact of test-set construction rather than genuine M2MS ability. A simpler check: count the actual contaminated fraction of the released test set for each of the six LLMs measured in Appendix B; if it exceeds the reported 1%, the GPT-4 comparison is suspect.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the instance-level contamination measurement used to screen the test set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the redundancy and coherence metrics and the threshold-based filtering approach adopted for sample selection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the factual-error taxonomy and the human-evaluation procedure for cross-lingual summarization factuality."}],"review_version":1}