{"id":"b888f96b-d343-4020-b4ba-2ad87643c727","arxiv_id":"2412.15487","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Using multiple LLMs to generate and select summaries raises ROUGE and BLEU scores on ArXiv and GovReport compared with single-LLM chunk-and-concatenate baselines.","lead":"This paper uses several AI language models to write summaries of long documents and then lets one AI, or all of them, choose the best summary. The authors report large gains on automatic metrics versus a single-model baseline, but a missing control condition makes the source of the gains unclear.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim is confounded: single-LLM baselines in §6.1 use one-pass concatenated chunk summaries, while multi-LLM pipelines in §3 use a second chunking and summarization stage; the missing control leaves the gain attributable to an extra aggregation pass rather than to multiple LLMs.","rationale":"The reader's weakest_assumption is exactly the confound I would flag: the single-LLM baselines are not structurally matched to the multi-LLM pipeline. I agree with that diagnosis, and I would keep the REJECT verdict. The stress-test pass did not reveal a different, more fundamental flaw; the missing control is sufficient because it attacks the causal attribution in the abstract rather than a secondary implementation detail. The paper has useful ingredients—a clear centralized/decentralized framework, a cost analysis in §B, and several ablations—but those ablations vary model combinations, prompt specialization, and model count while holding the two-stage multi-LLM pipeline fixed, so they show robustness of the framework, not that 'multiple LLMs' are the source of the gains. The human-evaluation section is also weak: it compares only 10 summary pairs from one setup, uses raters who are authors or close associates, and drops the Relevance criterion; however, this is secondary support for the second contribution claim. The concrete control experiment would settle the primary issue: if a single LLM with hierarchical re-summarization reaches comparable scores, the central claim is unsupported; if there is a large gap, the claim would be on much firmer ground. Because the current manuscript does not include that control, the reader's REJECT verdict stands unmodified.","tokens_in":20329,"tokens_out":5061,"duration_ms":31831,"concrete_test":"Run a hierarchical single-LLM baseline for each model in Table 2 (GPT-3.5, GPT-4o, GPT-4o mini, LLaMA3-8B): use identical 4K-char chunking and first-stage prompts, concatenate the per-chunk summaries, then apply the same second round of chunking and summarization that §3 gives to the multi-LLM pipeline, with only that one model generating the final summary and no multi-model selection. Compare ROUGE-1, ROUGE-L, BLEU-1, and BLEU-4 on ArXiv and GovReport against the multi-LLM rows of Table 2. If a single-LLM hierarchical baseline reaches within about 0.01 of the multi-LLM scores, or overlaps with bootstrap confidence intervals, the reported 3x gains are largely an artifact of the missing aggregation control. Also run a k=1 variant of each multi-LLM topology (one model generating multiple draft summaries and selecting among them) to isolate generation multiplicity from selection behavior.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing risk is that the comparison in §6.1 is not apples-to-apples. The abstract claims multi-LLM summarization 'significantly outperform[s] the baselines that leverage only a single LLM by up to 3x,' but the baseline condition omits the hierarchical aggregation stage that is built into every multi-LLM condition. Section 6.1 says single-LLM baselines use the same chunking and 'the final summary represents a concatenation of the generated summaries'—one pass per chunk, with no second aggregation stage. Section 3 defines the multi-LLM framework as a 'two stage process' that chunks, summarizes each chunk, and then applies 'a second round of chunking and summarization on the concatenated intermediate results.' Consequently, the observed gains in Tables 2, 3, 5, 6, 7, and 8 could be produced by the extra aggregation pass (which lets the final summary be written from a compressed global view of the document) rather than by 'multiple LLMs' per se. The ablations in §6.3 all inherit the same two-stage structure, so they cannot isolate the contribution of model multiplicity. A single-LLM hierarchical control is therefore required before the central attribution can be accepted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-LLM summarization framework in which k LLMs generate candidate summaries and either a single central LLM (centralized) or all k LLMs (decentralized) evaluate and iteratively refine them. Long documents are handled with a hierarchical two-stage procedure: the document is chunked, each chunk is summarized, and the concatenated intermediate summaries undergo a second round of chunking and summarization. Experiments on ArXiv and GovReport report ROUGE and BLEU gains over single-LLM baselines by up to 3x, together with ablations over model combinations, number of LLMs, prompting variants, and text length. A small human evaluation is used to claim alignment between the framework's preferences and human judgments.","tokens_in":20613,"tokens_out":6590,"duration_ms":51105,"significance":"If the attribution were established, this would be a useful empirical contribution: the paper includes detailed algorithm listings and prompts, provides a token-level cost analysis (Sections 4.3, 5.3, B.1, B.2), and explores several systematic ablations. The central comparison, however, is confounded by an architectural difference between the multi-LLM and single-LLM conditions, and the human-alignment claim rests on weak agreement in a small sample. The abstract's principal claim therefore needs substantial additional evidence before it is supported.","major_comments":[{"comment":"The single-LLM baselines are not adequate controls for the central claim. Section 6.1 states that baselines use the same chunking and that \"the final summary represents a concatenation of the generated summaries,\" whereas Section 3 defines the multi-LLM framework as a \"two stage process\" that adds \"a second round of chunking and summarization on the concatenated intermediate results.\" Every multi-LLM condition inherits this second aggregation stage, so the large gains in Tables 2, 3, 5, 6, 7, and 8 are also consistent with the possibility that a second hierarchical pass over the concatenated chunk summaries—rather than the participation of multiple LLMs—causes the improvement. The ablations in Section 6.3 cannot isolate model multiplicity because they retain the same two-stage structure. A required control is a single-LLM pipeline using the identical two-stage chunk-then-resummarize procedure, with the same prompts at each stage and a comparable number of inference calls; without it, the attribution in the abstract is unsupported.","section":"§3 vs. §6.1"},{"comment":"The contribution that \"the best performing method in the framework aligns with human judgments\" is not supported by the presented evidence. The human raters evaluate only the first 10 pairs of summaries generated before the final evaluation step of one pipeline (decentralized, one-round maximum), not the final outputs of the best-performing method. The reported agreement between human and machine choices is κ = 0.2, 0.6, 0.1, and 0.2 across the four comparisons, and the machine choices in Table 10 are split 5/5 between GPT-3.5 and GPT-4o mini, making the statement that humans \"generally prefer\" GPT-4o mini and that this aligns with the framework difficult to sustain. This claim should be removed, substantially weakened, or re-tested on final summaries from the full framework with a larger sample and a pre-specified analysis.","section":"§6.5, Tables 9–10"},{"comment":"The main results select GPT-3.5 as the evaluator and tie-breaker after comparing GPT-4o mini, GPT-3.5, and GPT-4o on the same test sets and observing that GPT-3.5 \"emerged as the best-scoring evaluator\" (Appendix C.1). This is a post-hoc selection of the evaluation configuration on the test data, which inflates the headline scores. The evaluator and tie-breaking model should be chosen on a held-out validation split, or the paper should report all evaluator configurations without designating one as best based on test-set performance.","section":"Appendix C.1, Tables 2, 5"},{"comment":"The abstract and Section 6.2 use \"significantly outperform\" without any significance test, confidence interval, or per-document paired comparison. All tables report single point estimates, and some \"up to 3x\" improvements correspond to small absolute differences (e.g., ArXiv BLEU-4 in Table 2: 0.021 vs. 0.043). The authors should report paired tests across documents (for example, bootstrap or signed-rank tests) and effect sizes for the central multi-LLM versus single-LLM comparisons.","section":"§6.2, Tables 2, 3, 5, 6, 8"}],"minor_comments":[{"comment":"There is a typo in the second paragraph: \"under decentralized and decentralized frameworks\" should presumably read \"under decentralized and centralized frameworks.\"","section":"§6.2"},{"comment":"Section 6.1 says the baseline generation prompt is identical to that in the first round and cites Figure 6, but Figure 6 is the prompt used for the second-stage concatenated chunks; the figure reference should be aligned with the described stage to avoid confusion.","section":"§6.1, Figures 2 and 6"},{"comment":"The algorithms use S both for an ordered set of summaries and for the input text, and line 1 calls CREATE SUMMARY (S) without defining this function; the notation should be disambiguated.","section":"Algorithms 1 and 2"},{"comment":"The displayed formula for the final summary S* uses the notation \"r ∈ 1, . . . , kk\" and would be much easier to parse with explicit vote-vector indexing; the current typesetting makes the tie-break rule hard to read.","section":"§5.1.2"},{"comment":"The claim that Table 3 shows improvements \"across the board\" cannot be checked directly because Table 3 reports only multi-LLM scores; adding baseline rows or deltas would make the comparison explicit.","section":"§6.3, Table 3"},{"comment":"The human raters are disclosed to be authors and close associates; even with anonymized and randomized summaries, this is a potential source of bias and should be discussed as a limitation or addressed with independent raters.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The load-bearing issue is the missing single-LLM two-stage control. If the authors supply such a control and the gains persist, the paper could be suitable for publication; if the control matches the multi-LLM scores, the central claim collapses. I recommend major revision rather than rejection because the fix is an experimental addition, but the revision should be judged strictly on the new control, the evaluator-selection issue, and the reanalysis of the human evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know two things about this one. First, it is a straightforward application of multi-agent debate and LLM-as-judge selection to long-document summarization, with new ROUGE/BLEU numbers on ArXiv and GovReport. Second, the central comparison is not clean: the multi-LLM pipeline has a two-stage chunk-then-resummarize structure, while the single-LLM baseline gets one pass per chunk and simple concatenation. No single-LLM two-stage control exists, so the reported 70% average gains and the \"up to 3x\" headline cannot be attributed to the fact that multiple LLMs are involved. The stress-test note is right on this; it holds up on reading.\n\nWhat the paper does well: the framework is clearly described with algorithms and prompts, the cost analysis is thoughtful, and the ablations cover model combinations, number of LLMs, specialized prompts, and short vs long text. Credit also for reporting that extra rounds and a third LLM do not help, which cuts against the conversational motivation but is honest. The human evaluation is a real effort, though its conclusion is weaker than the abstract suggests.\n\nThe soft spots beyond the confound: the evaluator and tie-breaker (GPT-3.5) were chosen after looking at the same test sets, which is selection bias; there are no error bars or significance tests; the human study uses 10 summary pairs from 7 raters, several of whom are coauthors, and the LLM-human agreement is strong only for conciseness (κ=0.6), while coherence and fluency are 0.2 and 0.1. The claim that the best method \"aligns with human judgments\" overstates this.\n\nMy take: the paper is not a reject. The pipeline as an engineering recipe might be useful, and the underlying question matters. But the manuscript in its current form cannot support the causal claim that multiple LLMs cause the improvement. A revision that adds a single-LLM two-stage control, runs the baselines on the same protocol, reports variance, and reframes the contribution as a pipeline comparison rather than an ablation of model multiplicity could make this a solid empirical paper. As it stands, the results are suggestive but the load-bearing comparison is missing.\n\nSend it to peer review with a request for major revision, and make sure the reviewers focus on the missing control and the post-hoc evaluator choice. A serious referee can help the authors fix this; a desk reject would waste a decent empirical effort.","headline":"A useful empirical recipe undercut by a missing control: the multi-LLM pipeline adds a two-stage aggregation pass the single-LLM baseline does not have, so the headline gains are not cleanly attributable to multiple models.","tokens_in":21233,"tokens_out":4506,"would_cite":false,"duration_ms":33158,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multiple LLMs generating and then selecting summaries outperforms single-LLM summarization by up to 3x on ROUGE/BLEU metrics.","keywords":["text summarization","multi-LLM","centralized summarization","decentralized summarization","long document summarization","LLM evaluation","ROUGE","BLEU"],"falsifier":"Run the strongest single LLM through the same two-stage chunk-then-resummarize pipeline used by the multi-LLM framework, and compare its ROUGE and BLEU scores with the multi-LLM results in Table 2; if the single LLM with hierarchical aggregation matches or exceeds the multi-LLM scores, the claimed multi-LLM advantage is not attributable to collaboration.","tokens_in":20124,"feed_emoji":"📝","tokens_out":5408,"duration_ms":44214,"temperature":0.7,"pith_summary":"The paper tries to establish that document summarization improves when several different LLMs each draft a summary and then one central LLM (centralized) or all of them (decentralized) select the best candidate, rather than having a single LLM summarize the text alone. On the ArXiv and GovReport datasets, the multi-LLM pipelines score higher on ROUGE-1, ROUGE-L, BLEU-1, and BLEU-4 than all four single-LLM baselines, with individual improvements up to three times the baseline values. The authors further claim that the summaries their framework selects align with human preferences for coherence, conciseness, and fluency. If true, this is evidence that diversity in generation plus collaborative selection can substitute for a single more powerful model or elaborate prompt engineering.","feed_headline":"Multiple LLMs beat a single one on summaries, up to 3x","feed_subtitle":"Having several models draft and vote on summaries lifts ROUGE/BLEU scores on long documents.","key_machinery":"The mechanism is a two-stage chunk-and-resummarize pipeline plus a selection round. First the long document is chunked and each chunk is summarized; then the chunk summaries are concatenated, re-chunked, and summarized again. At each stage, k different LLMs independently generate candidate summaries of the same input using a shared prompt, producing diversity; an evaluation phase then either has one central LLM pick the best candidate (with a confidence score that can trigger another round) or has every LLM vote for the best candidate, requiring a majority. The selection step is the load-bearing part: it converts model diversity into a single chosen summary. The number of participating models, the choice of evaluator model, and the maximum number of rounds are the knobs the experiments vary.","core_discovery":"The central claim is that a generate-then-evaluate loop over k LLMs produces better summaries of long documents than any single LLM used in one pass. In the centralized variant, one central LLM reads all candidate summaries and picks one; in the decentralized variant, every participating LLM votes for the best candidate and the majority winner is chosen, with a tie-breaker model as fallback. Across both variants, with two LLMs as generators and one LLM as evaluator and tie-breaker, the framework outperforms the single-LLM baselines on every reported metric, with average improvements around 70 percent and individual gains up to three times. The one-round version matches or beats the multi-round version, so the authors conclude that even the cheapest configuration delivers the gains. A small human study of ten summary pairs finds that the model-picked winner is often the human-preferred winner, with conciseness agreeing at a kappa of 0.6.","pith_inferences":["Beyond the paper: a fair test would give a single LLM the same two-stage chunk-summarize-resummarize pipeline used by the multi-LLM framework; until such a baseline is run, part of the gain could come from the extra aggregation pass rather than from multi-LLM collaboration.","Beyond the paper: with exactly two LLMs, the decentralized majority rule is equivalent to unanimity, so the reported consensus behavior is really a single agreeing vote plus the tie-breaker, not a broad consensus.","Beyond the paper: one testable extension is whether the same generate-and-select loop improves a single LLM's output when the candidates are produced by different prompts or temperatures, which would isolate the selection step as the active ingredient.","Beyond the paper: the human-alignment claim rests on ten summary pairs, so agreement estimates such as kappa equal to 0.6 are suggestive rather than definitive."],"forward_implications":["Even the cheapest configuration, two LLMs and a single round of generation and evaluation, delivers the reported gains, so the method does not require many models or many rounds.","Conversational rounds beyond the first do not improve scores, which implies that the framework's value lies in selection rather than iterative refinement.","Changing the evaluator and tie-breaker model, or adding a third LLM, keeps the multi-LLM scores above the single-LLM baselines, so the effect is not tied to one particular evaluator.","Specialized prompts aimed at coherence or precision can feed the same framework and still outperform the single-LLM baselines, indicating the approach is not bound to a single prompt formula.","Short-text summarization also benefits, with the centralized one-round configuration showing the largest short-text gains on ArXiv introductions."],"supporting_citations":[{"why":"Supplies the long-context failure mode, LLMs overlooking middle content, that motivates the multi-LLM design.","marker":"(Liu et al., 2023)"},{"why":"Provides the ArXiv dataset used for the main summarization experiments.","marker":"(Cohan et al., 2018)"},{"why":"Provides the GovReport dataset used for the main summarization experiments.","marker":"(Huang et al., 2021)"},{"why":"Introduces the multi-agent debate approach that grounds the paper's multi-LLM collaboration strategy.","marker":"(Liang et al., 2024)"},{"why":"Introduces the round-table consensus framework that motivates the decentralized voting and refinement design.","marker":"(Chen et al., 2024)"},{"why":"Supplies the coherence, conciseness, and fluency rating criteria used in the human evaluation.","marker":"(Conroy and Dang, 2008)"},{"why":"Shows LLM summaries can rival or beat fine-tuned models, providing context for the single-LLM baseline choice.","marker":"(Goyal et al., 2023)"}],"fun_headline_variants":["Multi-LLM summaries beat single LLM by up to 3x","Generate-and-evaluate with multiple LLMs improves summaries 3x","Multiple LLM draft-and-vote strategy yields up to 3x better summaries","Voting between LLM summaries lifts quality up to threefold","Several LLMs draft, then vote: summaries improve up to 3x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes the single-LLM baselines are the right control; those baselines summarize each chunk once and concatenate the results, while the multi-LLM pipeline adds a second chunk-and-resummarize stage, so some of the gain could come from that extra aggregation pass rather than from having multiple models.","fun_headline_variants_meta":{"raw":{"variants":["Multi-LLM summaries beat single LLM by up to 3x","Generate-and-evaluate with multiple LLMs improves summaries 3x","Multiple LLM draft-and-vote strategy yields up to 3x better summaries","Voting between LLM summaries lifts quality up to threefold","Several LLMs draft, then vote: summaries improve up to 3x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00125,"raw_usage":{"total_tokens":5098,"prompt_tokens":890,"completion_tokens":4208,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":4111}},"tokens_in":506,"tokens_out":4208,"duration_ms":26422,"temperature":1.0,"reasoning_tokens":4111,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:23:16.547803+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the strongest single LLM through the same two-stage chunk-then-resummarize pipeline used by the multi-LLM framework, and compare its ROUGE and BLEU scores with the multi-LLM results in Table 2; if the single LLM with hierarchical aggregation matches or exceeds the multi-LLM scores, the claimed multi-LLM advantage is not attributable to collaboration.","supporting_citations":[],"review_version":1}