{"id":"3dcc199b-aba2-4a98-8fc0-74a00a38f6ca","arxiv_id":"2412.02819","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CNNSum adds a 16k-128k Chinese novel summarization benchmark with human annotations and evidence that short-context fine-tuning, base models, and RoPE-base scaling improve long-context summarization.","lead":"CNNSum is a new Chinese benchmark for summarizing long novel excerpts, with 695 human-written summaries for texts from 16,000 to 128,000 tokens. The paper tests many language models and fine-tuning methods and reports that small models are cost-effective, that prompt placement strongly affects quality, and that base models often fine-tune better than chat versions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gold summaries are built from LLM chapter synopses, not full text; unmeasured synopsis omissions or distortions could bias every ROUGE/BERTScore comparison.","rationale":"The reader identified the trustworthiness of the gold summaries as the weakest assumption, citing the LLM-assisted annotation process and the absence of inter-annotator agreement statistics. My stress-test agrees and sharpens the concern: the pipeline in Section 3.3 means human annotators read LLM-generated chapter synopses rather than the original chapters. This is a concrete, potentially systematic source of reference error, because any omission or distortion in the synopsis propagates into the gold summary regardless of human effort. The reader's proposed fixes (releasing data, reporting agreement statistics, confidence intervals) are appropriate, but the specific test that would settle the matter is to compare the current gold against gold produced directly from source text. If that comparison shows large divergence, the central benchmark claim is weakened independently of the other issues the reader lists. Since the reader already returned CONDITIONAL, my analysis does not move the verdict; it strengthens the condition. I am not raising any objection to the fine-tuning experiments' methodology other than the checkpoint-selection issue already noted by the reader, and I do not see internal inconsistency in the experimental logic. The paper has real value in its multi-scale sampling and extensive benchmarking, but the reference-validity question must be resolved before the benchmark's reliability claim can be accepted.","tokens_in":33718,"tokens_out":2452,"duration_ms":27520,"concrete_test":"Select 30-50 samples from the 2XL and 3XL subsets. Have two independent annotators produce gold summaries directly from the full chapter text, without access to the existing LLM-generated synopses or the current CNNSum gold. Then (a) measure per-sample ROUGE-L and BERTScore agreement between the existing CNNSum gold and the new direct-from-source gold, and (b) recompute the model rankings from Table 2 on this subset using both gold versions. If agreement is low (for example, median ROUGE-L below 0.6) or if the ranking of the top commercial and open-source models changes materially, the benchmark's reference quality is not sufficient to support the paper's comparative claims.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The benchmark's central value is that its references are trustworthy, since all model scores and the reliability comparison against CLongEval are computed against them. Section 3.3 describes the annotation pipeline as: first generate a plot synopsis for each chapter with a commercial LLM, then have human annotators read that synopsis, select key plots, and rewrite them into a final summary. The annotators therefore never directly read the full chapter text; they work from an LLM-compressed representation. If the synopsis model omits a plot event, distorts a character fact, or imposes its own emphasis, that error is inherited by the gold summary. A model that correctly retrieves the omitted event is then scored as having lower ROUGE overlap, while a model that mirrors the synopsis structure or wording can score higher. No inter-annotator agreement, error-rate audit, or check that human rewrites recover facts absent from the synopses is reported. Only 2XL/3XL samples receive a second consistency review, and even that review presumably compares the merged summary against the same synopses rather than the original text. The downstream claims about model quality, extrapolation, and the superiority of CNNSum over CLongEval all rest on this unvalidated gold. The length-stratified sampling is a real improvement, but it cannot establish reliable evaluation if the references themselves are systematically incomplete.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces CNNSum, a Chinese long-context novel summarization benchmark constructed from 103 web-serialized novels. It provides four length-targeted subsets (L/16k, XL/32k, 2XL/64k, 3XL/128k) totaling 695 samples, with summaries produced by 23 annotators who read LLM-generated chapter synopses and merge/rewrite them. The paper reports extensive zero-shot evaluations of commercial and open-source LLMs, a qualitative error taxonomy of abnormal outputs, and fine-tuning experiments with concatenated short summaries and RoPE scaling methods. Its main claims are that advanced LLMs often produce subjective/vague summaries; small long-context models are cost-effective; prompt-version gaps are large but mitigated by fine-tuning; Base versions fine-tune better; short-context concatenated data improves long-context summarization, especially for ABF-scaled RoPE; and CNNSum provides more reliable evaluations than CLongEval-LStSum.","tokens_in":33960,"tokens_out":7496,"duration_ms":74710,"significance":"The resource is potentially valuable: the corpus is new, the multi-scale length-targeted sampling with explicit leakage filtering addresses a real gap in the literature, and the qualitative error taxonomy plus the cheap fine-tuning recipe are actionable for practitioners. If the reference summaries can be validated against the source text, CNNSum could become a standard benchmark for Chinese long-context summarization, and the fine-tuning findings could guide practical model extension. However, the paper's broader empirical claims currently outrun the evidence: the gold summaries are only indirectly grounded in the source text, selective sample exclusion and post-hoc checkpoint selection are acknowledged in the appendices, and the 'more reliable than other benchmarks' claim is partly self-referential. The paper is transparent about several of these issues, which is commendable, but they need to be resolved or substantially softened before the conclusions can be accepted.","major_comments":[{"comment":"The gold summaries are produced from LLM-generated chapter synopses (Appendix B lists Qwen, Doubao, Kimi, and Gemini as synopsis generators), and the annotators 'read each synopsis' rather than the original chapter text. This is load-bearing because every ROUGE-L and BERTScore in Tables 2-3, the error-type analysis in Section 4.4, and the CNNSum-vs-CLongEval reliability comparison in Section 4.6 are computed against these references. If the synopsis model omits, distorts, or re-emphasizes plot events, those errors are inherited by the gold summaries, and a model that correctly retrieves the omitted content would be scored lower. The paper reports no inter-annotator agreement, no error-rate audit, and no test of whether the human rewrites recover facts absent from the synopses; the 2XL/3XL consistency review appears to compare against the same synopses. Please provide validation such as a random sample re-annotated directly from full chapters, a factuality check of references against the source text, and inter-annotator agreement statistics.","section":"3.3"},{"comment":"The claim that 'CNNSum provides more reliable evaluation results' is argued by showing that fine-tuned model scores on CNNSum differ from scores on CLongEval-LStSum (e.g., original RoPE vs. PI behavior, 3XL vs. Large in Figures 2-4). Since CNNSum was deliberately constructed with narrow length ranges around the target lengths, the benchmark's design mechanically removes the mixed-length effect that drives CLongEval's misleading aggregate scores; reading the reliability conclusion off CNNSum itself is close to circular. A convincing demonstration requires an independent criterion: for example, human quality ratings of the fine-tuned outputs, agreement of CNNSum-based rankings with another judge, or per-sample error analysis showing that CNNSum's 3XL scores track per-sample difficulty better than CLongEval's. Without such validation, the superiority claim should be softened.","section":"4.6"},{"comment":"For each fine-tuning configuration, the authors 'evaluated multiple checkpoints between 400 and 500 steps and selected the best result' (and similarly for the 32k continuation), and the same test benchmarks are then used for the extrapolation comparisons in Sections 4.5-4.6. This post-hoc checkpoint selection on the test sets makes the reported differences (e.g., Base vs. Chat, PI vs. NTK in Figures 2-4 and Table 11) optimistic and difficult to reproduce. Please report checkpoint selection on a held-out validation set, show the trajectory across all evaluated checkpoints, or at least provide variance/error bars across the three repetitions that the paper says were run.","section":"D.2"},{"comment":"The paper excludes 29% of Gemini samples and 8% of Doubao 3XL samples due to content-safety blocks and length limits, and notes that 'these cases introduce score bias,' but it does not quantify the direction or magnitude of that bias. Because Tables 2-3 compare these commercial models against each other and against open-source models, the exclusion can change rankings (for instance, Gemini's high recall and stability in Table 3 could be affected). Please report the number of excluded samples per subset and re-run the main comparisons on the common subset of samples that all models could process, or provide a sensitivity analysis that bounds the effect of the missing samples.","section":"D.1"},{"comment":"No confidence intervals, error bars, or significance tests are reported for any of the ROUGE-L or BERTScore comparisons. Several headline conclusions rest on small differences (e.g., BERTScore gaps of 0.1-0.5 in Table 3, or prompt-MSE values in Table 2), while other differences are large but based on excluded or post-hoc-selected samples. Given the annotation and exclusion issues above, the quantitative claims need at least bootstrap confidence intervals or per-sample standard errors, and the fine-tuning claims need variability across the three repetitions.","section":"Tables 2-3 and Figures 2-4"}],"minor_comments":[{"comment":"The notation '2~3XL' should be written as '2XL and 3XL' for clarity.","section":"3.3"},{"comment":"The text says Chinese-team models 'achieve high encoding efficiency with a vocabulary size of less than 10k,' but Table 8 reports Yi at 64,000, ChatGLM at 65,024, and InternLM at 92,544; this should be 'less than 100k'.","section":"C.1"},{"comment":"The captions should state explicitly that the scores are ROUGE-L, define 's' and the PI/NTK settings, and explain the axis labels; the current figures are hard to interpret in grayscale.","section":"Figures 2-4"},{"comment":"The heading 'Why Do GPT-4o Fail?' is stronger than the evidence supports; 'underperform' or 'why does GPT-4o underperform' would be more proportionate, since GPT-4o still produces coherent summaries on most subsets.","section":"4.4"},{"comment":"The ProLong LDS comparisons in Table 7 use different context windows across datasets (16k for CNNSum-L, 9k for LongBench-VCSUM, 32k for others); the reported LDS differences may partly reflect window size rather than intrinsic long-dependency structure, and this caveat should be stated.","section":"A"},{"comment":"Phrases such as 'significantly superior in design and construction' and 'making it one of the best annotation strategies' are unsupported claims; they should be tempered or justified with evidence.","section":"1 and 3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is close to being acceptable if the authors treat their own caveats as central rather than peripheral. The reference-quality validation and the self-referential reliability claim are the two points I would insist on; the post-hoc checkpoint selection and sample-exclusion bias also need transparent handling. I would encourage the editor to send the benchmark-construction section to a reviewer with human-annotation methodology expertise, since the dataset's reuse value depends on the trustworthiness of the gold summaries."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Main takeaway: this paper contributes a genuinely new resource — a Chinese long-context summarization benchmark with 695 human-annotated samples across 16k, 32k, 64k, and 128k brackets, built from web novels with leakage filtering. The multi-scale sampling is thoughtful, and the broad model comparisons, prompt placement experiments, and fine-tuning with concatenated short data give the community useful guidance. It is not a restatement of prior work.\n\nSoft spots are real but mostly fixable. The annotation pipeline generates gold summaries from LLM chapter synopses that human annotators merge and rewrite; the paper does not report inter-annotator agreement, an error-rate audit, or any check that rewrites recover facts absent from the synopses. If the synopsis model omits or distorts an event, that error propagates into every ROUGE and BERTScore. This does not kill the benchmark, but it means the gold standard is not independently validated against source text.\n\nSecond, fine-tuning numbers are inflated by post-hoc checkpoint selection (Appendix D.2), where the best checkpoint per model is chosen on the evaluation data. That undermines the quantitative comparisons, including extrapolation claims. The acknowledged exclusions (29% Gemini, 8% Doubao) and lack of error bars add fragility.\n\nThird, the claim that CNNSum is more reliable than CLongEval is partly self-referential, though the moving-median analysis in Figure 3 gives it independent support: CLongEval's uniform length distribution really can let shorter, easier samples dominate averages.\n\nVerdict: worth a serious referee, not a desk rejection. The dataset is a solid contribution. A revision should release code and data with a commit hash, report agreement statistics and confidence intervals, and either drop checkpoint selection or show results without it. The central resource is valuable; the quantitative wrapping is softer than the prose.","headline":"CNNSum is a genuinely useful new Chinese long-context summarization benchmark, but the gold summaries rest on unvalidated LLM synopses and the fine-tuning numbers are inflated by post-hoc checkpoint selection.","tokens_in":752,"tokens_out":1236,"would_cite":true,"duration_ms":38082,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CNNSum is a multi-scale Chinese novel summarization benchmark that the paper argues gives more reliable long-context evaluation, and fine-tuning on concatenated short summaries substantially improves long-context summarization.","keywords":["CNNSum","long-context summarization","Chinese novels","LLM evaluation","fine-tuning","RoPE extrapolation","position interpolation","human annotation"],"falsifier":"Have two independent annotators write summaries directly from the source text, without seeing LLM synopses, for a random sample of about fifty inputs per CNNSum subset, and score them against the released gold with the same ROUGE-L and BERTScore pipeline; if human-to-gold agreement is no higher than typical model-to-gold agreement, or if model rankings change when evaluated against the independent summaries, the gold summaries carry annotation bias and the reliability claim collapses.","tokens_in":33506,"feed_emoji":"📚","tokens_out":9845,"duration_ms":92241,"temperature":0.7,"pith_summary":"The paper tries to establish two things: that CNNSum, a 695-sample Chinese novel summarization benchmark spanning 16k to 128k tokens, offers more reliable long-context summarization evaluation than existing benchmarks, and that fine-tuning language models on concatenated short summaries substantially improves their long-context summarization ability. The benchmark is built from a newly collected corpus of 103 web novels with human annotation assisted by LLM-generated chapter synopses, organized into four length subsets. Benchmarking many commercial and open models, the authors find that one flagship model's subjective commentary produces vague summaries, that prompt placement and Base-versus-Chat model versions create large performance gaps, and that small long-context models can be more cost-effective. If the paper is right, it supplies a reusable evaluation resource for Chinese long-context summarization and a cheap fine-tuning recipe that activates extrapolation in models with scaled rotary position embeddings.","feed_headline":"Fine-tuning on short summaries boosts long-context models","feed_subtitle":"A 695-sample Chinese novel benchmark from 16k to 128k tokens reveals the recipe and the failure modes.","key_machinery":"The central objects are the CNNSum construction itself and the position-embedding property it exploits. CNNSum uses a sliding-window chapter sampler that accepts a window only when its token length falls inside a target range around each subset's nominal length, with a lower bound that keeps samples from being trivially short; this produces four subsets whose mean lengths sit near 16k, 32k, 64k, and 128k. The second mechanism is Adjusted Base Frequency (ABF), a pre-training choice that raises the base of rotary position embeddings so position frequencies decay more slowly; the paper argues that fine-tuning on concatenated short data activates this latent extrapolation ability, while adding interpolation methods such as PI, NTK, or YaRN on top of ABF requires careful validation.","core_discovery":"The central claim, stated on the paper's own terms, is that CNNSum is a multi-scale Chinese novel summarization benchmark that avoids the leakage, short-length, and narrow-length-distribution problems of prior benchmarks, and that the empirical picture it gives is different from and more reliable than the main existing Chinese long-context summarization benchmark. Using CNNSum, the paper reports that long-context summarization relies mainly on memory of key plots rather than on reasoning, that small models can match or beat larger ones, that Base versions fine-tune better than Chat versions, and that concatenating short summaries into 16k and 32k training sequences substantially improves extrapolation to 128k for models whose RoPE base is scaled by Adjusted Base Frequency.","pith_inferences":["If the gold summaries are as faithful as claimed, the same concatenated-short-data recipe should transfer to other narrative domains, including English novels, because the mechanism is positional extrapolation rather than Chinese-specific annotation.","A decisive test of the reliability claim would be to build a second gold set by having independent annotators summarize a random subset of the same inputs directly from the raw text, bypassing LLM synopses, and check whether model rankings on CNNSum survive.","The observation that subjective commentary lowers ROUGE suggests automatic metrics may penalize a legitimate summarization style; a human-preference study over CNNSum outputs could decide whether vague subjective summaries are truly worse or merely less n-gram-overlapping."],"forward_implications":["A summarization benchmark's length distribution should be controlled and reported; uniform-length sample mixes can hide extrapolation failures by letting short, easy samples carry the average score.","Fine-tuning on concatenated short summaries is a cheap alternative to expensive long-context instruction data for improving long-context summarization.","Base model versions are the better starting point for this fine-tuning; instruction-tuned versions are more prone to repetition and prompt sensitivity under extrapolation.","For models with ABF-scaled RoPE, plain-RoPE fine-tuning is more stable than layering on PI, NTK, or YaRN interpolation, which can degrade high-frequency position information.","Long-context summarization performance should be interpreted primarily as stable memory of key plots plus instruction-following at length, not as a direct reflection of reasoning ability."],"supporting_citations":[{"why":"CLongEval-LStSum is the main comparison benchmark; CNNSum's reliability claim rests on showing that its uniform length sampling produces misleading extrapolation scores.","marker":"(Qiu et al., 2024)"},{"why":"BooookScore's incremental-updating annotation protocol is the template CNNSum adapts with human merging and rewriting of LLM chapter synopses.","marker":"(Chang et al., 2024)"},{"why":"BookSum is the prior book-length summarization dataset and the source of the long data reused by CLongEval, against which CNNSum's novelty is defined.","marker":"(Kry´sci´nski et al., 2022)"},{"why":"Adjusted Base Frequency is the RoPE scaling that the fine-tuning experiments rely on to show extrapolation potential in models like Yi.","marker":"(Xiong et al., 2024)"},{"why":"YaRN is one of the interpolation methods tested on top of ABF-scaled models; the paper reports training observations that differ from YaRN's expectations.","marker":"(Peng et al., 2024)"},{"why":"Position Interpolation is the main interpolation method compared in the extrapolation experiments and is shown to behave differently on ABF models.","marker":"(Fei et al., 2024)"},{"why":"NTK-aware scaling is the other interpolation method compared, with results showing faster degradation at large scale factors.","marker":"(u/LocalLLaMA, 2023)"},{"why":"ROUGE-L is the primary automatic metric; every model comparison in the paper is computed against it.","marker":"(Lin, 2004)"},{"why":"BERTScore is the secondary automatic metric used for commercial models to check whether ROUGE trends persist.","marker":"(Zhang et al., 2020)"},{"why":"ProLong supplies the long-dependency score used to argue that CNNSum samples contain genuine long-range dependencies, not just long sequences.","marker":"(Chen et al., 2024b)"}],"fun_headline_variants":["Short summaries teach long-context models","Chinese novel benchmark reveals summarization tricks","Small models win on long summaries, study finds","CNNSum: 695 samples unlock long-context summarization","Short-context data boosts long summarization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's gold summaries must faithfully capture the key plots of the novels, but they are produced by merging LLM-generated chapter synopses, so systematic LLM omissions or distortions could propagate into every score in the paper.","fun_headline_variants_meta":{"raw":{"variants":["Short summaries teach long-context models","Chinese novel benchmark reveals summarization tricks","Small models win on long summaries, study finds","CNNSum: 695 samples unlock long-context summarization","Short-context data boosts long summarization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000551,"raw_usage":{"total_tokens":2634,"prompt_tokens":956,"completion_tokens":1678,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":1608}},"tokens_in":572,"tokens_out":1678,"duration_ms":12726,"temperature":1.0,"reasoning_tokens":1608,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:04:39.142497+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two independent annotators write summaries directly from the source text, without seeing LLM synopses, for a random sample of about fifty inputs per CNNSum subset, and score them against the released gold with the same ROUGE-L and BERTScore pipeline; if human-to-gold agreement is no higher than typical model-to-gold agreement, or if model rankings change when evaluated against the independent summaries, the gold summaries carry annotation bias and the reliability claim collapses.","supporting_citations":[],"review_version":1}