Pith. sign in

REVIEW 5 major objections 6 minor 13 references

CNNSum: Exploring Long-Context Summarization with Large Language Models in Chinese Novels

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CNNSum is a multi-scale Chinese novel summarization benchmark that the paper argues gives more reliable long-context evaluation, and fine-tuning on concatenated short summaries substantially improves long-context summarization.

desk verdict CNNSum is a genuinely useful new Chinese long-context summarization benchmark, but the gold summaries rest on unvalidated LLM synopses and the fine-tuning numbers are inflated by post-hoc checkpoint selection. read the letter →

arxiv 2412.02819 v5 pith:5EGLICJJ submitted 2024-12-03 cs.CL cs.AI

Lingxiao Wei , He Yan , Xiangju Lu , Junmin Zhu , Jun Wang , Wei Zhang This is my paper · ORCID
classification cs.CLcs.AI
keywords CNNSumlong-contextsummarizationChinesenovelsLLMevaluationfine-tuningRoPEextrapolationpositioninterpolationhumanannotation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish two things: that CNNSum, a 695-sample Chinese novel summarization benchmark spanning 16k to 128k tokens, offers more reliable long-context summarization evaluation than existing benchmarks, and that fine-tuning language models on concatenated short summaries substantially improves their long-context summarization ability. The benchmark is built from a newly collected corpus of 103 web novels with human annotation assisted by LLM-generated chapter synopses, organized into four length subsets. Benchmarking many commercial and open models, the authors find that one flagship model's subjective commentary produces vague summaries, that prompt placement and Base-versus-Chat model versions create large performance gaps, and that small long-context models can be more cost-effective. If the paper is right, it supplies a reusable evaluation resource for Chinese long-context summarization and a cheap fine-tuning recipe that activates extrapolation in models with scaled rotary position embeddings.

What carries the argument

The central objects are the CNNSum construction itself and the position-embedding property it exploits. CNNSum uses a sliding-window chapter sampler that accepts a window only when its token length falls inside a target range around each subset's nominal length, with a lower bound that keeps samples from being trivially short; this produces four subsets whose mean lengths sit near 16k, 32k, 64k, and 128k. The second mechanism is Adjusted Base Frequency (ABF), a pre-training choice that raises the base of rotary position embeddings so position frequencies decay more slowly; the paper argues that fine-tuning on concatenated short data activates this latent extrapolation ability, while adding interpolation methods such as PI, NTK, or YaRN on top of ABF requires careful validation.

What would settle it

Have two independent annotators write summaries directly from the source text, without seeing LLM synopses, for a random sample of about fifty inputs per CNNSum subset, and score them against the released gold with the same ROUGE-L and BERTScore pipeline; if human-to-gold agreement is no higher than typical model-to-gold agreement, or if model rankings change when evaluated against the independent summaries, the gold summaries carry annotation bias and the reliability claim collapses.

Watch

Extended reading notes

Core claim

The central claim, stated on the paper's own terms, is that CNNSum is a multi-scale Chinese novel summarization benchmark that avoids the leakage, short-length, and narrow-length-distribution problems of prior benchmarks, and that the empirical picture it gives is different from and more reliable than the main existing Chinese long-context summarization benchmark. Using CNNSum, the paper reports that long-context summarization relies mainly on memory of key plots rather than on reasoning, that small models can match or beat larger ones, that Base versions fine-tune better than Chat versions, and that concatenating short summaries into 16k and 32k training sequences substantially improves extrapolation to 128k for models whose RoPE base is scaled by Adjusted Base Frequency.

Load-bearing premise

The benchmark's gold summaries must faithfully capture the key plots of the novels, but they are produced by merging LLM-generated chapter synopses, so systematic LLM omissions or distortions could propagate into every score in the paper.

Editorial extensions

If this is right

  • A summarization benchmark's length distribution should be controlled and reported; uniform-length sample mixes can hide extrapolation failures by letting short, easy samples carry the average score.
  • Fine-tuning on concatenated short summaries is a cheap alternative to expensive long-context instruction data for improving long-context summarization.
  • Base model versions are the better starting point for this fine-tuning; instruction-tuned versions are more prone to repetition and prompt sensitivity under extrapolation.
  • For models with ABF-scaled RoPE, plain-RoPE fine-tuning is more stable than layering on PI, NTK, or YaRN interpolation, which can degrade high-frequency position information.
  • Long-context summarization performance should be interpreted primarily as stable memory of key plots plus instruction-following at length, not as a direct reflection of reasoning ability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the gold summaries are as faithful as claimed, the same concatenated-short-data recipe should transfer to other narrative domains, including English novels, because the mechanism is positional extrapolation rather than Chinese-specific annotation.
  • A decisive test of the reliability claim would be to build a second gold set by having independent annotators summarize a random subset of the same inputs directly from the raw text, bypassing LLM synopses, and check whether model rankings on CNNSum survive.
  • The observation that subjective commentary lowers ROUGE suggests automatic metrics may penalize a legitimate summarization style; a human-preference study over CNNSum outputs could decide whether vague subjective summaries are truly worse or merely less n-gram-overlapping.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The manuscript introduces CNNSum, a Chinese long-context novel summarization benchmark constructed from 103 web-serialized novels. It provides four length-targeted subsets (L/16k, XL/32k, 2XL/64k, 3XL/128k) totaling 695 samples, with summaries produced by 23 annotators who read LLM-generated chapter synopses and merge/rewrite them. The paper reports extensive zero-shot evaluations of commercial and open-source LLMs, a qualitative error taxonomy of abnormal outputs, and fine-tuning experiments with concatenated short summaries and RoPE scaling methods. Its main claims are that advanced LLMs often produce subjective/vague summaries; small long-context models are cost-effective; prompt-version gaps are large but mitigated by fine-tuning; Base versions fine-tune better; short-context concatenated data improves long-context summarization, especially for ABF-scaled RoPE; and CNNSum provides more reliable evaluations than CLongEval-LStSum.

Significance. The resource is potentially valuable: the corpus is new, the multi-scale length-targeted sampling with explicit leakage filtering addresses a real gap in the literature, and the qualitative error taxonomy plus the cheap fine-tuning recipe are actionable for practitioners. If the reference summaries can be validated against the source text, CNNSum could become a standard benchmark for Chinese long-context summarization, and the fine-tuning findings could guide practical model extension. However, the paper's broader empirical claims currently outrun the evidence: the gold summaries are only indirectly grounded in the source text, selective sample exclusion and post-hoc checkpoint selection are acknowledged in the appendices, and the 'more reliable than other benchmarks' claim is partly self-referential. The paper is transparent about several of these issues, which is commendable, but they need to be resolved or substantially softened before the conclusions can be accepted.

major comments (5)
  1. [3.3] The gold summaries are produced from LLM-generated chapter synopses (Appendix B lists Qwen, Doubao, Kimi, and Gemini as synopsis generators), and the annotators 'read each synopsis' rather than the original chapter text. This is load-bearing because every ROUGE-L and BERTScore in Tables 2-3, the error-type analysis in Section 4.4, and the CNNSum-vs-CLongEval reliability comparison in Section 4.6 are computed against these references. If the synopsis model omits, distorts, or re-emphasizes plot events, those errors are inherited by the gold summaries, and a model that correctly retrieves the omitted content would be scored lower. The paper reports no inter-annotator agreement, no error-rate audit, and no test of whether the human rewrites recover facts absent from the synopses; the 2XL/3XL consistency review appears to compare against the same synopses. Please provide validation such as a random sample re-annotated directly from full chapters, a factuality check of references against the source text, and inter-annotator agreement statistics.
  2. [4.6] The claim that 'CNNSum provides more reliable evaluation results' is argued by showing that fine-tuned model scores on CNNSum differ from scores on CLongEval-LStSum (e.g., original RoPE vs. PI behavior, 3XL vs. Large in Figures 2-4). Since CNNSum was deliberately constructed with narrow length ranges around the target lengths, the benchmark's design mechanically removes the mixed-length effect that drives CLongEval's misleading aggregate scores; reading the reliability conclusion off CNNSum itself is close to circular. A convincing demonstration requires an independent criterion: for example, human quality ratings of the fine-tuned outputs, agreement of CNNSum-based rankings with another judge, or per-sample error analysis showing that CNNSum's 3XL scores track per-sample difficulty better than CLongEval's. Without such validation, the superiority claim should be softened.
  3. [D.2] For each fine-tuning configuration, the authors 'evaluated multiple checkpoints between 400 and 500 steps and selected the best result' (and similarly for the 32k continuation), and the same test benchmarks are then used for the extrapolation comparisons in Sections 4.5-4.6. This post-hoc checkpoint selection on the test sets makes the reported differences (e.g., Base vs. Chat, PI vs. NTK in Figures 2-4 and Table 11) optimistic and difficult to reproduce. Please report checkpoint selection on a held-out validation set, show the trajectory across all evaluated checkpoints, or at least provide variance/error bars across the three repetitions that the paper says were run.
  4. [D.1] The paper excludes 29% of Gemini samples and 8% of Doubao 3XL samples due to content-safety blocks and length limits, and notes that 'these cases introduce score bias,' but it does not quantify the direction or magnitude of that bias. Because Tables 2-3 compare these commercial models against each other and against open-source models, the exclusion can change rankings (for instance, Gemini's high recall and stability in Table 3 could be affected). Please report the number of excluded samples per subset and re-run the main comparisons on the common subset of samples that all models could process, or provide a sensitivity analysis that bounds the effect of the missing samples.
  5. [Tables 2-3 and Figures 2-4] No confidence intervals, error bars, or significance tests are reported for any of the ROUGE-L or BERTScore comparisons. Several headline conclusions rest on small differences (e.g., BERTScore gaps of 0.1-0.5 in Table 3, or prompt-MSE values in Table 2), while other differences are large but based on excluded or post-hoc-selected samples. Given the annotation and exclusion issues above, the quantitative claims need at least bootstrap confidence intervals or per-sample standard errors, and the fine-tuning claims need variability across the three repetitions.
minor comments (6)
  1. [3.3] The notation '2~3XL' should be written as '2XL and 3XL' for clarity.
  2. [C.1] The text says Chinese-team models 'achieve high encoding efficiency with a vocabulary size of less than 10k,' but Table 8 reports Yi at 64,000, ChatGLM at 65,024, and InternLM at 92,544; this should be 'less than 100k'.
  3. [Figures 2-4] The captions should state explicitly that the scores are ROUGE-L, define 's' and the PI/NTK settings, and explain the axis labels; the current figures are hard to interpret in grayscale.
  4. [4.4] The heading 'Why Do GPT-4o Fail?' is stronger than the evidence supports; 'underperform' or 'why does GPT-4o underperform' would be more proportionate, since GPT-4o still produces coherent summaries on most subsets.
  5. [A] The ProLong LDS comparisons in Table 7 use different context windows across datasets (16k for CNNSum-L, 9k for LongBench-VCSUM, 32k for others); the reported LDS differences may partly reflect window size rather than intrinsic long-dependency structure, and this caveat should be stated.
  6. [1 and 3.3] Phrases such as 'significantly superior in design and construction' and 'making it one of the best annotation strategies' are unsupported claims; they should be tempered or justified with evidence.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CNNSum's claims rest on external model measurements, not on fitting or self-citation.

full rationale

The paper's central claims—benchmarking LLMs on CNNSum, fine-tuning with concatenated short summaries, and comparing CNNSum with CLongEval—are empirical. The benchmark construction (Sections 3.1–3.3) uses a new corpus and length-stratified sampling; the gold summaries are generated from LLM chapter synopses that human annotators merge and rewrite, an acknowledged data-quality assumption (Section 3.3, Limitations in Section 6), but not a fitted parameter or a prediction derived from the benchmark itself. The fine-tuning experiments use external model checkpoints and held-out summarization data, and the improvement on 16k–128k subsets is read off the same CNNSum references; while the trustworthiness of those references is a validity risk (no inter-annotator agreement or fact-recovery audit is reported), it does not make the derivation circular. The 'more reliable evaluation' claim in Section 4.6 is supported by head-to-head extrapolation curves, human inspection of output degradation, and comparisons against CLongEval, not by definition. No load-bearing self-citations, uniqueness theorems, or ansatz-smuggling citations were found. Therefore no circular step meets the evidentiary bar of an exhibited reduction to inputs.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on domain assumptions about annotation quality, corpus representativeness, leakage filtering, and metric validity. The free parameters are mostly experimental design choices and post-hoc checkpoint selection rather than fitted theoretical constants. No new physical or conceptual entities are introduced, so the graviton problem does not apply.

free parameters (3)
  • Fine-tuning checkpoint selection = Best ROUGE-L among steps 400-500 (16k) and 200-300 (32k)
    Appendix D.2 states 'we evaluated multiple checkpoints between 400 and 500 steps and selected the best result'; this is post-hoc selection on the benchmark and inflates reported improvements.
  • PI/NTK interpolation scale s = 2, 4, 8, 16
    Extrapolation conclusions in Section 4.6 and Appendix G come from a grid over s; different s values give different extrapolation rankings, so the choice of s is part of what the results support.
  • Training sequence concatenation lengths = 14k-18k (average 16k) and 30k-34k (average 32k)
    Fine-tuning data lengths were chosen to match CNNSum L and XL; this design choice is reasonable but is an assumption that the target lengths should mirror training lengths.
assumptions (5)
  • domain assumption Chinese web-serialized novels with clear chapter structure and long plot arcs are representative of long-context summarization
    Section 3.1 selects 103 books and removes short-story collections, implicitly assuming long dependencies are what matters.
  • domain assumption LLM-generated per-chapter synopses, merged and rewritten by human annotators, are accurate enough to serve as gold summaries
    Section 3.3 relies on commercial LLMs for initial synopses and a single secondary review; no inter-annotator agreement is reported.
  • domain assumption Filtering out books identified by Qwen2-72B-Instruct sufficiently removes leakage for all evaluated models
    Section 3.1 removes 27 books after the detector recognizes them, but this does not guarantee other models have not seen the remaining corpus.
  • domain assumption ProLong's long-dependency score computed with Qwen2.5-7B is a valid measure of dataset quality
    Appendix A substitutes Qwen2.5-7B for the default OPT-350M and treats ProLong's LDS as meaningful for CNNSum.
  • domain assumption ROUGE-L with jieba tokenization is an adequate automatic proxy for summary quality
    Section 4.2 uses ROUGE-L as the primary metric; the paper itself acknowledges its divergence from human preferences, so metric validity is an assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CNNSum: Exploring Long-Context Summarization with Large Language Models in Chinese Novels." pith.science (2026). https://pith.science/paper/5EGLICJJ

@misc{pith2026241202819,
  author       = {Pith},
  title        = {Pith review of: CNNSum: Exploring Long-Context Summarization with Large Language Models in Chinese Novels},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5EGLICJJ}},
  note         = {Machine review of arXiv:2412.02819}
}
read the original abstract

Large language models (LLMs) have been well-researched in various long-context tasks. However, the scarcity of long-context summarization datasets hinders progress in this area. To address this, we introduce CNNSum, a multi-scale long-context summarization benchmark based on Chinese novels, featuring human-driven annotations across four subsets totaling 695 samples, with lengths ranging from 16k to 128k. We benchmark numerous LLMs and conduct detailed human assessments to summarize abnormal output types. Furthermore, we extensively explore how to improve long-context summarization. In our study: (1) Advanced LLMs may generate much subjective commentary, leading to vague summaries. (2) Currently, long-context summarization mainly relies on memory ability. The advantages of Large LLMs are hard to utilize, thus small LLMs are more cost-effective. (3) Different prompt types paired with various version models may cause large performance gaps. In further fine-tuning, these can be mitigated, and the Base version models perform better. (4) LLMs with RoPE-base scaled exhibit strong extrapolation potential; using short-context data can significantly improve long-context summarization performance. However, further applying other interpolation methods requires careful selection. (5) CNNSum provides more reliable evaluation results than other benchmarks. We release CNNSum to advance future research.(https://github.com/CxsGhost/CNNSum)

Figures

Figures reproduced from arXiv: 2412.02819 by the authors.

Figure 1
Figure 1. Results of fine-tuned models on CNNSum. pendix H.7. Training small long-context models is significantly less challenging and cheaper, and thus they are cost-effective for novel summarization. 4.5 Ablation Study for Fine-Tuning We have performed analyses based on various gen￾eral LLMs. However, fine-tuning powerful mod￾els for summarization tasks needs further explo￾ration. We fine-tune Yi-6B series and Qwen1.5-7B se… view at source ↗
Figure 2
Figure 2. Results comparison of Fine-tuned Qwen1.5-7B on CNNSum and CLongEval-LStSum. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Moving Median ROUGE-L (window size=15) of fine-tuned models on CLongEval-LStSum Large demonstrate better extrapolation potential and are more suitable for efficient fine-tuning. For example, we observe that Qwen1.5-7B-Chat exhibits worse stability when extrapolating on 2XL, having more completely meaningless repetition. Prompt-IB vs Prompt-IE In [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Results comparison of Fine-tuned Yi-6B on CNNSum and CLongEval-LStSum. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: The sample length distribution of each subset, [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 9 canonical work pages

  1. [1]

    内容必须严格和小说的内容事实相符，准确描述客观事实和人物相关信息。

    work page

  2. [2]

    不要杜撰、改写原文的人物信息和客观事实，不要生成没有的情节和人物名字。 3.当小说中明确说明了人物的年龄、外貌、职业等身份信息时，一定要输出，如果没有明 确说明身份信息，可以不输出。

    work page

  3. [3]

    If no such details are specified, do not include them

    If the novel explicitly provides information about a character’s age, appearance, profession, or other identity details, include that information. If no such details are specified, do not include them

    work page

  4. [4]

    In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence , pages 8299– 8307

    Beyond the limits: a survey of techniques to extend the context length in large language models. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence , pages 8299– 8307. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz,...

    work page 2020

  5. [5]

    嗤啦

    Benchmarking benchmark leakage in large language models. arXiv preprint arXiv:2404.18824. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng ...

    arXiv 2024

  6. [8]

    字数不超过400字，语言要凝练，不要输出细节。

    work page

  7. [9]

    Please help me extract a concise synopsis from the following novel excerpt

    不要在结尾输出“故事、讲述、揭示“等总结性语句。 下面是小说内容： Chapter Here ———————————————————————————————————————— You are a highly skilled artificial intelligence trained in Chinese language comprehension and summarization. Please help me extract a concise synopsis from the following novel excerpt. The requirements are as follows:

    work page

  8. [10]

    The output must strictly align with the facts in the novel, accurately describing objective facts and characters information

    work page

Show all 13 references
  1. [11]

    Avoid generating non-existent events or character names

    Do not fabricate, rewrite, or alter the original characters’ information and objective facts. Avoid generating non-existent events or character names

  2. [13]

    the story,

    The summary should not exceed 400 words. The language should be concise, without unnecessary details. Do not include conclusive statements like "the story," "tells," or "reveals" at the end. Be- low is the novel content: Chapter Here

  3. [2023]

    In Proceed- ings of the 17th Conference of the European Chap- ter of the Association for Computational Linguistics, pages 1650–1669

    Longeval: Guidelines for human evaluation of faithfulness in long-form summarization. In Proceed- ings of the 17th Conference of the European Chap- ter of the Association for Computational Linguistics, pages 1650–1669. Wojciech Kry´sci´nski, Nazneen Rajani, Divyansh Agar- wal,...

    2022

  4. [2024]

    In The Twelfth International Conference on Learning Representations

    Booookscore: A systematic exploration of book-length summarization in the era of llms. In The Twelfth International Conference on Learning Representations. Guanzheng Chen, Xin Li, Zaiqiao Meng, Shangsong Liang, and Lidong Bing. 2024a. Clex: Continuous length extrapolation for ...

    2024 arXiv

  5. [2025]

    In International Confer- ence on Learning Representations

    World model on million-length video and lan- guage with ringattention. In International Confer- ence on Learning Representations. Hao Liu, Matei Zaharia, and Pieter Abbeel. 2024a. Ring attention with blockwise transformers for near- infinite context. International Conference o...

    2022 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.