Pith. sign in

REVIEW 4 major objections 4 minor 35 references

StoryWriter: A Multi-Agent Framework for Long Story Generation

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuned StoryWriter models outscore GPT-4o on long stories

desk verdict Useful framework and dataset, but the evaluation doesn't prove the quality claims and the GPT-4o comparison is really about length adherence. read the letter →

arxiv 2506.16445 v1 pith:7NA5EUBE submitted 2025-06-19 cs.CL cs.AI

Haotian Xia , Hao Peng , Yunjia Qi , Xiaozhi Wang , Bin Xu , Lei Hou , Juanzi Li This is my paper · ORCID
classification cs.CLcs.AI
keywords longstorygenerationmulti-agentframeworkeventgraphnon-linearnarrationhistorycompressionsupervisedfine-tuningnarrativeevaluationlong-context
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that long story generation can be decomposed into three cooperating agents—an outline agent, a planning agent, and a writing agent—and that this division fixes the coherence and complexity failures of single-pass LLM generation. It reports that StoryWriter, built on GPT-4o-mini, outperforms DOC, Agents' Room, and GPT-4o-mini on both human and automated narrative-quality scores while producing stories averaging about 8,000 words. It also builds LongStory, a cleaned dataset of about 5,500 generated long stories, and shows that fine-tuning Llama3.1-8B and GLM4-9B on it yields models that beat GPT-4o on stories exceeding 2,000 words. A sympathetic reader would care because it points to a data-centric path for improving long-form narrative quality without requiring larger or longer-context models.

What carries the argument

The central mechanism is the three-agent architecture acting on an event graph. The outline agent builds a sequence of event tuples with characters and event-event relations; the planning agent decomposes events into sub-events and shuffles them across chapters via non-linear narration, preserving causal links; the writing agent uses ReIO (re-write input and output), where a Coordinator compresses the story history around the current event and rewrites the FinalWriter's output to keep it aligned with the outline. The event graph carries the global plot structure, and ReIO keeps the effective context short enough to avoid the attention degradation that the paper says appears beyond roughly 10,000 characters.

What would settle it

A counter-check would be a blind, pre-registered human study matching stories by length (or asking judges to rate truncated excerpts) and using at least a dozen independent raters with reported inter-annotator agreement: if StoryWriter's advantage over its baselines shrinks to noise under length control, the claimed superiority is an artifact of longer output.

Watch

Extended reading notes

Core claim

StoryWriter's central claim is that a multi-agent pipeline with an event-based outline, non-linear chapter planning, and dynamic history compression generates longer and better-rated stories than strong baselines. The paper presents human and GPT-4o-based evaluations across six dimensions—relevance, coherence, empathy, surprise, creativity, and complexity—reporting the best average scores and an average length of 8,081 words for StoryWriter. It further claims that supervised fine-tuning on the generated LongStory dataset transfers these strengths to smaller open models, with StoryWriter_GLM surpassing GPT-4o on stories longer than 2,000 words in both quality and length adherence.

Load-bearing premise

The evaluation pipeline captures narrative quality independently of output length and generator identity, with a small number of graduate students and a single GPT-4o judge producing scores reliable enough to support the 'significantly outperforms' claim without significance tests, error bars, or inter-annotator agreement.

Editorial extensions

If this is right

  • Structured event-based outlines, not raw long-context modeling, can carry the plot across thousands of words; removing the event outline collapses every evaluation score in the paper's ablation.
  • Decomposing events into sub-events and presenting them out of chronological order across chapters increases rated complexity and diversity while preserving coherence.
  • Compressing the writing history around the current event—rather than feeding the full history—avoids the attention degradation that appears beyond roughly 10,000 input characters.
  • Supervised fine-tuning on a generated long-story dataset improves both story quality and adherence to length constraints, even without explicit length-constraint training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same three-phase decomposition (outline, plan, write) could plausibly transfer to other long-form creative generation, such as screenplays or serialized non-fiction, where out-of-order presentation and global planning over long horizons are also central.
  • The improvement in length adherence after SFT, despite no explicit length training, suggests that simply training on long, structured outputs teaches length control; a controlled study that varies only the average training length could isolate this mechanism.
  • Because the evaluation used a small number of graduate raters and a single GPT-4o judge, a pre-registered human study with length-matched excerpts and inter-annotator agreement statistics would test whether the quality advantage is robust or partly an artifact of longer outputs.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents STORYWRITER, a multi-agent framework for long-form story generation composed of outline, planning, and writing agents. It also introduces LONGSTORY, a dataset of long stories generated by the framework, and fine-tuned models STORYWRITER_LLAMA and STORYWRITER_GLM. The authors evaluate against DOC, Agents' Room, and GPT-4o-mini, and report human and GPT-4o-based automatic scores across six narrative dimensions, as well as length-adherence scores for the fine-tuned models.

Significance. If the central claims were fully supported, the work would be a useful contribution: it provides a complete multi-agent recipe, a public dataset, and two fine-tuned models with code and data released. The design ideas—event-graph outlining, non-linear narration, and input/output rewriting for long-context generation—are sensible and potentially reusable. However, the evaluation evidence is underpowered and partly inconsistent, and the headline claim about surpassing GPT-4o is not supported by the paper's own table once the metrics are parsed correctly.

major comments (4)
  1. [Section 4, Table 3] The printed formula for the combined score, Sbar = (Sq + 20*Sl)/2, cannot reproduce any of the reported Overall values, because Sl is on a 0-100 scale and Sq on a 1-5 scale; the table's column order (Sl then Sq) and the plausible values are consistent with Sbar = (20*Sq + Sl)/2. This formula error must be corrected, and it changes how the Overall column is interpreted.
  2. [Section 4, Table 3] The claim in Section 1 that the fine-tuned models 'even surpass GPT-4o' on stories exceeding 2,000 words is not supported on the quality dimension: in the [2k,4k) bucket GPT-4o's Sq is 4.5 versus 4.1 for STORYWRITER_GLM, and GPT-4o's combined score 90.85 also exceeds STORYWRITER_GLM's 90.65. The only buckets where STORYWRITER_GLM leads the combined score are [4k,10k) and [10k,20k), and that lead comes almost entirely from the length-adherence component Sl, not from the quality score Sq. The abstract's 'significantly outperforms ... in both story quality and length' is therefore not substantiated for the fine-tuned comparison.
  3. [Section 4 and Abstract] The dataset statistics are internally inconsistent: the Abstract states about 6,000 stories with an average length of 8,000 words; the Introduction states about 6,000 stories with an average of 15,000 words; and Section 4 states a final curated set of 5,500 stories with an average of about 8,000 words. The authors must reconcile these numbers and state the exact count and average length used for training.
  4. [Section 3.1, Table 1] The report 'significantly outperforms' is not backed by any statistical test: no number of human evaluators is given, no inter-annotator agreement is reported, and no error bars or significance tests accompany the average scores. Because STORYWRITER's stories are much longer (average 8,081 words) than the baselines (1,078-3,134 words), the framework-level quality comparison is confounded with output length, and the claimed 'significantly outperforms' is not established.
minor comments (4)
  1. [Abstract] The Abstract contains a typo: 'STORYWRITER_GLM and STORYWRITER_GLM' should presumably be 'STORYWRITER_LLAMA and STORYWRITER_GLM'.
  2. [Section 3.3] Figure 3 is referenced in Section 3.3 but is not present in the submitted text; the authors should ensure it is included with clear axis labels and a caption that explains the window configurations.
  3. [Section 4] The paper states that the training does not involve explicit ability enhancement for following length constraints, which appears to contradict the observation that trained models adhere better to length constraints; consider discussing a possible mechanism for this effect.
  4. [Section 4] The paper says 'we adopt the same evaluation dataset MoPS in § 3.1' but then applies the LongBench-Write length-constrained setting; please clarify which exact test instances and prompts are used for each bucket.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the framework is evaluated against external MoPS test premises, the length metric is stated explicitly, and LONGSTORY training data is tested on held-out data rather than used to define the measured outcomes.

full rationale

The paper's central claims are supported by comparisons against external baselines on the MoPS test set, whose premises come from Ma et al. (2024) and are not generated by the paper. The content-quality judge is GPT-4o, not the generator, and no parameter of the framework is fitted to the test-set scores. The length score S_l is defined in Equation 1 as an explicit function of actual and required response length, so the length-adherence numbers are computed from a stated formula rather than predicted from fitted coefficients. The weighted score Sbar is printed in Section 4; although the printed formula appears inconsistent with the tabulated numbers, that is a correctness issue, not a circularity, and the metric itself is externally specified by LongBench-Write rather than derived from the paper's own conclusions. The LONGSTORY dataset is generated by STORYWRITER, but the fine-tuned models are evaluated on the held-out MoPS test premises; training on one's own generated data and testing on an external set is not circular. Self-citations to LongAlign and LongWriter are used as tooling and metric provenance, with the metric formula restated in the paper, so no load-bearing claim depends on an unverified self-citation or on a uniqueness theorem imported from the authors. The sliding-window setting in Section 3.3 is selected by the paper's own controlled comparison and then ablated, so it is a tuned design choice, not a fitted input relabeled as a prediction. No step of the derivation reduces to its inputs by definition or by construction.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim relies on two hand-picked choices (compression window, cleaning thresholds) and on the assumption that LLM and human judges measure story quality fairly. No new physical entities are introduced; the 'agents' are software components, not entities with independent evidence.

free parameters (2)
  • Sliding window length [2, k-1] for history compression = [2, k-1] (k-2 tokens compressed)
    Chosen empirically as best among five window options in Figure 3 (Section 3.3). The central ReIO mechanism depends on this compression choice.
  • Data cleaning criteria for LONGSTORY = not specified numerically
    Stories that are 'too short', 'do not meet format requirements', or 'exhibit low quality' are removed; thresholds are not defined, introducing post-hoc selection into the dataset construction.
assumptions (3)
  • domain assumption Causal and logical event links can be preserved under non-linear narration (analepsis and prolepsis), so readers reconstruct the event chain
    Invoked in Section 2.3 to justify the NLN strategy; borrowed from Genette, but not empirically tested here.
  • domain assumption GPT-4o as an automatic judge produces valid story-quality scores across the six dimensions
    Used as the primary automated evaluation in Sections 3 and 4; no validation against human judgments is reported.
  • domain assumption Dynamic compression of the writing history retains all information essential for later coherence
    ReIO compresses context in Section 2.4; the paper optimizes window length but does not test whether compressed summaries preserve all plot-critical facts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of StoryWriter: A Multi-Agent Framework for Long Story Generation." pith.science (2026). https://pith.science/paper/7NA5EUBE

@misc{pith2026250616445,
  author       = {Pith},
  title        = {Pith review of: StoryWriter: A Multi-Agent Framework for Long Story Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7NA5EUBE}},
  note         = {Machine review of arXiv:2506.16445}
}
abstract

Long story generation remains a challenge for existing large language models (LLMs), primarily due to two main factors: (1) discourse coherence, which requires plot consistency, logical coherence, and completeness in the long-form generation, and (2) narrative complexity, which requires an interwoven and engaging narrative. To address these challenges, we propose StoryWriter, a multi-agent story generation framework, which consists of three main modules: (1) outline agent, which generates event-based outlines containing rich event plots, character, and event-event relationships. (2) planning agent, which further details events and plans which events should be written in each chapter to maintain an interwoven and engaging story. (3) writing agent, which dynamically compresses the story history based on the current event to generate and reflect new plots, ensuring the coherence of the generated story. We conduct both human and automated evaluation, and StoryWriter significantly outperforms existing story generation baselines in both story quality and length. Furthermore, we use StoryWriter to generate a dataset, which contains about $6,000$ high-quality long stories, with an average length of $8,000$ words. We train the model Llama3.1-8B and GLM4-9B using supervised fine-tuning on LongStory and develop StoryWriter_GLM and StoryWriter_GLM, which demonstrates advanced performance in long story generation.

Figures

Figures reproduced from arXiv: 2506.16445 by the authors.

Figure 1
Figure 1. Results on MoPS (Ma et al., 2024) with dif￾ferent required story lengths. Details are placed in § 4. using a premise, often a brief beginning or theme, as input to create a complete narrative (Alhussain and Azmi, 2021). Since the emergence of large language models (LLMs; Ouyang et al., 2022), the quality of generated stories using LLMs has steadily improved (Xie and Riedl, 2024). However, generating long stories, pa… view at source ↗
Figure 2
Figure 2. An overview of the three-stage story generation framework. The process consists of (from left to right): (1) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Results for different window lengths. The [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 17 canonical work pages

  1. [1]

    Arwa I Alhussain and Aqil M Azmi. 2021. Automatic story generation: A survey of approaches. ACM Computing Surveys (CSUR), 54(5):1--38

    2021

  2. [2]

    Fatma Alkaaf and Ali Al-Bulushi. 2017. Tell and write, the effect of storytelling strategy for developing story writing skills among grade seven learners. Open Journal of Modern Linguistics, 7(2):119--141

    work page 2017

  3. [3]

    Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. 2024 a . Longalign: A recipe for long context alignment of large language models. arXiv preprint arXiv:2401.18058

    arXiv 2024

  4. [4]

    Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024 b . https://arxiv.org/abs/2408.07055 Longwriter: Unleashing 10,000+ word generation from long context llms . Preprint, arXiv:2408.07055

    arXiv 2024

  5. [5]

    Suchanek, and Chlo \'e Clavel

    Cyril Chhun, Pierre Colombo, Fabian M. Suchanek, and Chlo \'e Clavel. 2022. https://aclanthology.org/2022.coling-1.509/ Of human criteria and automatic metrics: A benchmark of the evaluation of story generation . In Proceedings of the 29th International Conference on Computational Linguistics, pages 5794--5836, Gyeongju, Republic of Korea. International C...

    work page 2022

  6. [6]

    Suchanek, and Chloé Clavel

    Cyril Chhun, Fabian M. Suchanek, and Chloé Clavel. 2024. https://arxiv.org/abs/2405.13769 Do language models enjoy their own stories? prompting large language models for automatic story evaluation . Preprint, arXiv:2405.13769

    arXiv 2024

  7. [7]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  8. [8]

    Angela Fan, Mike Lewis, and Yann Dauphin. 2019. https://doi.org/10.18653/v1/P19-1254 Strategies for structuring story generation . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2650--2660, Florence, Italy. Association for Computational Linguistics

    doi:10.18653/v1/p19-1254 2019

Show all 35 references
  1. [9]

    Gérard Genette. 1972. Narrative Discourse: An Essay in Method. Cornell University Press, Ithaca, NY. Translated by Jane E. Lewin

    1972

  2. [10]

    Gérard Genette. 1980. Narrative Discourse: An Essay in Method. Cornell University Press, Ithaca

    1980

  3. [11]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594

    2024 arXiv

  4. [12]

    David Herman. 2002. Story Logic: Problems and Possibilities of Narrative. University of Wisconsin Press, Madison, WI

    2002

  5. [13]

    David Herman. 2017. https://doi.org/10.12677/WLS.2017.53013 Narratology's union with cognitive science—a review of david herman's narrative theory and the cognitive science . World Literature Studies, 5(3):13--24

    2017

  6. [14]

    Fantine Huot, Reinald Kim Amplayo, Jennimaria Palomaki, Alice Shoshana Jakobovits, Elizabeth Clark, and Mirella Lapata. 2024. https://arxiv.org/abs/2410.02603 Agents' room: Narrative generation through multi-step collaboration . Preprint, arXiv:2410.02603

    2024 arXiv

  7. [15]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024 a . Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157--173

    2024

  8. [16]

    Xiang Liu, Peijie Dong, Xuming Hu, and Xiaowen Chu. 2024 b . Longgenbench: Long-context generation benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 865--883

    2024

  9. [17]

    Yan Ma, Yu Qiao, and Pengfei Liu. 2024. https://arxiv.org/abs/2406.05690 Mops: Modular story premise synthesis for open-ended automatic story generation . Preprint, arXiv:2406.05690

    2024 arXiv

  10. [18]

    Aleksandr Migal, Daria Seredina, Ludmila Telnina, Nikita Nazarov, Anastasia Kolmogorova, and Nikolay Mikhaylovskiy. 2024. Overview of long story generation challenge (lsgc) at inlg 2024. In Proceedings of the 17th International Natural Language Generation Conference: Generatio...

    2024

  11. [19]

    John W Oller Jr. 1983. Story writing principles and esl teaching. Tesol Quarterly, 17(1):39--53

    1983

  12. [20]

    OpenAI. 2024 a . https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/ Gpt-4o mini: Advancing cost-efficient intelligence . Accessed: 2025-02-04

    2024

  13. [21]

    OpenAI. 2024 b . https://openai.com/index/hello-gpt-4o/ Hello gpt-4o . Accessed: 2025-02-04

    2024

  14. [22]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...

    2022

  15. [23]

    Yufei Tian, Tenghao Huang, Miri Liu, Derek Jiang, Alexander Spangher, Muhao Chen, Jonathan May, and Nanyun Peng. 2024. Are large language models capable of generating human-level narratives? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Process...

    2024

  16. [24]

    Qianyue Wang, Jinwu Hu, Zhengping Li, Yufeng Wang, Yu Hu, Mingkui Tan, et al. 2024. Generating long-form story using dynamic hierarchical outlining with memory-enhancement. arXiv preprint arXiv:2412.13575

    2024 arXiv

  17. [25]

    Xiaozhi Wang, Yulin Chen, Ning Ding, Hao Peng, Zimu Wang, Yankai Lin, Xu Han, Lei Hou, Juanzi Li, Zhiyuan Liu, et al. 2022. Maven-ere: A unified large-scale dataset for event coreference, temporal, causal, and subevent relation extraction. In Proceedings of the 2022 Conference...

    2022

  18. [26]

    Xiaozhi Wang, Hao Peng, Yong Guan, Kaisheng Zeng, Jianhui Chen, Lei Hou, Xu Han, Yankai Lin, Zhiyuan Liu, Ruobing Xie, et al. 2023 a . Maven-arg: Completing the puzzle of all-in-one event understanding dataset with event argument annotation. arXiv preprint arXiv:2311.09105

    2023 arXiv

  19. [27]

    Yichen Wang, Kevin Yang, Xiaoming Liu, and Dan Klein. 2023 b . Improving pacing in long-form story planning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10788--10845

    2023

  20. [28]

    Yuxin Wang, Jieru Lin, Zhiwei Yu, Wei Hu, and B \"o rje F Karlsson. 2023 c . Open-world story generation with structured knowledge enhancement: A comprehensive survey. Neurocomputing, page 126792

    2023

  21. [29]

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. 2023. https://arxiv.org/abs/2308.08155 Autogen: Enabling next-gen llm applications via mu...

    2023 arXiv

  22. [30]

    Kaige Xie and Mark Riedl. 2024. Creating suspenseful stories: Iterative planning with large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2391--2407

    2024

  23. [31]

    Kevin Yang, Dan Klein, Nanyun Peng, and Yuandong Tian. 2023 a . Doc: Improving long story coherence with detailed outline control. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3378--3465

    2023

  24. [32]

    Kevin Yang, Dan Klein, Nanyun Peng, and Yuandong Tian. 2023 b . https://arxiv.org/abs/2212.10077 Doc: Improving long story coherence with detailed outline control . Preprint, arXiv:2212.10077

    2023 arXiv

  25. [33]

    Yao Yao, Zuchao Li, and Hai Zhao. 2024. https://arxiv.org/abs/2405.12528 Sirllm: Streaming infinite retentive llm . Preprint, arXiv:2405.12528

    2024 arXiv

  26. [34]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  27. [35]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.