REVIEW 4 major objections 5 minor 28 references
BookWorld: From Novels to Interactive Agent Societies for Creative Story Generation
T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read BookWorld's book-based agent societies beat direct generation and HoLLMwood in 75.36% of story comparisons.
desk verdict System is worth reading, but the 75.36% win rate is undefined and Table 1 contradicts the 'consistently outperforms' claim; send to review with major revision demands. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a two-tier agent architecture with role agents and a world agent, organized through scenes as the minimal narrative unit. Role agents hold static traits, dynamic goals and states, short-term memory, and retrieval-augmented long-term memory drawn from the original text; the world agent owns a discrete weighted graph map, location profiles, worldview settings, and environment-response generation. A scene is a bounded set of rounds in which an initiator chooses an action, targets reply, and the world agent reacts to environmental actions; travel consumes time units, so geographical constraints shape the narrative. The final LLM rephrasing pass turns structured action logs into novel-style prose.
What would settle it
Show the same story pairs to a panel of human readers blind to which system wrote them; if the panel's majority winner disagrees with the AI judge on more than about a quarter of the pairs, or if equalizing story length and formatting reverses the judge's preferences, then the 75.36% claim is falsified.
Extended reading notes
Core claim
BookWorld's central claim is that a carefully extracted set of character profiles, worldview entries, map constraints, and scene-structured interactions is enough to let LLM-based agents re-animate a fictional world. The system does not write stories top-down; it lets characters act, react, remember, travel, and respond to environment feedback, and only then converts the resulting simulation log into narrative prose. On five quality axes—anthropomorphism, character fidelity, immersion and setting, writing quality, and storyline quality (or creativity in free mode)—BookWorld is reported to outperform direct generation on every metric across all evaluated base models, and to outperform HoLLMwood specifically on immersion and character fidelity. The authors also report that the scene module, environment responses, and worldview settings each contribute to the quality gain, with environment feedback most affecting immersion and the scene module most affecting storyline adherence.
Load-bearing premise
The load-bearing premise is that the AI judge used for pairwise story comparison is actually measuring story quality; the paper validates that judge against a small set of human readers with only moderate agreement, so if the judge has hidden stylistic preferences, the 75.36% win rate and every comparative conclusion collapse.
Editorial extensions
If this is right
- Any sufficiently text-rich novel can be converted into an interactive society: the extraction pipeline produced 9,912 worldview settings from 16 novels and 453 character/act presets without manual worldbuilding.
- User-supplied scripts can steer the simulation along a chosen outline while agents still improvise within it, opening a direct route to interactive games and controllable fan-fiction generation.
- The approach is model-dependent: with Llama-3.3-70B, BookWorld loses ground to baselines on some dimensions, so the benefit of simulation grows with the base model's instruction-following ability.
- Because simulation histories are stored as retrievable memories, the same agents can be reused across separate story runs, making long-running serialized or interactive narratives feasible.
- Removing any of the three structural modules—scene organization, environment feedback, or extracted worldview settings—degrades at least one quality dimension, so the architecture's gains are compositional rather than coming from a single prompt trick.
Reading between the lines
- Editorial inference: the character-fidelity advantage likely owes more to retrieval of original dialogue snippets than to the long-term memory summarizer; an ablation that keeps verbatim excerpt retrieval while removing memory would isolate the real source.
- Editorial inference: the LLM judge may be rewarding output that looks more like published prose, since BookWorld's rephrasing pass produces polished narrative text while direct generation often yields plainer responses; a length- and format-controlled comparison is needed to confirm the 75.36% margin is about story substance.
- Editorial inference: the scene-based time model and map constraints already resemble episode structure, so the architecture should transfer to screenplays, game lore, and serialized web fiction; the paper's own Werewolf limitation suggests social-deduction rules, not narrative genres, are the hard boundary.
- Editorial inference: script mode could turn BookWorld into a human-AI co-writing tool, but the paper does not measure whether user interventions improve or harm the final story, making that a natural next experiment.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces BookWorld, a multi-agent simulation system for story generation grounded in existing novels. It extracts character and worldview data from source texts, initializes role agents and a world agent, simulates scenes with spatial, environmental, and world-knowledge constraints, and rephrases simulation logs into novel-style stories. The authors claim that BookWorld outperforms direct generation and HoLLMwood baselines, reporting a win rate of 75.36% based on pairwise LLM-as-judge comparisons across five dimensions and seven base models, with a human-agreement validation reported in Appendix A.
Significance. If substantiated, the work would be a useful contribution to creative story generation and character-driven simulation: grounding agent societies in established fictional worldviews rather than synthetic personas is a timely and underexplored direction. The system design is plausible and reasonably complete, covering data extraction, a world agent, map-based movement, script/free modes, and retrieval-augmented memory, and the authors state that code is released. However, the significance is currently limited by evaluation weaknesses: the headline 75.36% figure is not defined, the 'consistently outperforms' claim is contradicted by the paper's own Table 1, and the evaluation protocol omits sample sizes, confidence intervals, and significance tests, relying on a single LLM judge with moderate human agreement.
major comments (4)
- [Section 4.3 and Table 1] The statement that 'BookWorld consistently outperforms the direct generation baseline across all evaluated metrics and models' is contradicted by Table 1. For Llama-3.3-70B without script, win rates versus direct generation are 34.6% (An), 42.3% (CF), 53.8% (IS), 34.6% (WQ), and 47.8% (Cr); with script, SQ is 42.3%. The abstract's 75.36% win rate is undefined: the paper never states which cells of Table 1 are aggregated, whether the baseline is direct generation or HoLLMwood, whether the with-script and without-script conditions are pooled, or how the average is weighted across the seven models. These claims need to be redefined and revised to match the data actually reported.
- [Appendix A and Section 4.1] The reliability of the LLM judge is load-bearing for all comparative conclusions. The paper does not report the number of comparison pairs per cell, the total number of stories generated, confidence intervals, or any significance tests. Appendix A reports Cohen's kappa values between 0.637 and 0.786, which is moderate-to-good agreement rather than the 'high level of consistency' claimed, and the sampled number of pairs Y is left unspecified. Please report exact N per condition, per-cell confidence intervals or exact binomial tests, and the value of Y; this is especially important for the win rates near 50%, which are currently indistinguishable from chance.
- [Section 3.4 versus Appendix B] The count of worldview settings is internally inconsistent. Section 3.4 states 9912 settings, with 44.8% Chinese and 55.2% English. Appendix B states 9,142 entries, with 4,449 from Chinese novels and 4,693 from English novels, and Table 5 sums to 9,142. The percentages in Section 3.4 do not match either total. These numbers must be reconciled.
- [Limitations] The Limitations section does not mention the fragility of the evaluation: the central comparative claims rest entirely on pairwise comparisons by a single LLM judge, with human validation only on an unspecified number of pairs and with only moderate-to-good agreement. The paper should explicitly acknowledge this limitation and discuss the implications for the low win-rate cells, such as those for Llama-3.3-70B, where the differences may not be statistically distinguishable from chance.
minor comments (5)
- [Abstract and Section 1] 'e.t.c.' should be written as 'etc.'.
- [Table 8] The environment-response prompt misspells 'Environment' as 'Enviroment' in two places.
- [Section 4.2] The sentence 'Each experiment simulates 2 to 4 scenes, averaging a total of 4,230 words' is ambiguous; please specify whether this is the average length per generated story and state the number of stories used in the evaluation.
- [Table 5] The row 'A Song of Ice and Fire (Part)' is marked as Chinese (zh), but the work is originally English; please clarify whether the extracted source was a Chinese translation and, if so, state this explicitly.
- [Section 4.1] The definition of the Anthropomorphism metric as 'attributing human characteristics to non-human entities while preserving their original nature' is unusual for story generation; please clarify what evaluators were asked to judge and provide the exact evaluation prompt.
Circularity Check
No circular derivation: BookWorld is an empirical systems paper, with only a minor non-load-bearing self-citation and evaluation-reliability caveats, not a circular argument.
full rationale
The paper is an empirical systems paper, not a derivation: Section 3 presents a pipeline (data extraction, initialization, simulation, rephrasing) and Section 4 reports pairwise LLM-judged win rates against direct generation and HoLLMwood. There are no equations in which an output is constructed from its own target, and the claimed 75.36% win rate is a summary statistic over Table 1, not a parameter fitted to the evaluation data. The only self-citation is in Section 3.4: 'We provide an automatic extraction method based on incremental updating inspired by Yuan et al. (2024).' Yuan et al. (2024) shares two co-authors with this paper, but the citation is used only as methodological inspiration for a preprocessing component, not as the load-bearing justification for the central claim of story-quality superiority, and the evaluation is against external baselines. Appendix A validates the gpt-4o judge against five human annotators with Cohen's kappa 0.637-0.786; the unspecified sample size Y and the Section 4.3 claim that BookWorld 'consistently outperforms the direct generation baseline across all evaluated metrics and models' contradicted by Table 1's Llama-3.3-70B row are empirical validity problems, not circularity. No circular step is identified.
Assumptions & free parameters
free parameters (4)
- Judge model =
gpt-4o-2024-08-06
- Number of dialogue turns =
not specified
- Evaluation sample size =
not reported
- Scenes per simulation =
2 to 4
assumptions (4)
- domain assumption LLMs can faithfully role-play established characters when given a profile and retrieved excerpts.
- domain assumption The automatic extraction method captures sufficient character and worldview information from source books.
- domain assumption Pairwise LLM judgment is a reliable proxy for story quality, as calibrated by human agreement.
- domain assumption The discrete map and scene-based structure adequately represent the fictional world's geography and narrative cohesion.
Cite this review
Pith. "Pith review of BookWorld: From Novels to Interactive Agent Societies for Creative Story Generation." pith.science (2026). https://pith.science/paper/4475O64W
@misc{pith2026250414538,
author = {Pith},
title = {Pith review of: BookWorld: From Novels to Interactive Agent Societies for Creative Story Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/4475O64W}},
note = {Machine review of arXiv:2504.14538}
}
read the original abstract
Recent advances in large language models (LLMs) have enabled social simulation through multi-agent systems. Prior efforts focus on agent societies created from scratch, assigning agents with newly defined personas. However, simulating established fictional worlds and characters remain largely underexplored, despite its significant practical value. In this paper, we introduce BookWorld, a comprehensive system for constructing and simulating book-based multi-agent societies. BookWorld's design covers comprehensive real-world intricacies, including diverse and dynamic characters, fictional worldviews, geographical constraints and changes, e.t.c. BookWorld enables diverse applications including story generation, interactive games and social simulation, offering novel ways to extend and explore beloved fictional works. Through extensive experiments, we demonstrate that BookWorld generates creative, high-quality stories while maintaining fidelity to the source books, surpassing previous methods with a win rate of 75.36%. The code of this paper can be found at the project page: https://bookworld2025.github.io/.
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Altera AL, Andrew Ahn, Nic Becker, Stephanie Carroll, Nico Christie, Manuel Cortes, Arda Demirci, Melissa Du, Frankie Li, Shuying Luo, et al. 2024. Project sid: Many-agent simulations toward ai civilization. arXiv preprint arXiv:2411.00114
arXiv 2024
-
[4]
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201
arXiv 2023
-
[5]
Jing Chen, Xinyu Zhu, Cheng Yang, Chufan Shi, Yadong Xi, Yuxiang Zhang, Junjie Wang, Jiashu Pu, Rongsheng Zhang, Yujiu Yang, et al. 2024. Hollmwood: Unleashing the creativity of large language models in screenwriting via role playing. arXiv preprint arXiv:2406.11683
arXiv 2024
-
[6]
Yun-Shiuan Chuang, Nikunj Harlalka, Siddharth Suresh, Agam Goyal, Robert Hawkins, Sijia Yang, Dhavan Shah, Junjie Hu, and Timothy T Rogers. 2024. The wisdom of partisan crowds: Comparing collective intelligence in humans and llm-based agents. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 46
work page 2024
-
[7]
Gordon Dai, Weijia Zhang, Jinhan Li, Siqi Yang, Srihas Rao, Arthur Caetano, Misha Sra, et al. 2024. Artificial leviathan: Exploring social evolution of llm agents through the lens of hobbesian social contract theory. arXiv preprint arXiv:2406.14373
arXiv 2024
-
[8]
Seraphina Goldfarb-Tarrant, Tuhin Chakrabarty, Ralph Weischedel, and Nanyun Peng. 2020. Content planning for neural story generation with aristotelian rescoring. arXiv preprint arXiv:2009.09870
arXiv 2020
Show all 28 references
-
[9]
Carlos G \'o mez-Rodr \' guez and Paul Williams. 2023. A confederacy of models: A comprehensive evaluation of llms on creative writing. arXiv preprint arXiv:2310.08433
2023 arXiv
-
[10]
Senyu Han, Lu Chen, Li-Min Lin, Zhengshan Xu, and Kai Yu. 2024. Ibsen: Director-actor agent collaboration for controllable and interactive drama script generation. arXiv preprint arXiv:2407.01093
2024 arXiv
-
[11]
Dong Huang, Qingwen Bu, Jie M Zhang, Michael Luck, and Heming Cui. 2023. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13010
2023 arXiv
-
[12]
Mete Ismayilzada, Claire Stevenson, and Lonneke van der Plas. 2024. Evaluating creative short story generation in humans and large language models. arXiv preprint arXiv:2411.02316
2024 arXiv
-
[13]
Max Kreminski, Melanie Dickinson, and Michael Mateas. 2021. Winnow: a domain-specific language for incremental story sifting. In Proceedings of the AAAI conference on artificial intelligence and interactive digital entertainment, volume 17, pages 156--163
2021
-
[14]
Michael Lebowitz. 1984. Creating characters in a story-telling universe. Poetics, 13(3):171--194
1984
-
[15]
Cheng Li, Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang, Weishi Mi, Yaying Fei, Xiaoyang Feng, Song Yan, HaoSheng Wang, et al. 2023. Chatharuhi: Reviving anime character in reality via large language model. arXiv preprint arXiv:2308.09597
2023 arXiv
-
[16]
R. McKee. 1999. https://books.google.com.hk/books?id=nEadQgAACAAJ Story: Substance, Structure, Style, and the Principles of Screenwriting . Methuen film. Methuen
1999
-
[17]
James R Meehan. 1977. Tale-spin, an interactive program that writes stories. In Ijcai, volume 77, pages 91--98
1977
-
[18]
Piotr Mirowski, Kory W Mathewson, Jaylen Pittman, and Richard Evans. 2023. Co-writing screenplays and theatre scripts with language models: Evaluation by industry professionals. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1--34
2023
-
[19]
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1--22
2023
-
[20]
Yufei Tian, Tenghao Huang, Miri Liu, Derek Jiang, Alexander Spangher, Muhao Chen, Jonathan May, and Nanyun Peng. 2024. Are large language models capable of generating human-level narratives? arXiv preprint arXiv:2407.13248
2024 arXiv
-
[21]
Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. 2023 a . Avalon's game of thoughts: Battle against deception through recursive contemplation. arXiv preprint arXiv:2310.01320
2023 arXiv
-
[22]
Yuxin Wang, Jieru Lin, Zhiwei Yu, Wei Hu, and B \"o rje F Karlsson. 2023 b . Open-world story generation with structured knowledge enhancement: A comprehensive survey. Neurocomputing, page 126792
2023
-
[23]
Weiqi Wu, Hongqiu Wu, Lai Jiang, Xingyuan Liu, Jiale Hong, Hai Zhao, and Min Zhang. 2024. From role-play to drama-interaction: An llm solution. arXiv preprint arXiv:2405.14231
2024 arXiv
-
[24]
Ziyi Yang, Zaibin Zhang, Zirui Zheng, Yuxian Jiang, Ziyue Gan, Zhiyu Wang, Zijian Ling, Jinsong Chen, Martz Ma, Bowen Dong, Prateek Gupta, Shuyue Hu, Zhenfei Yin, Guohao Li, Xu Jia, Lijun Wang, Bernard Ghanem, Huchuan Lu, Chaochao Lu, Wanli Ouyang, Yu Qiao, Philip Torr, and Ji...
2024 arXiv
-
[25]
Lili Yao, Nanyun Peng, Ralph Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. 2019. Plan-and-write: Towards better automatic storytelling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7378--7385
2019
-
[26]
Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022. Wordcraft: story writing with large language models. In Proceedings of the 27th International Conference on Intelligent User Interfaces, pages 841--852
2022
-
[27]
Xinfeng Yuan, Siyu Yuan, Yuhan Cui, Tianhe Lin, Xintao Wang, Rui Xu, Jiangjie Chen, and Deqing Yang. 2024. Evaluating character understanding of large language models via character profiling from fictional works. arXiv preprint arXiv:2404.12726
2024 arXiv
-
[28]
Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, et al. 2023. Sotopia: Interactive evaluation for social intelligence in language agents. arXiv preprint arXiv:2310.11667
2023 arXiv
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.