{"id":"f3d349aa-da00-4012-a25d-e64fbad7b01d","arxiv_id":"2506.16445","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A multi-agent framework for long story generation, a 5,500-story dataset, and fine-tuned 8B and 9B models that improve length adherence and story quality over baselines.","lead":"StoryWriter is a multi-agent system that writes long, coherent stories by first planning an event graph, deciding which events appear in each chapter out of order, and compressing earlier chapters while writing. The authors use it to create a dataset of 5,500 stories and train open 8B and 9B models that follow length instructions better than GPT-4o on long-form prompts.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Correctly parsed, Table 3 shows the fine-tuned models do not surpass GPT-4o on quality or combined score at 2k-4k words; the 'surpasses GPT-4o' claim rests on length-adherence scores, and the printed formula for the weighted score is inconsistent with the reported Overall values.","rationale":"The reader's weakest_assumption was that the evaluation pipeline (small, unspecified human panel; single GPT-4o judge; no significance testing) is too fragile to support 'significantly outperforms.' That concern is valid and I partially agree. But the more load-bearing problem is internal: even taking the reported numbers at face value, the paper's own Table 3 contradicts the strongest quantitative claim in Section 1. The parse I propose is not an assumption about external truth; it is forced by the arithmetic, since the reported 'Overall' values can only be reproduced by reading each bucket as (Sl, Sq) and using Sbar = (20*Sq + Sl)/2. On that reading, the fine-tuned models never beat GPT-4o on quality below 10k words and lose on the combined score in the exact bucket (2k-4k) named by the claim. This matters because the Section 1 and abstract claims are what a typical reader takes away; if 'surpasses GPT-4o' means 'higher length-adherence score plus a length-weighted composite,' that should be said plainly, and the quality comparisons should be reported separately. The framework contribution (multi-agent event outline, NLN planning, ReIO compression) is coherent and the ablations in Table 2 show each module contributes, though Table 2 itself contains a labeling inconsistency: its STORYWRITER row matches Table 1's auto-eval row while the caption implies human evaluation. The dataset and released models are potentially useful, but the headline claims need correction and additional length-controlled, significance-tested evaluation before they should be accepted. Since the reader already assigned CONDITIONAL, my reading does not change the verdict label; it adds a more specific condition: correct the Table 3 parse and formula, and revise or qualify the 'surpasses GPT-4o' claim.","tokens_in":10991,"tokens_out":24393,"duration_ms":185838,"concrete_test":"Re-parse Table 3 with the column order (Sl, Sq) per bucket, verifying that Sbar = (20*Sq + Sl)/2 reproduces each printed 'Overall' value; then recompute Sq and Sbar for STORYWRITER_LLAMA, STORYWRITER_GLM, and GPT-4o in the [2k,4k) and [4k,10k) buckets. If GPT-4o's Sq (4.5/4.3) exceeds both fine-tuned models' Sq (3.8/3.5 and 4.1/4.0), the Section 1 'surpasses GPT-4o' claim fails for quality. Then run a length-controlled comparison on MoPS test premises: restrict outputs to a matched length band (e.g., 2,000-3,000 words), collect human and GPT-4o six-dimension scores on paired outputs, and apply a paired significance test (e.g., Wilcoxon signed-rank) with a pre-registered minimum number of raters and inter-annotator agreement, to determine whether any quality advantage survives once length and judge variance are controlled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim that STORYWRITER_LLAMA and STORYWRITER_GLM 'surpass GPT-4o on stories exceeding 2,000 words' (Section 1) is not supported by the paper's own Table 3 once the table is parsed correctly. Each length bucket in Table 3 must be read as (Sl, Sq), length score on 0-100 followed by quality on 1-5, because this is the only ordering that reproduces the reported 'Overall' values through the weighted formula. The stated formula in Section 4, Sbar = (Sq + 20*Sl)/2, cannot reproduce any Overall value; the data only fit Sbar = (20*Sq + Sl)/2, so the printed formula is wrong. With the corrected parse, GPT-4o has higher Sq than STORYWRITER_GLM in every bucket below 10,000 words (e.g., 4.5 vs 4.1 at [2k,4k); 4.3 vs 4.0 at [4k,10k)), and GPT-4o's combined Sbar (90.85) also exceeds STORYWRITER_GLM's (90.65) in the [2k,4k) bucket. The fine-tuned models only overtake GPT-4o on the combined score at 4k+ lengths, and that advantage is driven almost entirely by the length-adherence component Sl, not by narrative quality. Thus the abstract's 'significantly outperforms in both story quality and length' is not supported for the quality dimension in the fine-tuned comparison. The same length confound threatens the framework-level claim in Table 1: STORYWRITER stories average 8,081 words versus 1,078-3,134 for baselines, with no length-controlled evaluation, no significance tests, and no inter-annotator agreement reported. Corroborating internal inconsistencies: the dataset average length is stated as 15,000 words in Section 1 but 8,000 in the Abstract and Section 4, with counts given as 6,000 and 5,500.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents STORYWRITER, a multi-agent framework for long-form story generation composed of outline, planning, and writing agents. It also introduces LONGSTORY, a dataset of long stories generated by the framework, and fine-tuned models STORYWRITER_LLAMA and STORYWRITER_GLM. The authors evaluate against DOC, Agents' Room, and GPT-4o-mini, and report human and GPT-4o-based automatic scores across six narrative dimensions, as well as length-adherence scores for the fine-tuned models.","tokens_in":11408,"tokens_out":8402,"duration_ms":62827,"significance":"If the central claims were fully supported, the work would be a useful contribution: it provides a complete multi-agent recipe, a public dataset, and two fine-tuned models with code and data released. The design ideas—event-graph outlining, non-linear narration, and input/output rewriting for long-context generation—are sensible and potentially reusable. However, the evaluation evidence is underpowered and partly inconsistent, and the headline claim about surpassing GPT-4o is not supported by the paper's own table once the metrics are parsed correctly.","major_comments":[{"comment":"The printed formula for the combined score, Sbar = (Sq + 20*Sl)/2, cannot reproduce any of the reported Overall values, because Sl is on a 0-100 scale and Sq on a 1-5 scale; the table's column order (Sl then Sq) and the plausible values are consistent with Sbar = (20*Sq + Sl)/2. This formula error must be corrected, and it changes how the Overall column is interpreted.","section":"Section 4, Table 3"},{"comment":"The claim in Section 1 that the fine-tuned models 'even surpass GPT-4o' on stories exceeding 2,000 words is not supported on the quality dimension: in the [2k,4k) bucket GPT-4o's Sq is 4.5 versus 4.1 for STORYWRITER_GLM, and GPT-4o's combined score 90.85 also exceeds STORYWRITER_GLM's 90.65. The only buckets where STORYWRITER_GLM leads the combined score are [4k,10k) and [10k,20k), and that lead comes almost entirely from the length-adherence component Sl, not from the quality score Sq. The abstract's 'significantly outperforms ... in both story quality and length' is therefore not substantiated for the fine-tuned comparison.","section":"Section 4, Table 3"},{"comment":"The dataset statistics are internally inconsistent: the Abstract states about 6,000 stories with an average length of 8,000 words; the Introduction states about 6,000 stories with an average of 15,000 words; and Section 4 states a final curated set of 5,500 stories with an average of about 8,000 words. The authors must reconcile these numbers and state the exact count and average length used for training.","section":"Section 4 and Abstract"},{"comment":"The report 'significantly outperforms' is not backed by any statistical test: no number of human evaluators is given, no inter-annotator agreement is reported, and no error bars or significance tests accompany the average scores. Because STORYWRITER's stories are much longer (average 8,081 words) than the baselines (1,078-3,134 words), the framework-level quality comparison is confounded with output length, and the claimed 'significantly outperforms' is not established.","section":"Section 3.1, Table 1"}],"minor_comments":[{"comment":"The Abstract contains a typo: 'STORYWRITER_GLM and STORYWRITER_GLM' should presumably be 'STORYWRITER_LLAMA and STORYWRITER_GLM'.","section":"Abstract"},{"comment":"Figure 3 is referenced in Section 3.3 but is not present in the submitted text; the authors should ensure it is included with clear axis labels and a caption that explains the window configurations.","section":"Section 3.3"},{"comment":"The paper states that the training does not involve explicit ability enhancement for following length constraints, which appears to contradict the observation that trained models adhere better to length constraints; consider discussing a possible mechanism for this effect.","section":"Section 4"},{"comment":"The paper says 'we adopt the same evaluation dataset MoPS in § 3.1' but then applies the LongBench-Write length-constrained setting; please clarify which exact test instances and prompts are used for each bucket.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of cs.CL and the framework is plausible, but the evaluation section needs substantial revision. I recommend major revision. Please also check that the authors do not overstate the 'surpass GPT-4o' claim in the final version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a workmanlike systems paper. The three-agent pipeline — event-graph outline, non-linear narration planning, and ReIO history compression — is put together sensibly, and they ship a 5,500-story dataset plus fine-tuned 8B/9B models. Those are real artifacts. But the evaluation is not tight enough for the claims. In particular, the abstract's 'significantly outperforms in both story quality and length' is only half supported. For the framework comparison (Table 1), STORYWRITER does get higher human and auto scores than DOC, Agents' Room, and GPT-4o-mini, but it also generates 8,000+ words versus 1,000–3,000 for the baselines. That length gap is a confound: longer stories might mechanically score higher on 'complexity' or 'creativity,' and coherence is known to drop with length. No significance tests, no error bars, no inter-annotator agreement are reported. The human evaluator pool is 'graduate students in an English program' with TOEFL 108+, but the count isn't given.\n\nThe bigger problem is the fine-tuning claim. Section 1 says STORYWRITER_LLAMA and STORYWRITER_GLM 'surpass GPT-4o on stories exceeding 2,000 words.' If you actually parse Table 3, the combined score formula printed in the paper, \\bar{S} = (S_q + 20*S_l)/2, is wrong: no row reproduces its numbers. The data only fit \\bar{S} = (20*S_q + S_l)/2. With that corrected parse, GPT-4o beats STORYWRITER_GLM on quality (S_q) in every bucket below 10,000 words — e.g., 4.5 vs 4.1 at [2k,4k), 4.3 vs 4.0 at [4k,10k) — and GPT-4o's combined score in the [2k,4k) bucket is 90.85 vs 90.65. The fine-tuned models only overtake GPT-4o at 4k+ words, and that advantage is almost entirely from length adherence (S_l), not narrative quality. So the 'surpasses GPT-4o' statement is true only if you weight length adherence enough, which the paper's own formula doesn't even do correctly.\n\nThere are also internal inconsistencies: the dataset is described as 6,000 stories in the abstract and 5,500 in Section 4; average length is 15,000 words in the introduction but 8,000 in the abstract and Section 4. These should have been caught before submission.\n\nWhat's genuinely useful: LONGSTORY is a new resource, and the two SFT models do show better length adherence than their backbones. The ReIO sliding-window idea is plausible and the ablation in Table 2 shows each component matters, though again without significance testing. The paper deserves a serious referee — it should go to review, not be desk-rejected — but the authors need to fix the formula, reconcile the numbers, add significance tests and inter-annotator agreement, and add a length-controlled evaluation before the strong claims can stand.\n\nMy recommendation: engage with it as a systems+dataset paper, but treat the quality claims with a grain of salt until the evaluation is fixed.\n\nRegards","headline":"Useful framework and dataset, but the evaluation doesn't prove the quality claims and the GPT-4o comparison is really about length adherence.","tokens_in":12015,"tokens_out":6495,"would_cite":true,"duration_ms":49000,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned StoryWriter models outscore GPT-4o on long stories","keywords":["long story generation","multi-agent framework","event graph","non-linear narration","history compression","supervised fine-tuning","narrative evaluation","long-context generation"],"falsifier":"A counter-check would be a blind, pre-registered human study matching stories by length (or asking judges to rate truncated excerpts) and using at least a dozen independent raters with reported inter-annotator agreement: if StoryWriter's advantage over its baselines shrinks to noise under length control, the claimed superiority is an artifact of longer output.","tokens_in":10784,"feed_emoji":"✍️","tokens_out":7669,"duration_ms":62338,"temperature":0.7,"pith_summary":"This paper claims that long story generation can be decomposed into three cooperating agents—an outline agent, a planning agent, and a writing agent—and that this division fixes the coherence and complexity failures of single-pass LLM generation. It reports that StoryWriter, built on GPT-4o-mini, outperforms DOC, Agents' Room, and GPT-4o-mini on both human and automated narrative-quality scores while producing stories averaging about 8,000 words. It also builds LongStory, a cleaned dataset of about 5,500 generated long stories, and shows that fine-tuning Llama3.1-8B and GLM4-9B on it yields models that beat GPT-4o on stories exceeding 2,000 words. A sympathetic reader would care because it points to a data-centric path for improving long-form narrative quality without requiring larger or longer-context models.","feed_headline":"Fine-tuned StoryWriter models outscore GPT-4o on long stories","feed_subtitle":"Planning agents that compress history and shuffle sub-events yield longer, higher-rated stories.","key_machinery":"The central mechanism is the three-agent architecture acting on an event graph. The outline agent builds a sequence of event tuples with characters and event-event relations; the planning agent decomposes events into sub-events and shuffles them across chapters via non-linear narration, preserving causal links; the writing agent uses ReIO (re-write input and output), where a Coordinator compresses the story history around the current event and rewrites the FinalWriter's output to keep it aligned with the outline. The event graph carries the global plot structure, and ReIO keeps the effective context short enough to avoid the attention degradation that the paper says appears beyond roughly 10,000 characters.","core_discovery":"StoryWriter's central claim is that a multi-agent pipeline with an event-based outline, non-linear chapter planning, and dynamic history compression generates longer and better-rated stories than strong baselines. The paper presents human and GPT-4o-based evaluations across six dimensions—relevance, coherence, empathy, surprise, creativity, and complexity—reporting the best average scores and an average length of 8,081 words for StoryWriter. It further claims that supervised fine-tuning on the generated LongStory dataset transfers these strengths to smaller open models, with StoryWriter_GLM surpassing GPT-4o on stories longer than 2,000 words in both quality and length adherence.","pith_inferences":["The same three-phase decomposition (outline, plan, write) could plausibly transfer to other long-form creative generation, such as screenplays or serialized non-fiction, where out-of-order presentation and global planning over long horizons are also central.","The improvement in length adherence after SFT, despite no explicit length training, suggests that simply training on long, structured outputs teaches length control; a controlled study that varies only the average training length could isolate this mechanism.","Because the evaluation used a small number of graduate raters and a single GPT-4o judge, a pre-registered human study with length-matched excerpts and inter-annotator agreement statistics would test whether the quality advantage is robust or partly an artifact of longer outputs."],"forward_implications":["Structured event-based outlines, not raw long-context modeling, can carry the plot across thousands of words; removing the event outline collapses every evaluation score in the paper's ablation.","Decomposing events into sub-events and presenting them out of chronological order across chapters increases rated complexity and diversity while preserving coherence.","Compressing the writing history around the current event—rather than feeding the full history—avoids the attention degradation that appears beyond roughly 10,000 input characters.","Supervised fine-tuning on a generated long-story dataset improves both story quality and adherence to length constraints, even without explicit length-constraint training."],"supporting_citations":[{"why":"Provides the MoPS dataset of story premises used for evaluation and for generating LongStory.","marker":"Ma et al., 2024"},{"why":"DOC baseline, the detailed-outline method StoryWriter must outperform.","marker":"Yang et al., 2023b"},{"why":"Agents' Room multi-agent baseline used as a comparison point.","marker":"Huot et al., 2024"},{"why":"Supplies HANNA's six evaluation dimensions used in both human and automated scoring.","marker":"Chhun et al., 2022"},{"why":"Defines the LongBench-Write length score and reports long-context baselines that StoryWriter_GLM surpasses.","marker":"Bai et al., 2024b"},{"why":"GPT-4o acts as automated judge and as the closed-source baseline outperformed on stories exceeding 2,000 words.","marker":"OpenAI, 2024b"},{"why":"AutoGen framework in which StoryWriter's agents are implemented.","marker":"Wu et al., 2023"},{"why":"Llama 3.1 base model for the StoryWriter_LLAMA fine-tune.","marker":"Dubey et al., 2024"}],"fun_headline_variants":["Multi-agent StoryWriter beats GPT-4o at long tales","Agent pipeline writes longer, better-scored stories than GPT-4o","StoryWriter's agents craft 8k-word stories that outrank GPT-4o","Agent orchestration yields 8k-word stories that beat GPT-4o","Multi-agent StoryWriter outwrites GPT-4o with 8k-word stories"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation pipeline captures narrative quality independently of output length and generator identity, with a small number of graduate students and a single GPT-4o judge producing scores reliable enough to support the 'significantly outperforms' claim without significance tests, error bars, or inter-annotator agreement.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent StoryWriter beats GPT-4o at long tales","Agent pipeline writes longer, better-scored stories than GPT-4o","StoryWriter's agents craft 8k-word stories that outrank GPT-4o","Agent orchestration yields 8k-word stories that beat GPT-4o","Multi-agent StoryWriter outwrites GPT-4o with 8k-word stories"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000548,"raw_usage":{"total_tokens":2605,"prompt_tokens":922,"completion_tokens":1683,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":1582}},"tokens_in":538,"tokens_out":1683,"duration_ms":11746,"temperature":1.0,"reasoning_tokens":1582,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:25:32.959812+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A counter-check would be a blind, pre-registered human study matching stories by length (or asking judges to rate truncated excerpts) and using at least a dozen independent raters with reported inter-annotator agreement: if StoryWriter's advantage over its baselines shrinks to noise under length control, the claimed superiority is an artifact of longer output.","supporting_citations":[{"cited_title":"Suchanek, and Chlo \\'e Clavel","cited_arxiv_id":null,"evidence_quote":"Supplies HANNA's six evaluation dimensions used in both human and automated scoring."}],"review_version":1}