{"id":"e4c14b9d-0937-49cd-b6e4-834bba1d62a7","arxiv_id":"2608.08160","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new 100-environment benchmark shows that state-of-the-art LLM narrators frequently contradict established story facts and fail to preserve plot commitments over long interactive sessions.","lead":"The paper introduces NCP-Bench, a benchmark of 100 movie-based interactive stories in which an automated auditor checks whether an AI narrator keeps story facts and plot commitments straight across many turns. It reports that even the strongest tested LLM survives only 42% of environments after 20 turns, showing that fluent text does not guarantee logical consistency.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline rates may be inflated by the audit loop's conservative fact-update and commitment-status checks, which are only weakly validated; a stale fact ledger can manufacture conflicts that the double-check then confirms.","rationale":"The paper is a serious benchmark effort with genuine strengths: it decouples the narrator from the auditor, includes a secondary conflict confirmation, tests three auditor backbones, compares adversarial vs. natural inputs, adds a memory-augmented baseline, and includes a human narrator pilot on one environment showing the task is solvable. These make the qualitative conclusion that current narrators are brittle reasonably plausible. My concern is narrower and quantitative: the headline rates come from a stateful audit loop whose fact-update and commitment-status components are themselves LLM outputs, and those components are validated only indirectly. The Fact Update prompt's explicit 'do not add / do not negate when uncertain' bias, combined with a Conflict Check that trusts the pre-turn ledger, can manufacture fact conflicts out of stale state; the Commitment Status Check's 'default to PENDING' rule can suppress genuine achievement satisfactions. The human verification in §5.4 covers terminal conflict decisions for one narrator model, not intermediate ledger states or satisfaction statuses, and no repeated runs are reported, so the point estimates in Table 1 have unknown variance. The reader's weakest assumption captures part of this (auditor reliability and false positives), but my formulation emphasizes the state-update mechanism and the undercount of satisfactions, which the reader did not explicitly flag. I therefore keep the CONDITIONAL verdict: the benchmark should be released with archived artifacts, repeated runs should be added, and the full audit state (ledger, trajectory, commitment statuses) should be validated on a stratified sample with human annotators. If the bias is small, the paper's claims stand; if it is large, the numeric headline must be revised even though the broad qualitative finding is likely to survive.","tokens_in":25201,"tokens_out":9719,"duration_ms":100365,"concrete_test":"Sample 20–30 runs stratified across narrator models (including GPT-5.2) and auditor backbones, and re-run the offline audit with two independent human annotators. For every turn, annotators (a) reconstruct the true fact ledger from the full transcript, (b) judge whether each confirmed conflict is a real logical contradiction given the complete history, and (c) judge each commitment marked PENDING as genuinely pending or actually satisfied. Compare the human-confirmed conflict rates and satisfaction counts with the pipeline's outputs. If the corrected fact-conflict rate is more than 10 percentage points lower for any model, or if any run scored as not satisfying all achievement commitments in fact did, the headline survival and success numbers require revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The core numbers—40–68% fact-conflict rates and 'only isolated runs satisfying all achievement commitments'—are produced by the LLM audit pipeline in §4.1 and Appendix H. The weak point is not only the conflict classifier but the state-update components that feed it. The Fact Update prompt contains an explicit 'default bias': 'when uncertain, do not add; when uncertain, do not negate; preserve Ft unless the response makes change unavoidable.' If a narrator response clearly establishes a state change (e.g., a door is unlocked), a conservative fact updater may fail to add the new fact and negate the old one. On a later turn the narrator acts consistently with the new state, but the Conflict Check—which 'treat[s] pre-turn facts as a starting snapshot'—flags a contradiction against the stale ledger. The Conflict Double-Check re-audits using the same stale pre-turn facts, so it does not catch this class of false positives; it only filters conflicts that are unsupported given the (possibly wrong) ledger. Similarly, the Commitment Status Check defaults to PENDING unless a satisfaction condition is 'explicitly met by at least one item in current facts or recorded in interaction history'; if the ledger missed the relevant fact, genuine achievement satisfactions are undercounted, directly suppressing the success rate. The human verification in §5.4 reviews only terminal conflict outputs for GPT-4o-mini, not intermediate ledger states or PENDING statuses, and it does not cover GPT-5.2 (the model whose 42% survival drives the abstract). Because every model-environment pair was run once with temperature 0.6, the reported point estimates also have unknown variance. A systematic under-update bias of this kind could move the headline numbers materially without being detected by the paper's current checks.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper formalizes Narrative Commitment Preservation (NCP) for interactive narrative agents and introduces NCP-Bench, a benchmark of 100 movie-synopsis-derived environments with explicit initial fact ledgers, commitment sets, and reference trajectories. A fixed-prompt LLM audit pipeline checks each narrator response for fact, commitment, and player-input conflicts, updates the fact ledger and trajectory progress, and tracks commitment satisfaction. Experiments with six LLM narrators under an adversarial player agent report rapid survival decline (GPT-5.2: 42% survival at 20 turns), fact conflict rates of 40–68%, and almost no runs satisfying all achievement commitments. Additional experiments compare auditor backbones, a memory-augmented agent baseline, and natural versus adversarial player inputs, plus a single-environment human pilot.","tokens_in":25517,"tokens_out":4556,"duration_ms":49676,"significance":"If the audit pipeline is trustworthy, NCP-Bench is a useful contribution: it turns an important but previously preference-based failure mode—long-horizon logical consistency under open-ended user intervention—into an explicit, checkable benchmark. The dataset construction is careful (manual cleaning, point-of-view constraints, expert review of specifications), the paper ships data/code/prompts, and the main qualitative finding (state-of-the-art LLMs frequently contradict established facts over many turns) is plausible and broadly consistent with prior work on hallucination and self-contradiction. The comparison against natural inputs, the memory-augmented baseline, and the auditor-sensitivity ablation are valuable sanity checks. However, the headline quantitative claims rest on an LLM audit pipeline whose fact-update and commitment-status components are only weakly validated, so the exact conflict and success rates should be treated as provisional until that validation is supplied.","major_comments":[{"comment":"The headline rates in Table 1 and Figure 4 are produced by the LLM audit pipeline, and that pipeline has a load-bearing validity gap. The Fact Update prompt contains an explicit default bias ('when uncertain, do not add; when uncertain, do not negate; preserve Ft unless the response makes change unavoidable'), while the Conflict Check prompt tells the auditor to 'treat pre-turn facts as a starting snapshot.' Together these can leave the ledger stale: a narrator response may clearly establish a state change, yet the fact updater fails to record it, and a later consistent response is then flagged as a conflict against the old snapshot. The Conflict Double-Check re-audits with the same pre-turn facts, so it cannot catch this class of false positives; it only filters conflicts that are unsupported relative to a possibly wrong ledger. Similarly, the Commitment Status Check defaults to PENDING unless satisfaction is 'explicitly met by at least one item in current facts or recorded in interaction history,' so a missed ledger fact also suppresses genuine achievement satisfactions and deflates the near-zero success counts. The human verification in §5.4 covers only 100 final outputs for GPT-4o-mini and does not audit intermediate ledger states, PENDING statuses, or any GPT-5.2 runs. I therefore cannot take the specific numbers (40–68% fact-conflict rates, 'only isolated runs satisfying all achievement commitments') as established. Please add a validation study that samples intermediate turns across models and has human annotators judge (a) whether each fact update is correct, (b) whether each confirmed conflict is a true conflict, and (c) whether each PENDING/SATISFIED commitment status is correct; report precision/recall for each audit component. An alternative or complement is to run the same narrators under a more permissive fact-update policy and show the conflict and success rates are not materially changed.","section":"§4.1, §5.4, Appendix H"},{"comment":"All main results are based on a single stochastic run per model-movie pair (temperature = 0.6, top-p = 0.95, no repeated seeds), yet the headline claims are precise quantities such as '42% survival rate after 20 turns,' '32.92 average turns,' and the per-model fact-conflict percentages. Survival after 20 turns is a binary trajectory outcome, so with one run per environment the sampling noise is large; the observed cross-model ordering (e.g., GPT-5.2 above GPT-4o-mini and DeepSeek-V3.2) could change under re-sampling. Please report multiple seeds with confidence intervals, or at least provide a variance estimate and show that the qualitative ordering is stable. This is fixable within the manuscript's scope and is needed before the fine-grained performance claims can be accepted.","section":"§5.1, Figure 4, Table 1"},{"comment":"The auditor-sensitivity ablation shows that the aggregate conflict-profile ordering is stable, but the strictness of success-related outcomes is not: the number of conflict-free runs for GPT-4o-mini ranges from 0 (GPT-5.2 auditor) to 5 (Gemini-2.5-Flash and GPT-5.4-mini auditors), and trajectory progress at termination ranges from 7.67% to 16.64%. The paper's qualitative claim that 'only isolated runs' succeed does survive under two of the three auditors, but the exact conclusion depends on auditor strictness, and pairwise correlation across all 100 movies does not tell us which auditor is accurate. This reinforces the need for gold-standard human adjudication of audit outputs, as requested above, rather than a comparison among LLM auditors alone.","section":"§5.4, Table 2"}],"minor_comments":[{"comment":"The human baseline is a pilot on a single environment (Iron Man) and is used to conclude that the task is 'well-defined and achievable by human standards.' That generalization is too strong from one environment; please either add a few human runs on additional environments or qualify the claim as a single-environment demonstration.","section":"Appendix B, Table 5"},{"comment":"The appendix says all prompts are listed 'in structured form for clarity' and that complete verbatim prompts are available at a URL, but several prompts are only excerpted (e.g., the Narrative Response prompt). For a benchmark whose core protocol is prompt-based, the paper itself should contain the full verbatim prompts or state clearly that the appendix contains only condensed versions and that the official versions are release artifacts.","section":"Appendix H"},{"comment":"The histogram tick labels render as garbled '/uni000000...' tokens in the provided text; the figure needs to be regenerated with readable numeric labels.","section":"Figure 6"},{"comment":"Genre-level cells with n=1 or n=2 (Sport, Animation, Biography) are interpreted in the text despite high variance; the authors do add a caution at the end of Appendix G, but the main text still cites the Qwen3-235B-A22B 52.38% Biography trajectory-progress value. Please move that discussion behind a clearer warning or restrict claims to genres with n≥8.","section":"Table 7"},{"comment":"The Limitations section appropriately acknowledges that 'auditor judgments may still be imperfect,' but it does not connect this to the potential stale-ledger mechanism described in Major Comment 1. A sentence explaining which audit components are most at risk would help readers calibrate confidence in the headline numbers.","section":"§6, Limitations"}],"recommendation":"major_revision","confidential_remarks":"The central idea is well matched to a benchmark-oriented venue, and I see no circularity or novelty-disclosure problem: the paper measures model behavior rather than deriving results from fitted parameters. The main risk is that the LLM audit pipeline's state-update and commitment-status components are insufficiently validated, which directly affects the headline conflict and success rates. The requested validation study (human adjudication of intermediate ledger states and audit outputs across models) is within the manuscript's scope and would, in my view, make the paper acceptable. The single-seed stochastic evaluation is a second, more straightforward issue that should also be fixed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a real contribution. NCP-Bench gives the field a concrete, auditable task definition for long-horizon consistency in interactive narrative—explicit fact ledgers, commitment sets, reference trajectories, an adversarial player, and automated auditing. None of the prior work combines these pieces. The data construction is careful: movie synopses converted by an agentic pipeline, with human cleaning, POV constraints, and diversity selection. The human narrator baseline on Iron Man is a nice touch; it shows the task is solvable.\n\nWhat it does well: the auditor sensitivity analysis compares three different LLM auditors and finds high pairwise correlations, which suggests the aggregate conflict profiles are not tied to one evaluator. Human verification of 100 terminal outputs found only 4 disputed fact-conflict flags. The qualitative failure analysis is grounded in concrete examples.\n\nNow the soft spots. The stress-test concern about the fact-update prompt's 'default bias' lands. The prompt explicitly says when uncertain, don't add, don't negate, preserve the ledger unless change is unavoidable. If the updater misses a legitimate state change—a door unlocked, a character moved—the ledger goes stale; the next turn's conflict check treats the stale pre-turn facts as the starting snapshot and can flag the narrator for acting consistently with the new state. The double-check re-audits with the same stale facts, so it filters only unsupported claims, not this class of false positives. The commitment status check has a parallel bias: it defaults to PENDING unless satisfaction is explicit, so genuine achievements can be undercounted. The human verification only covers terminal outputs for GPT-4o-mini, not intermediate ledger states and not GPT-5.2, whose 42% survival anchors the abstract. So the point estimates—40–68% fact conflicts, 'only isolated runs satisfying all achievements'—should be treated as upper bounds on failure, not calibrated truth.\n\nAlso, every model-environment pair was run once. With temperature 0.6, we have no sense of variance. That is fixable.\n\nBut the big qualitative conclusion—current LLM narrators are brittle over long horizons—is likely to survive these checks. Even with natural, cooperative inputs, the majority of runs still end in conflict, and no run satisfied all achievements. The human baseline shows the benchmark is not impossibly hard. The citation pattern is solid; the related work is genuinely engaged.\n\nRecommendation: yes, it deserves peer review. The right call is conditional acceptance after the authors archive artifacts, add repeated runs, and do a proper audit validation that checks intermediate ledger states and covers the headline model. The benchmark will be useful to the AI4G and LLM-agent communities, and the audit pitfalls are instructive for anyone building LLM-based evaluation pipelines.","headline":"Useful new benchmark with a believable big finding, but the audit pipeline's conservative state updates likely inflate the headline failure numbers.","tokens_in":26090,"tokens_out":3250,"would_cite":true,"duration_ms":30350,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that long-horizon logical consistency in interactive narrative is a measurable constraint-satisfaction problem, and that state-of-the-art LLM narrators fail it badly under adversarial player input.","keywords":["Narrative Commitment Preservation","interactive narrative","long-horizon consistency","LLM-as-judge auditing","adversarial player agent","fact ledger","movie-synopsis benchmark"],"falsifier":"Run the NCP-Bench audit pipeline on a set of human-written narrative continuations that expert annotators agree are conflict-free, and count false conflicts: if the auditor flags a large share of these as inconsistent, the reported survival rates are artificially low and the benchmark's central numbers would need revision. Alternatively, have human judges label a random sample of terminal outputs from all six models and compare their conflict verdicts with the auditor's.","tokens_in":25018,"feed_emoji":"🎭","tokens_out":6425,"duration_ms":56312,"temperature":0.7,"pith_summary":"The paper tries to establish that interactive narrative is not mainly a creative-writing task but a long-horizon constraint-satisfaction task, and that current large language models are bad at it. It formalizes Narrative Commitment Preservation (NCP): a narrator must keep an explicit fact ledger, honor fixed story commitments (invariants, ordering rules, and achievements), and advance a reference trajectory, even when a free-form player tries to skip, negate, or rewrite the plot. To test this, the paper builds NCP-Bench, 100 movie-derived environments with automatically auditable specifications, and runs six frontier LLM narrators against an adversarial player. The headline results are low survival: the best model keeps only 42% of environments conflict-free after 20 turns, fact conflicts occur in 40–68% of runs, and almost no run satisfies all achievement commitments within 100 turns. The paper concludes that linguistic fluency does not buy logical reliability, and that the gap is a benchmarkable, currently unsolved problem.","feed_headline":"LLM narrators lose the plot: best model survives 42% of episodes","feed_subtitle":"New benchmark: AI storytellers rarely keep facts and plot promises straight under player pressure.","key_machinery":"The load-bearing mechanism is a structured narrative specification $\\langle F_0, C, R \\rangle$: an initial fact ledger $F_0$, a fixed set of commitments $C$ (invariants, ordering rules, and achievements), and an ordered reference trajectory $R$ whose nodes each have a trigger event and a key delta. At each turn, a prompt-fixed LLM auditor performs conflict checks (fact, commitment, and player-input), extracts fact updates, advances trajectory position, and updates commitment status; suspected conflicts get a secondary double-check before an episode terminates. This narrator-decoupled auditing is what makes commitment preservation measurable and reproducible.","core_discovery":"On the paper's own terms, the central discovery is that state-of-the-art LLM narrators systematically break the logical promises of a story world under plausible adversarial play. Using a fixed evaluation loop that audits each narrator response against a fact ledger, a commitment set, and a trajectory, the authors measure that the best model survives only 42% of episodes to 20 turns, that fact contradictions are the dominant failure (40–68% of runs), and that only isolated runs reach the 100-turn limit or satisfy all achievement commitments. Even a memory-augmented agent that reduces commitment violations does not increase full successes, and even cooperative (non-adversarial) players do not make the task easy; the authors also show a human narrator can complete an example environment under the same adversarial pressure, indicating the benchmark is solvable in principle.","pith_inferences":["A testable extension would be to vary the player agent's adversarial strategy (e.g., absent-minded, malicious, or speedrun) and see whether survival curves shift; the paper uses one specific stress-testing policy.","The auditor-strictness spread (success counts from 0 to 5 across auditors) suggests that future work should publish both the primary auditor's numbers and a human-normalized calibration score before comparing models.","If commitment preservation is treated as a first-class training signal, the benchmark could be used as a reward function for fine-tuning narrator models; the paper only evaluates, it does not train.","The failure taxonomy (contradiction, premature disclosure, reality rewriting, ignoring input) could be converted into a diagnostic suite for agent memory designs, predicting which architectural changes reduce which failure type."],"forward_implications":["Reproducible benchmarking of commitment preservation becomes possible: any narrator agent can be plugged into the same fixed audit loop and compared on survival rate, trajectory progress, and satisfied commitments.","Fluency-focused story generation systems cannot be assumed to be reliable game masters; explicit state tracking or external memory is not sufficient on its own, since the memory-augmented baseline still fails.","Adversarial player inputs amplify but do not create the difficulty; cooperative play still ends in conflict for the majority of runs, so the bottleneck is long-horizon consistency itself.","Because the human pilot succeeds on an example environment, the benchmark measures a real, solvable capability rather than an impossible task.","The same NCP abstraction is claimed to transfer to coding agents, instruction-following systems, and tool use, where commitments are interface contracts and tests that must not be silently broken."],"supporting_citations":[{"why":"Supplies the movie synopsis corpus from which the 100 narrative environments are built.","marker":"Bamman et al., 2013"},{"why":"Establishes the interactive-drama assumption that player agency and authorial plot control must coexist, the tension NCP formalizes.","marker":"Mateas & Stern, 2003"},{"why":"Frames narrative as planning and constraint satisfaction, the view the paper extends to LLM narrators.","marker":"Riedl & Young, 2010"},{"why":"Provides the notion of explicit plot-state tracking and commitments that the narrative specification draws on.","marker":"Rashkin et al., 2020"},{"why":"HiAgent is the memory-augmented baseline whose comparison shows memory alone does not solve commitment preservation.","marker":"Hu et al., 2025"},{"why":"Documents self-contradictory LLM hallucinations and the lack of a training penalty for inconsistency, explaining the failure mechanism.","marker":"Mündler et al., 2024"},{"why":"Explains why models accommodate adversarial user claims instead of upholding established facts, the behavioral failure the player agent exploits.","marker":"Wei et al., 2023"}],"fun_headline_variants":["LLM storytellers break plot promises: best survives 42% of runs","AI narrators fail narrative consistency: best only 42% survival","NCP-Bench: LLM storytellers fail to keep narrative commitments","Adversarial play defeats LLM narrators: best survives 42% of episodes","LLM narrators contradict story facts in up to 68% of runs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole measurement depends on the frozen LLM auditors correctly distinguishing genuine logical conflicts from acceptable narrative changes in free-form text; the paper only spot-checks this with three auditor models and human review of 100 outputs from one narrator model.","fun_headline_variants_meta":{"raw":{"variants":["LLM storytellers break plot promises: best survives 42% of runs","AI narrators fail narrative consistency: best only 42% survival","NCP-Bench: LLM storytellers fail to keep narrative commitments","Adversarial play defeats LLM narrators: best survives 42% of episodes","LLM narrators contradict story facts in up to 68% of runs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001412,"raw_usage":{"total_tokens":5686,"prompt_tokens":913,"completion_tokens":4773,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":4673}},"tokens_in":529,"tokens_out":4773,"duration_ms":35016,"temperature":1.0,"reasoning_tokens":4673,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:19:20.817367+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the NCP-Bench audit pipeline on a set of human-written narrative continuations that expert annotators agree are conflict-free, and count false conflicts: if the auditor flags a large share of these as inconsistent, the reported survival rates are artificially low and the benchmark's central numbers would need revision. Alternatively, have human judges label a random sample of terminal outputs from all six models and compare their conflict verdicts with the auditor's.","supporting_citations":[{"cited_title":"Plot- machines: Outline-conditioned generation with dynamic plot state tracking","cited_arxiv_id":null,"evidence_quote":"Provides the notion of explicit plot-state tracking and commitments that the narrative specification draws on."}],"review_version":1}