Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narratives

Bei Shi; Derek F. Wong; Jianhao Yan; Ka Hou Kam; Runnan Wang; Xuebo Liu; Yingpeng Ma; Yue Zhang; Yulong Chen

REVIEW 3 major objections 5 minor 21 references

Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narratives

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that long-horizon logical consistency in interactive narrative is a measurable constraint-satisfaction problem, and that state-of-the-art LLM narrators fail it badly under adversarial player input.

desk verdict Useful new benchmark with a believable big finding, but the audit pipeline's conservative state updates likely inflate the headline failure numbers. read the letter →

arxiv 2608.08160 v1 pith:H7U45MRY submitted 2026-08-08 cs.CL cs.AI

Yingpeng Ma , Jianhao Yan , Bei Shi , Ka Hou Kam , Runnan Wang , Xuebo Liu , Yulong Chen , Yue Zhang

show 1 more author

Derek F. Wong

This is my paper · ORCID

classification cs.CLcs.AI

keywords NarrativeCommitmentPreservationinteractivelong-horizonconsistencyLLM-as-judgeauditingadversarialplayeragentfactledgermovie-synopsisbenchmark

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that interactive narrative is not mainly a creative-writing task but a long-horizon constraint-satisfaction task, and that current large language models are bad at it. It formalizes Narrative Commitment Preservation (NCP): a narrator must keep an explicit fact ledger, honor fixed story commitments (invariants, ordering rules, and achievements), and advance a reference trajectory, even when a free-form player tries to skip, negate, or rewrite the plot. To test this, the paper builds NCP-Bench, 100 movie-derived environments with automatically auditable specifications, and runs six frontier LLM narrators against an adversarial player. The headline results are low survival: the best model keeps only 42% of environments conflict-free after 20 turns, fact conflicts occur in 40–68% of runs, and almost no run satisfies all achievement commitments within 100 turns. The paper concludes that linguistic fluency does not buy logical reliability, and that the gap is a benchmarkable, currently unsolved problem.

What carries the argument

The load-bearing mechanism is a structured narrative specification $\langle F_0, C, R \rangle$: an initial fact ledger $F_0$, a fixed set of commitments $C$ (invariants, ordering rules, and achievements), and an ordered reference trajectory $R$ whose nodes each have a trigger event and a key delta. At each turn, a prompt-fixed LLM auditor performs conflict checks (fact, commitment, and player-input), extracts fact updates, advances trajectory position, and updates commitment status; suspected conflicts get a secondary double-check before an episode terminates. This narrator-decoupled auditing is what makes commitment preservation measurable and reproducible.

What would settle it

Run the NCP-Bench audit pipeline on a set of human-written narrative continuations that expert annotators agree are conflict-free, and count false conflicts: if the auditor flags a large share of these as inconsistent, the reported survival rates are artificially low and the benchmark's central numbers would need revision. Alternatively, have human judges label a random sample of terminal outputs from all six models and compare their conflict verdicts with the auditor's.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that state-of-the-art LLM narrators systematically break the logical promises of a story world under plausible adversarial play. Using a fixed evaluation loop that audits each narrator response against a fact ledger, a commitment set, and a trajectory, the authors measure that the best model survives only 42% of episodes to 20 turns, that fact contradictions are the dominant failure (40–68% of runs), and that only isolated runs reach the 100-turn limit or satisfy all achievement commitments. Even a memory-augmented agent that reduces commitment violations does not increase full successes, and even cooperative (non-adversarial) players do not make the task easy; the authors also show a human narrator can complete an example environment under the same adversarial pressure, indicating the benchmark is solvable in principle.

Load-bearing premise

The whole measurement depends on the frozen LLM auditors correctly distinguishing genuine logical conflicts from acceptable narrative changes in free-form text; the paper only spot-checks this with three auditor models and human review of 100 outputs from one narrator model.

Editorial extensions

If this is right

Reproducible benchmarking of commitment preservation becomes possible: any narrator agent can be plugged into the same fixed audit loop and compared on survival rate, trajectory progress, and satisfied commitments.
Fluency-focused story generation systems cannot be assumed to be reliable game masters; explicit state tracking or external memory is not sufficient on its own, since the memory-augmented baseline still fails.
Adversarial player inputs amplify but do not create the difficulty; cooperative play still ends in conflict for the majority of runs, so the bottleneck is long-horizon consistency itself.
Because the human pilot succeeds on an example environment, the benchmark measures a real, solvable capability rather than an impossible task.
The same NCP abstraction is claimed to transfer to coding agents, instruction-following systems, and tool use, where commitments are interface contracts and tests that must not be silently broken.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension would be to vary the player agent's adversarial strategy (e.g., absent-minded, malicious, or speedrun) and see whether survival curves shift; the paper uses one specific stress-testing policy.
The auditor-strictness spread (success counts from 0 to 5 across auditors) suggests that future work should publish both the primary auditor's numbers and a human-normalized calibration score before comparing models.
If commitment preservation is treated as a first-class training signal, the benchmark could be used as a reward function for fine-tuning narrator models; the paper only evaluates, it does not train.
The failure taxonomy (contradiction, premature disclosure, reality rewriting, ignoring input) could be converted into a diagnostic suite for agent memory designs, predicting which architectural changes reduce which failure type.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing to know: this is a real contribution. NCP-Bench gives the field a concrete, auditable task definition for long-horizon consistency in interactive narrative—explicit fact ledgers, commitment sets, reference trajectories, an adversarial player, and automated auditing. None of the prior work combines these pieces. The data construction is careful: movie synopses converted by an agentic pipeline, with human cleaning, POV constraints, and diversity selection. The human narrator baseline on Iron Man is a nice touch; it shows the task is solvable.

What it does well: the auditor sensitivity analysis compares three different LLM auditors and finds high pairwise correlations, which suggests the aggregate conflict profiles are not tied to one evaluator. Human verification of 100 terminal outputs found only 4 disputed fact-conflict flags. The qualitative failure analysis is grounded in concrete examples.

Now the soft spots. The stress-test concern about the fact-update prompt's 'default bias' lands. The prompt explicitly says when uncertain, don't add, don't negate, preserve the ledger unless change is unavoidable. If the updater misses a legitimate state change—a door unlocked, a character moved—the ledger goes stale; the next turn's conflict check treats the stale pre-turn facts as the starting snapshot and can flag the narrator for acting consistently with the new state. The double-check re-audits with the same stale facts, so it filters only unsupported claims, not this class of false positives. The commitment status check has a parallel bias: it defaults to PENDING unless satisfaction is explicit, so genuine achievements can be undercounted. The human verification only covers terminal outputs for GPT-4o-mini, not intermediate ledger states and not GPT-5.2, whose 42% survival anchors the abstract. So the point estimates—40–68% fact conflicts, 'only isolated runs satisfying all achievements'—should be treated as upper bounds on failure, not calibrated truth.

Also, every model-environment pair was run once. With temperature 0.6, we have no sense of variance. That is fixable.

But the big qualitative conclusion—current LLM narrators are brittle over long horizons—is likely to survive these checks. Even with natural, cooperative inputs, the majority of runs still end in conflict, and no run satisfied all achievements. The human baseline shows the benchmark is not impossibly hard. The citation pattern is solid; the related work is genuinely engaged.

Recommendation: yes, it deserves peer review. The right call is conditional acceptance after the authors archive artifacts, add repeated runs, and do a proper audit validation that checks intermediate ledger states and covers the headline model. The benchmark will be useful to the AI4G and LLM-agent communities, and the audit pitfalls are instructive for anyone building LLM-based evaluation pipelines.

Referee Report

3 major / 5 minor

Summary. The paper formalizes Narrative Commitment Preservation (NCP) for interactive narrative agents and introduces NCP-Bench, a benchmark of 100 movie-synopsis-derived environments with explicit initial fact ledgers, commitment sets, and reference trajectories. A fixed-prompt LLM audit pipeline checks each narrator response for fact, commitment, and player-input conflicts, updates the fact ledger and trajectory progress, and tracks commitment satisfaction. Experiments with six LLM narrators under an adversarial player agent report rapid survival decline (GPT-5.2: 42% survival at 20 turns), fact conflict rates of 40–68%, and almost no runs satisfying all achievement commitments. Additional experiments compare auditor backbones, a memory-augmented agent baseline, and natural versus adversarial player inputs, plus a single-environment human pilot.

Significance. If the audit pipeline is trustworthy, NCP-Bench is a useful contribution: it turns an important but previously preference-based failure mode—long-horizon logical consistency under open-ended user intervention—into an explicit, checkable benchmark. The dataset construction is careful (manual cleaning, point-of-view constraints, expert review of specifications), the paper ships data/code/prompts, and the main qualitative finding (state-of-the-art LLMs frequently contradict established facts over many turns) is plausible and broadly consistent with prior work on hallucination and self-contradiction. The comparison against natural inputs, the memory-augmented baseline, and the auditor-sensitivity ablation are valuable sanity checks. However, the headline quantitative claims rest on an LLM audit pipeline whose fact-update and commitment-status components are only weakly validated, so the exact conflict and success rates should be treated as provisional until that validation is supplied.

major comments (3)

[§4.1, §5.4, Appendix H] The headline rates in Table 1 and Figure 4 are produced by the LLM audit pipeline, and that pipeline has a load-bearing validity gap. The Fact Update prompt contains an explicit default bias ('when uncertain, do not add; when uncertain, do not negate; preserve Ft unless the response makes change unavoidable'), while the Conflict Check prompt tells the auditor to 'treat pre-turn facts as a starting snapshot.' Together these can leave the ledger stale: a narrator response may clearly establish a state change, yet the fact updater fails to record it, and a later consistent response is then flagged as a conflict against the old snapshot. The Conflict Double-Check re-audits with the same pre-turn facts, so it cannot catch this class of false positives; it only filters conflicts that are unsupported relative to a possibly wrong ledger. Similarly, the Commitment Status Check defaults to PENDING unless satisfaction is 'explicitly met by at least one item in current facts or recorded in interaction history,' so a missed ledger fact also suppresses genuine achievement satisfactions and deflates the near-zero success counts. The human verification in §5.4 covers only 100 final outputs for GPT-4o-mini and does not audit intermediate ledger states, PENDING statuses, or any GPT-5.2 runs. I therefore cannot take the specific numbers (40–68% fact-conflict rates, 'only isolated runs satisfying all achievement commitments') as established. Please add a validation study that samples intermediate turns across models and has human annotators judge (a) whether each fact update is correct, (b) whether each confirmed conflict is a true conflict, and (c) whether each PENDING/SATISFIED commitment status is correct; report precision/recall for each audit component. An alternative or complement is to run the same narrators under a more permissive fact-update policy and show the conflict and success rates are not materially changed.
[§5.1, Figure 4, Table 1] All main results are based on a single stochastic run per model-movie pair (temperature = 0.6, top-p = 0.95, no repeated seeds), yet the headline claims are precise quantities such as '42% survival rate after 20 turns,' '32.92 average turns,' and the per-model fact-conflict percentages. Survival after 20 turns is a binary trajectory outcome, so with one run per environment the sampling noise is large; the observed cross-model ordering (e.g., GPT-5.2 above GPT-4o-mini and DeepSeek-V3.2) could change under re-sampling. Please report multiple seeds with confidence intervals, or at least provide a variance estimate and show that the qualitative ordering is stable. This is fixable within the manuscript's scope and is needed before the fine-grained performance claims can be accepted.
[§5.4, Table 2] The auditor-sensitivity ablation shows that the aggregate conflict-profile ordering is stable, but the strictness of success-related outcomes is not: the number of conflict-free runs for GPT-4o-mini ranges from 0 (GPT-5.2 auditor) to 5 (Gemini-2.5-Flash and GPT-5.4-mini auditors), and trajectory progress at termination ranges from 7.67% to 16.64%. The paper's qualitative claim that 'only isolated runs' succeed does survive under two of the three auditors, but the exact conclusion depends on auditor strictness, and pairwise correlation across all 100 movies does not tell us which auditor is accurate. This reinforces the need for gold-standard human adjudication of audit outputs, as requested above, rather than a comparison among LLM auditors alone.

minor comments (5)

[Appendix B, Table 5] The human baseline is a pilot on a single environment (Iron Man) and is used to conclude that the task is 'well-defined and achievable by human standards.' That generalization is too strong from one environment; please either add a few human runs on additional environments or qualify the claim as a single-environment demonstration.
[Appendix H] The appendix says all prompts are listed 'in structured form for clarity' and that complete verbatim prompts are available at a URL, but several prompts are only excerpted (e.g., the Narrative Response prompt). For a benchmark whose core protocol is prompt-based, the paper itself should contain the full verbatim prompts or state clearly that the appendix contains only condensed versions and that the official versions are release artifacts.
[Figure 6] The histogram tick labels render as garbled '/uni000000...' tokens in the provided text; the figure needs to be regenerated with readable numeric labels.
[Table 7] Genre-level cells with n=1 or n=2 (Sport, Animation, Biography) are interpreted in the text despite high variance; the authors do add a caution at the end of Appendix G, but the main text still cites the Qwen3-235B-A22B 52.38% Biography trajectory-progress value. Please move that discussion behind a clearer warning or restrict claims to genres with n≥8.
[§6, Limitations] The Limitations section appropriately acknowledges that 'auditor judgments may still be imperfect,' but it does not connect this to the potential stale-ledger mechanism described in Major Comment 1. A sentence explaining which audit components are most at risk would help readers calibrate confidence in the headline numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: NCP-Bench is an empirical benchmark whose results are measured by an external, human-validated audit pipeline, not derived from fitted parameters or self-cited premises.

full rationale

The paper's central claims are empirical measurements of LLM narrator behavior on a newly constructed benchmark, not derivations from equations whose outputs are built into their inputs. NCP-Bench externalizes facts, commitments, and trajectory nodes into structured specifications, and the evaluation loop uses fixed-prompt LLM auditors with double-checking and human verification (§5.4). The narrator and auditor are distinct components, and the task is shown to be solvable by a human narrator pilot (Appendix B), so the results are not forced by construction. The auditor being an LLM is an evaluation-validity concern, acknowledged in Limitations, but it is not circular reasoning: the audit does not reduce to the narrator's own output or to a fitted parameter. No self-citation chain carries the load, and no equation in the paper is equivalent to its input by definition. Therefore the circularity score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No fitted parameters are needed to produce the central result; the benchmark is a measurement protocol. Design thresholds such as the 100-turn limit and decoding settings are listed as free parameters because they are hand-chosen and affect the exact numbers. The main axioms concern the reliability of LLM-based auditing and the faithfulness of the movie-derived specifications.

free parameters (4)

Maximum interaction length (T_max) = 100 turns
Hand-set terminal condition; the claim that only isolated runs satisfy all achievement commitments is scoped to this horizon.
Decoding temperature = 0.6
Chosen decoding configuration; stochastic outputs make single-run survival estimates variable.
Top-p = 0.95
Chosen nucleus sampling parameter; affects output diversity and therefore conflict rates.
Output validation retry limit = 2 retries
Policy chosen in Section 5.1; a stricter or looser retry policy changes how invalid outputs are converted into terminations.

assumptions (5)

domain assumption LLM auditors approximate ground truth for fact, commitment, and player-input conflicts.
Section 4.1 runs a fixed-prompt LLM auditor with a secondary double-check to decide whether a response violates the ledger. Section 5.4 only partially validates this on one narrator model; human review is limited to 100 GPT-4o-mini outputs.
domain assumption The 100 generated narrative specifications faithfully encode each movie synopsis's constraints.
Section 3.1 uses an agentic pipeline plus expert correction; seven ambiguous specifications were flagged in post hoc review. The benchmark conclusions assume these specifications are valid test sets.
domain assumption The adversarial player agent produces plausible in-world interventions rather than nonsensical inputs.
Section 4.2 instructs a Gemini-2.5-Flash simulator to act as a difficult but in-world player; if its inputs are unrealistic, the stress test may not reflect real user behavior.
ad hoc to paper A single human pilot on the Iron Man environment generalizes to the other 99 tasks.
Appendix B reports one human narrator on one movie to argue the task is solvable; this is used as evidence that failures are model limitations rather than ill-posed tasks.
domain assumption Narrative consistency can be checked from linear, chronologically ordered transcripts.
Section 6 limitations states that the framework targets single-threaded narratives and excludes branching timelines, flashbacks, and parallel perspectives; this bounds the claim's scope.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narratives." pith.science (2026). https://pith.science/paper/H7U45MRY

@misc{pith2026260808160,
  author       = {Pith},
  title        = {Pith review of: Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narratives},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H7U45MRY}},
  note         = {Machine review of arXiv:2608.08160}
}

read the original abstract

The rapid advancement of Large Language Models (LLMs) is revolutionizing AI for Games by enabling open-ended and fluid interactive storytelling. However, existing research has largely overlooked the critical challenge of maintaining long-horizon logical consistency and narrative integrity against unconstrained user interventions. To address this, we formulate this challenge as Narrative Commitment Preservation (NCP), and take interactive narrative as our testbed. We introduce NCP-Bench, a benchmark of 100 narrative environments derived from movie synopses. Each environment includes a structured narrative specification (trajectory, commitments, and initial facts) that we can automatically check throughout the interaction between the player agent and the narrator agent. Experiments across state-of-the-art LLMs reveal a substantial long-horizon consistency gap: high linguistic quality does not guarantee commitment preservation; even strong models frequently generate logically conflicting content under adversarial interventions, with the best-performing model (GPT-5.2) achieving only 42% survival rate after 20 turns and fact conflict rates ranging from 40% to 68% across models, and only isolated runs satisfying all achievement commitments within the 100-turn limit.

Figures

Figures reproduced from arXiv: 2608.08160 by the authors.

**Figure 1.** Divergence between the canonical movie plot and an interactive narrative trajectory. The right panel (green) depicts the canonical plot trajectory. The left panel (orange) illustrates a high-agency interactive session with speedrun interventions (e.g., leveraging advanced gadgets to skip the capture sequence). Dashed lines indicate how interactive actions map to or circumvent original milestones. 1. Introduction The… view at source ↗

**Figure 2.** NCP-Bench turns each movie synopsis into a stateful interactive environment and audits every narrator response. Left: 100 movie synopses are converted into narrative specifications containing an initial fact ledger, reference trajectory, and narrative commitments. Center: an adversarial player and a narrator agent produce a turn-by-turn interaction history. Right: after each narrator response, the evaluator first ch… view at source ↗

**Figure 3.** Primary genre distribution of the 100 movies in NCPBench. Action (13%), Sci-Fi (12%), and Comedy (11%) constitute the largest proportions, followed by Western (9%), Fantasy (8%), Crime (7%), and a long tail of smaller genres. ensure dataset quality, we employ human experts to manually clean the pool by (i) removing duplicates, (ii) resolving character/movie mismatches, and (iii) filtering out cases where the synop… view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Survival rate of narrative environments across interaction turns. The vertical axis denotes the proportion of active environments preserved without logical conflicts or commitment violations (out of 100 initial movies), while the horizontal axis tracks the number of t…

**Figure 5.** Figure 5: Common failure modes of interactive narrative agents. Top-Left: Hallucination & Factual Contradiction. Top-Right: Triggering Unknown Facts. Bottom-Left: Forcibly Changing Reality. Bottom-Right: Ignoring Player Input. rative world states or tracked facts—e.g., the ledge…

**Figure 6.** Figure 6: Per-movie distributions of facts, commitments, and trajectory nodes in NCP-Bench. (Left) Count of movies by the number of initial facts. (Center) Count of movies by the number of commitments. (Right) Count of movies by the number of trajectory nodes. analysis of multi-…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 11 canonical work pages

[1]

R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y ., et al

Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y ., et al. Towards a human-like open-domain chatbot.arXiv preprint arXiv:2001.09977,

arXiv 2001
[4]

Prometheus: Inducing fine-grained evaluation capability in language models

10 Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narratives Kim, S., Shin, J., Jang, J., Longpre, S., Lee, H., Yun, S., Shin, R., Kim, S., Thorne, J., Seo, M., et al. Prometheus: Inducing fine-grained evaluation capability in language models. InInternational Conference on Learning Repre- sentations, volume 202...

work page 2024
[5]

Checkeval: A reliable llm-as-a-judge framework for evaluating text generation using checklists

Lee, Y ., Kim, J., Kim, J., Cho, H., Kang, J., Kang, P., and Kim, N. Checkeval: A reliable llm-as-a-judge framework for evaluating text generation using checklists. InPro- ceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 15782–15809,

work page 2025
[6]

Deepseek-v3

Liu, A., Mei, A., Lin, B., Xue, B., Wang, B., Xu, B., Wu, B., Zhang, B., Lin, C., Dong, C., et al. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556,

arXiv
[8]

Player-driven emergence in llm-driven game narrative

Peng, X., Quaye, J., Rao, S., Xu, W., Botchway, P., Brockett, C., Jojic, N., DesGarennes, G., Lobb, K., Xu, M., et al. Player-driven emergence in llm-driven game narrative. In 2024 IEEE Conference on Games (CoG), pp. 1–8. IEEE,

work page 2024
[15]

Lamda: Language models for dialog appli- cations.arXiv preprint arXiv:2201.08239,

Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kul- shreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y ., et al. Lamda: Language models for dialog appli- cations.arXiv preprint arXiv:2201.08239,

arXiv
[17]

Wang, L., Lian, J., Huang, Y ., Dai, Y ., Li, H., Chen, X., Xie, X., and Wen, J.-R

URL https: //openreview.net/forum?id=xggSxCFQbA. Wang, L., Lian, J., Huang, Y ., Dai, Y ., Li, H., Chen, X., Xie, X., and Wen, J.-R. Characterbox: Evaluating the role- playing capabilities of llms in text-based virtual worlds. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Com- putational Linguistics: Hu...

work page 2025
[18]

ISBN 979-8-89176-251-0

Association for Computational Lin- guistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/ 2025.acl-long.546. URL https://aclanthology. org/2025.acl-long.546/. Xia, C. S., Deng, Y ., Dunn, S., and Zhang, L. Agentless: De- mystifying llm-based software engineering agents.arXiv preprint arXiv:2407.01489,

arXiv 2025

Show all 21 references

[19]

Qwen3 technical report.arXiv preprint arXiv:2505.09388,

Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

arXiv
[20]

Score: Story coherence and retrieval enhancement for ai narratives

Yi, Q., He, Y ., Wang, J., Song, X., Qian, S., Yuan, X., Xin, Y ., Wang, Y ., Tang, J., Li, Y ., et al. Score: Story coherence and retrieval enhancement for ai narratives. arXiv preprint arXiv:2503.23512,
[21]

Interesting

Zhang, Y ., Li, Y ., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y ., Chen, Y ., et al. Siren’s song in the ai ocean: A survey on hallucination in large language models.Computational Linguistics, 51(4):1373–1418, 2025b. 12 Can LLM Agents Stick to the Script?...

2003
[2003]

Self- contradictory hallucinations of large language models: Evaluation, detection and mitigation

M¨undler, N., He, J., Jenko, S., and Vechev, M. Self- contradictory hallucinations of large language models: Evaluation, detection and mitigation. InInternational Conference on Learning Representations, volume 2024, pp. 40364–40393,

2024
[2010]

What makes a good conversation? how controllable attributes affect hu- man judgments

See, A., Roller, S., Kiela, D., and Weston, J. What makes a good conversation? how controllable attributes affect hu- man judgments. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gi...

2019
[2011]

Plot- machines: Outline-conditioned generation with dynamic plot state tracking

Rashkin, H., Celikyilmaz, A., Choi, Y ., and Gao, J. Plot- machines: Outline-conditioned generation with dynamic plot state tracking. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4274–4295,

2020
[2014]

11 Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narratives Team, K., Bai, T., Bai, Y ., Bao, Y ., Cai, S., Cao, Y ., Charles, Y ., Che, H., Chen, C., Chen, G., et al. Kimi k2. 5: Visual agentic intelligence.arXiv preprint arXiv:26...

arXiv
[2019]

J., Chung, J

Sun, Y ., Wang, P. J., Chung, J. J. Y ., Roemmele, M., Kim, T., and Kreminski, M. Drama llama: An llm-powered storylets framework for authorable responsiveness in in- teractive narrative.arXiv preprint arXiv:2501.09099,

arXiv
[2022]

Are large language models capable of generating human-level narratives? InPro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp

Tian, Y ., Huang, T., Liu, M., Jiang, D., Spangher, A., Chen, M., May, J., and Peng, N. Are large language models capable of generating human-level narratives? InPro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 17659–17681,

2024
[2023]

Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261,

Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprin...

arXiv
[2024]

Red teaming language models with language models

Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., and Irving, G. Red teaming language models with language models. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3419–3448,

2022
[2025]

P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al

Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276,

arXiv
[2026]

T., Liu, H., Liu, T., Wang, C., Liu, T., Zhang, Y ., Shipman, F., et al

Teleki, M., Bengali, V ., Dong, X., Janjur, S. T., Liu, H., Liu, T., Wang, C., Liu, T., Zhang, Y ., Shipman, F., et al. A survey on llms for story generation. InFindings of the Association for Computational Linguistics: EMNLP 2025, pp. 13954–13966,

2025

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y ., et al

Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y ., et al. Towards a human-like open-domain chatbot.arXiv preprint arXiv:2001.09977,

arXiv 2001

[2] [4]

Prometheus: Inducing fine-grained evaluation capability in language models

10 Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narratives Kim, S., Shin, J., Jang, J., Longpre, S., Lee, H., Yun, S., Shin, R., Kim, S., Thorne, J., Seo, M., et al. Prometheus: Inducing fine-grained evaluation capability in language models. InInternational Conference on Learning Repre- sentations, volume 202...

work page 2024

[3] [5]

Checkeval: A reliable llm-as-a-judge framework for evaluating text generation using checklists

Lee, Y ., Kim, J., Kim, J., Cho, H., Kang, J., Kang, P., and Kim, N. Checkeval: A reliable llm-as-a-judge framework for evaluating text generation using checklists. InPro- ceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 15782–15809,

work page 2025

[4] [6]

Deepseek-v3

Liu, A., Mei, A., Lin, B., Xue, B., Wang, B., Xu, B., Wu, B., Zhang, B., Lin, C., Dong, C., et al. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556,

arXiv

[5] [8]

Player-driven emergence in llm-driven game narrative

Peng, X., Quaye, J., Rao, S., Xu, W., Botchway, P., Brockett, C., Jojic, N., DesGarennes, G., Lobb, K., Xu, M., et al. Player-driven emergence in llm-driven game narrative. In 2024 IEEE Conference on Games (CoG), pp. 1–8. IEEE,

work page 2024

[6] [15]

Lamda: Language models for dialog appli- cations.arXiv preprint arXiv:2201.08239,

Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kul- shreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y ., et al. Lamda: Language models for dialog appli- cations.arXiv preprint arXiv:2201.08239,

arXiv

[7] [17]

Wang, L., Lian, J., Huang, Y ., Dai, Y ., Li, H., Chen, X., Xie, X., and Wen, J.-R

URL https: //openreview.net/forum?id=xggSxCFQbA. Wang, L., Lian, J., Huang, Y ., Dai, Y ., Li, H., Chen, X., Xie, X., and Wen, J.-R. Characterbox: Evaluating the role- playing capabilities of llms in text-based virtual worlds. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Com- putational Linguistics: Hu...

work page 2025

[8] [18]

ISBN 979-8-89176-251-0

Association for Computational Lin- guistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/ 2025.acl-long.546. URL https://aclanthology. org/2025.acl-long.546/. Xia, C. S., Deng, Y ., Dunn, S., and Zhang, L. Agentless: De- mystifying llm-based software engineering agents.arXiv preprint arXiv:2407.01489,

arXiv 2025