Pith

open record

sign in

arxiv: 2606.22454 · v1 · pith:LDC673VL · submitted 2026-06-21 · cs.CL · cs.AI

CASPER in the Machine: Insights into Character Variety in LLM-Generated Stories

Anneliese Brei , Abhisheik Sharma , Nicholas Sanaie , Lu Wang , Snigdha Chaturvedi This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-26 10:42 UTCgrok-4.3pith:LDC673VLrecord.jsonopen to challenge →

classification cs.CL cs.AI
keywords LLM-generated storiescharacter analysisnarratologystory comparisonfictional charactersautomatic categorizationhuman vs machine text
0
0 comments X

The pith

LLM-generated stories show character categories similar to human-written ones across eight narratological dimensions, with some differences in variety.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper applies narratological definitions of eight character dimensions to compare LLM-generated stories against recently published human-written ones. It automatically infers character categories in both sets and examines whether the characters match and whether LLMs produce a range of character types. A sympathetic reader would care because widespread use of LLMs for fiction raises questions about whether their outputs reproduce or diverge from human storytelling patterns in subtle ways beyond surface traits.

Core claim

After automatically inferring categories of characters within both LLM and human-written stories using eight narratological dimensions such as stylization and wholeness, the two sets of stories exhibit a number of interesting similarities and differences; LLMs and human-written stories have similar characters overall, and LLMs generate stories with a variety of characters.

What carries the argument

Automatic inference of character categories from stories based on eight narratological dimensions that assess portrayal rather than basic traits.

If this is right

  • LLMs can produce stories whose characters align with human ones on multiple portrayal dimensions.
  • Differences in specific dimensions can guide improvements to LLM story generation.
  • LLM outputs already contain varied character types rather than repeating narrow patterns.
  • Comparisons using these dimensions can extend to other genres or future model versions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the dimension-based categories hold, future work could test whether adjusting prompts changes the distribution of character types in LLM stories.
  • The approach could apply to measuring character consistency across a single long story rather than across many separate stories.
  • Similar inference methods might reveal whether character variety correlates with reader engagement metrics in human evaluations.

Load-bearing premise

The automatic inference of character categories from stories accurately reflects the eight narratological dimensions without substantial misclassification or loss of nuance.

What would settle it

A manual audit of a sample of inferred character categories that finds frequent mismatches with the narratological definitions would undermine the reported similarities and differences.

Figures

Figures reproduced from arXiv: 2606.22454 by Abhisheik Sharma, Anneliese Brei, Lu Wang, Nicholas Sanaie, Snigdha Chaturvedi.

Figure 1
Figure 1. Figure 1: We analyze characters in LLM-generated short stories (left) using 8 category-pairs (right) that consider how characters are portrayed. (e.g., a charac￾ter might be represented in a fragmented way, setting the tone for a disjointed mood within the story). We compare LLM-generated characters with human-written characters other and LLM-generated characters from dif￾ferent model sizes and families. Narratologi… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of CASPER framework, including the pipeline for the creation of the corpus (top row), experiments (middle row), and analysis of the categories (bottom row). tion, ICL-interleaved for coherence, wholeness, complexity, and transparency, ICL-basic for lit￾eralness, and Zero-shot-basic for dynamism and closure. A description of the results and F1 scores breakdown is given in Appendix B.3. 4 Evaluating… view at source ↗
Figure 3
Figure 3. Figure 3: RQ2: We compare the percentage of human￾written stories (blue) with the percentage of all LLM￾generated stories (green) w.r.t. categories. For the latter, we average generations from all LLMs and show stan￾dard deviation across LLMs with vertical lines. labels and rewrite A = ⟨ASty., ACoh., AW hol., ALit., AComp., AT ran., ADyn., AClos.⟩, such that each A is represented by majority labels (a label is in th… view at source ↗
Figure 4
Figure 4. Figure 4: , averages of smaller and larger LLMs are nearly identical. The only minor difference is that smaller LLMs generate 10.2% more stylized characters. Our findings are surprising because they indicate that, in general, though larger LLMs are better at text generation, they do not necessarily produce different types of characters than smaller LLMs. Overall, model size does not appear to affect the types of cha… view at source ↗
Figure 5
Figure 5. Figure 5: RQ4: Averaged categories of characters across families. Standard deviations across different￾sized models within a family are shown as vertical bars. the other families, Mistral prefers more stylized and less coherent, closed characters, indicating an interesting mixture of exaggerated characters who might raise questions from readers. We notice only Phi shows significant standard deviation (> 5%) across m… view at source ↗
Figure 6
Figure 6. Figure 6: RQ5: Distribution of categories, given each specified genre across all LLM-generated stories. on everyday life, these characters still represent abstract themes without relying on archetypes. Romance stories have nearly completely coherent, whole, transparent characters, which indicates the characters are straight-forward and not confused with emotions, as we might expect. Sci-Fi/Fantasy stories have the h… view at source ↗
Figure 7
Figure 7. Figure 7: Comparison of wrongly classified category labels for humans (blue) vs. LLMs (orange). The x-axis [PITH_FULL_IMAGE:figures/full_fig_p022_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Comparison of types of characters generated [PITH_FULL_IMAGE:figures/full_fig_p024_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: RQ6: Variability of categories across 3 identical inference calls (averaged over all open-sourced LLMs). X-axes are writing-prompt IDs. For a given writing-prompt, y-axes show whether the generated character has identical categories across all 3 inference calls (y = 1) or if at least one label differs (y = 0). For stylization, coherency, wholeness, literalness, complexity, transparency, dynamism and closu… view at source ↗
read the original abstract

As LLM-generated text is increasingly used, especially in fictional domains, we explore how much LLM-generated stories differ from human-written stories. In this work, we focus on characters. We borrow definitions from narratology to analyze eight intricate dimensions of character, such as stylization and wholeness. These dimensions consider more than just basic characteristics. They assess how characters are portrayed within their stories. After automatically inferring categories of characters within both LLM and human-written stories, we compare and contrast these two sets of stories. We consider the following overarching questions: (1) Do LLMs and human-written stories have similar characters? and (2) Do LLMs generate stories with a variety of characters? Our analysis includes research questions that focus on stories generated by popular LLMs and recently published human-written stories. We describe a number of interesting similarities, differences and key takeaways.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 1 minor

Summary. The paper claims to analyze character variety in LLM-generated versus human-written stories by borrowing eight narratological dimensions (e.g., stylization, wholeness) to automatically infer character categories from both sets of stories, then comparing and contrasting them to address whether LLMs produce similar characters to humans and whether they generate sufficient variety, based on popular LLMs and recent human stories, while describing similarities, differences, and takeaways.

Significance. If the automatic inference step is shown to be reliable, the work could provide a structured, theory-grounded comparison of character portrayal that goes beyond surface features, offering insights into LLM capabilities in creative domains. The explicit grounding in narratological frameworks is a constructive element that could support more nuanced evaluations of generated fiction.

major comments (1)
  1. [Methods] Methods section: The central claim that LLM and human stories can be meaningfully compared on character variety after automatic category inference depends on the inference faithfully recovering the eight narratological dimensions. However, no human-annotated gold labels, inter-annotator agreement, or error analysis on the inference procedure are reported. If the automatic labels systematically deviate from the narratological definitions, the reported similarities, differences, and takeaways become unreliable.
minor comments (1)
  1. [Abstract] Abstract: States the approach and questions but supplies no methods details, data, or results, which hinders assessment of whether the comparisons support the described similarities and differences.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback on the methods. We address the major comment below.

read point-by-point responses

  1. Referee: [Methods] Methods section: The central claim that LLM and human stories can be meaningfully compared on character variety after automatic category inference depends on the inference faithfully recovering the eight narratological dimensions. However, no human-annotated gold labels, inter-annotator agreement, or error analysis on the inference procedure are reported. If the automatic labels systematically deviate from the narratological definitions, the reported similarities, differences, and takeaways become unreliable.

    Authors: We agree that the reliability of the automatic inference is central to the validity of the comparisons and takeaways. The manuscript describes the procedure for mapping story text to the eight narratological dimensions but does not report human validation, IAA, or error analysis. In the revised version we will add a targeted human evaluation on a representative sample of both LLM and human stories: annotators will label a subset according to the same dimension definitions, we will report agreement with the automatic outputs, inter-annotator agreement, and a qualitative error analysis. This addition will directly address the concern that systematic deviations could undermine the reported similarities and differences. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical comparison relies on external narratological definitions and data without self-referential reductions

full rationale

The paper borrows eight narratological dimensions from external sources, applies automatic inference to categorize characters in LLM and human stories, then performs direct comparisons. No equations, parameter fits, predictions of fitted quantities, or self-citation chains appear in the provided text that would reduce any claim to its own inputs by construction. The inference step is presented as a methodological tool rather than a derived result, and the central questions (similarity and variety) are addressed via observed differences in the inferred categories. This is a standard empirical analysis pipeline with no load-bearing self-definition or renaming of known results.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review; no free parameters, axioms, or invented entities can be identified from provided text.

pith-pipeline@v0.9.1-grok · 5689 in / 857 out tokens · 22518 ms · 2026-06-26T10:42:12.901183+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 5 canonical work pages

  1. [1]

    InProceedings of the 51st Annual Meeting of the 8https://copyleaks.com/ai-content-detector Association for Computational Linguistics (Volume 1: Long Papers), pages 352–361

    Learning latent personas of film characters. InProceedings of the 51st Annual Meeting of the 8https://copyleaks.com/ai-content-detector Association for Computational Linguistics (Volume 1: Long Papers), pages 352–361. David Bamman, Sejal Popat, and Sheng Shen. 2019. An annotated dataset of literary entities. InProceedings of the 2019 Conference of the Nor...

    2019

  2. [2]

    machine: A comparative study on the detection of ai-generated content.ACM Transac- tions on Asian and Low-Resource Language Informa- tion Processing, 24(2):1–26

    Human vs. machine: A comparative study on the detection of ai-generated content.ACM Transac- tions on Asian and Low-Resource Language Informa- tion Processing, 24(2):1–26. Faeze Brahman and Snigdha Chaturvedi. 2020. Mod- eling protagonist emotions for emotion-aware story- telling. InProceedings of the 2020 Conference on Empirical Methods in Natural Langua...

    2020

  3. [3]

    let your characters tell their story

    “let your characters tell their story”: A dataset for character-centric narrative understanding. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 1734–1752, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics. Joseph Campbell. 2008.The hero with a thousand faces, volume 17. New World Library. Candic...

    work page arXiv 2021

  4. [4]

    InProceedings of the AAAI Conference on Artificial Intelligence, vol- ume 31

    Unsupervised learning of evolving relation- ships between literary characters. InProceedings of the AAAI Conference on Artificial Intelligence, vol- ume 31. Alicia Tsui Ying Chong, Hui Na Chua, Muhammed Basheer Jasser, and Richard T.K. Wong. 2023. Bot or human? detection of deepfake text with semantic, emoji, sentiment and linguistic features. In2023 IEEE...

    2023

  5. [5]

    Edward Morgan Forster

    What you read matters: The role of fiction genre in predicting interpersonal sensitivity.Psychol- ogy of aesthetics, creativity, and the arts, 7(4):370. Edward Morgan Forster. 1927.Aspects of the Novel. Harcourt, Brace. Sebastian Gehrmann, Hendrik Strobelt, and Alexan- der M Rush. 2019. Gltr: Statistical detection and visualization of generated text. InPr...

    1927

  6. [6]

    Spotting LLM s with binoculars: Zero-shot detection of machine-generated text

    Spotting llms with binoculars: Zero-shot detection of machine-generated text.Preprint, arXiv:2401.12070. Fouzi Harrag, Maria Dabbah, Kareem Darwish, and Ahmed Abdelali. 2020. Bert transformer model for detecting arabic gpt2 auto-generated tweets. InPro- ceedings of the Fifth Arabic Natural Language Pro- cessing Workshop, pages 207–214. Baruch Hochman. 198...

    work page arXiv 2020

  7. [7]

    Carol Pearson

    Panza: Design and analysis of a fully-local personalized text writing assistant.arXiv preprint arXiv:2407.10994. Carol Pearson. 1991. Awakening the heroes within: Twelve archetypes to help us find ourselves and trans- form our world.(No Title). James Phelan. 1989.Reading People, Reading Plots: Character, Progression, and the Interpretation of Nar- rative....

    work page arXiv 1991

  8. [8]

    InProceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Processing, pages 298–311

    Narrative theory for computational narrative understanding. InProceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Processing, pages 298–311. Vladimir Propp. 1968.Morphology of the Folktale, 2nd edition. University of Texas Press, Austin. Trans- lated by Laurence Scott, edited by Louis A. Wagner, with an introduction by Alan Dunde...

    2021

  9. [9]

    People who frequently use chatgpt for writing tasks are accurate and robust detectors of AI -generated text

    People who frequently use chatgpt for writing tasks are accurate and robust detectors of ai-generated text.arXiv preprint arXiv:2501.15654. Zhiyao Shu, Xiangguo Sun, and Hong Cheng. 2024. When llm meets hypergraph: A sociological analysis on personality via online social networks. InPro- ceedings of the 33rd ACM International Conference on Information and...

    work page arXiv 2024

  10. [10]

    You. . . heroes

    Evaluating character understanding of large language models via character profiling from fictional works. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8015–8036. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. J...

    2024

  11. [11]

    Naturalism {a_def} {b_def}

    Stylization vs. Naturalism {a_def} {b_def}

  12. [12]

    Incoherence {a_def} {b_def}

    Coherence vs. Incoherence {a_def} {b_def}

  13. [13]

    Fragmentariness {a_def} {b_def}

    Wholeness vs. Fragmentariness {a_def} {b_def}

  14. [14]

    Symbolism {a_def} {b_def}

    Literalness vs. Symbolism {a_def} {b_def}

  15. [15]

    Simplicity {a_def} {b_def}

    Complexity vs. Simplicity {a_def} {b_def}

  16. [16]

    Opacity {a_def} {b_def}

    Transparency vs. Opacity {a_def} {b_def}

  17. [17]

    Staticism {a_def} {b_def}

    Dynamism vs. Staticism {a_def} {b_def}

  18. [18]

    A" or "B

    Closure vs. Openness {a_def} {b_def} Classify the protagonist in the following narrative. Respond ONLY in valid JSON with exactly this format. Each category must be either "A" or "B" (not words): {“category1”: “A or B”, “category2”: “A or B”, “category3”: “A or B”, “category4”: “A or B”, “category5”: “A or B”, “category6”: “A or B”, “category7”: “A or B”,...

  19. [19]

    Each story is initially annotated by 2 in-person reviewers

    where they are originally described. Each story is initially annotated by 2 in-person reviewers. For any disagreement a third annotator chooses the final label. Table 2 provides the breakdown of Co- hen’s kappa inter-rater agreement and F1-macro scores. Note: Human annotations are very difficult, time- consuming and expensive. This is because annota- tors...

  20. [20]

    B, where A rep- resents the category, and B represents the op- posing category)

    Do LLMs prefer certain labels for certain traits?We count the number of times the LLM predicts a label (A vs. B, where A rep- resents the category, and B represents the op- posing category). Table 5 shows the break- down of the percentages of each label occur- rence. LLMs tend to under-classifystylized andclosedcharacters and over-classifyliteral characte...

  21. [21]

    gold labels Table 6

    Do LLMs and Human annotators share sim- ilar disagreements rates?We count the per- centage of disagreements between human an- notators and LLM predictions vs. gold labels Table 6. These columns have a moderately strong/strong positive correlation with Pear- son correlation (r)≈0.695 . The results indi- cate that characters who are difficult or sub- jectiv...

  22. [22]

    basic” ICL “dispersed

    Do classification errors occur because partic- ular stories/characters are tricky to analyze overall?Figure 7 shows how much a human annotator differs from the final gold label and the error rate of an LLM classifier across all 8 categories for each story. For a few sto- ries, LLMs make errors, while humans make no errors at all. There are also a few case...

    work page arXiv 2019

  23. [23]

    play it safe

    We observe that many of the distributions are similar to the distributions of the LLM-generated stories, indicating that when genres are specified, LLM-generated stories are biased toward gener- ating stories with particular categories of char- Source Family Source Name # Stories Avg Min Max Human r/WritingPrompts50 661.48±302.55 177 1739 r/shortstories20...