{"id":"1b412389-c3ef-4d21-a25f-00c319eceabe","arxiv_id":"2501.00273","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"The Sui Generis score quantifies plot-level uniqueness in LLM story generation and shows that GPT-4 and LLaMA-3 stories contain more repeated plot elements than human-written stories.","lead":"This paper introduces an automatic metric, the Sui Generis score, that measures how often a plot element in a story is echoed in alternative stories generated by the same LLM, and shows that LLM-written stories score lower than human-written ones. The metric correlates moderately with human surprise ratings, offering a practical tool for evaluating creative diversity in LLM outputs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The human-vs-LLM Sui Generis gap could be inflated by a source-dependent bias in GPT-4's plot-entailment judgments; the existing 20-pair human validation is unstratified and too weak to rule this out.","rationale":"The reader's weakest assumption correctly identifies the GPT-4 entailment function as the linchpin. I agree, and I sharpen the concern to a specific failure mode: source-dependent calibration bias. The headline comparison directly contrasts human-written segments (judged against LLM continuations) with LLM-generated segments (judged against same-model continuations), so any source-correlated bias in a(s,C) would artifactually create or inflate the very gap the paper claims. The cross-model experiment (LLaMA-3 stories scored with GPT-4 alternatives) only addresses self-generation bias, not source bias of the scorer, so it does not resolve this. The 20-pair validation in Section 6 is far too small, has fair inter-annotator agreement (kappa=0.33), and is not stratified by segment origin; it cannot detect a systematic offset. The paper's other evidence, such as the moderate correlation with human surprise (Spearman 0.55), is about whether the score tracks surprise within LLM-generated stories, not about whether the human-vs-LLM gap is real. I considered other weaknesses, including the unstated treatment of zero echo counts in Eq. 3 and the lack of significance tests on drop ratios, but those are either implementation details or secondary to the main claim. The source-bias concern is concrete, testable, and directly threatens the central result. The appropriate verdict remains CONDITIONAL: accept only if the authors provide a source-stratified validation of the entailment function or otherwise demonstrate that GPT-4's judgments are source-invariant.","tokens_in":12378,"tokens_out":5230,"duration_ms":53393,"concrete_test":"Conduct a stratified human validation: sample ~120 segment-continuation pairs, 60 with human-written source segments and 60 with LLM-written source segments, balanced across predicted echo scores (low/medium/high); have at least 5 trained annotators independently judge plot entailment with the same prompt template as Figure 2, then adjudicate disagreements. Compare GPT-4's judgment against the adjudicated human label separately for each source. If GPT-4's ROC AUC or agreement on human-source pairs is significantly lower than on LLM-source pairs, or if GPT-4 systematically says 'no' more often for human-source segments at equal human-judged entailment, the human-vs-LLM score gap in Figure 3 is partly an artifact of the scorer and the central claim needs qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 5, Figure 3) is that human-written story segments receive higher Sui Generis scores than LLM-generated segments. This comparison is built entirely on GPT-4's binary entailment function a(s,C) in Eq. 2, which decides whether a segment's plot appears in an alternative continuation. The score therefore measures 'echoed' relative to GPT-4's judgments, and those judgments are never validated separately for the two populations being compared. For LLM-generated stories, both the source segments and the alternative continuations are drawn from the same model family; for human stories, the source segments are human-written while the continuations are LLM-generated. If GPT-4's entailment decisions are influenced by surface-form or stylistic similarity rather than purely narrative equivalence, it will systematically under-detect echoes in human segments and over-detect them in LLM segments, directly inflating the headline gap. Section 6's human validation uses only 20 pairs with Fleiss kappa=0.33, giving no power to detect such a source-dependent calibration offset, and it does not stratify by segment origin or by score range. Section 7 acknowledges reliance on GPT-4's entailment capability but does not address source invariance. Thus the main quantitative claim is not yet shielded from a plausible scorer-bias artifact.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Sui Generis (SG) score, a fully automatic metric that measures the uniqueness of plot segments in a story by sampling alternative continuations from an LLM and using GPT-4 to judge whether each segment's plot is echoed in those continuations. On 100 stories from WritingPrompts and Wikipedia plot summaries, the authors report that LLM-generated stories (GPT-4 and LLaMA-3) have significantly lower SG scores than human-written stories, that LLM plot elements are echoed across different models, that LLM stories exhibit higher drop ratios (suggesting abrupt pacing), and that SG scores correlate moderately with human surprisal judgments. The metric is positioned as a narrative-level alternative to lexical or embedding-based diversity measures.","tokens_in":12634,"tokens_out":4799,"duration_ms":42817,"significance":"If the SG score is valid, it offers a practical and automatic way to quantify one aspect of LLM output diversity, with direct implications for human-AI co-creation and the evaluation of generative models. The paper has several strengths: the metric is operationally defined without fitted parameters, the experimental setup is described in sufficient detail to reproduce, the human studies were IRB-approved, and code is made available. The reported correlations with human surprisal and the paraphrase-robustness experiment are informative. However, the headline human-vs-LLM comparison rests entirely on GPT-4's binary entailment judgments, and the validation of that judgment function is too thin to rule out a systematic source-dependent bias. Because the central claim depends on this, the current evidence does not yet fully support the paper's strongest conclusion.","major_comments":[{"comment":"The GPT-4 entailment function a(s,C) is validated on only 20 pairs with a Fleiss kappa of 0.33, and the validation is not stratified by whether the source segment comes from a human-written or an LLM-generated story. Since the central comparison in Figure 3 contrasts human segments (scored against LLM continuations) with LLM segments (scored against LLM continuations), a systematic bias in GPT-4's judgments—such as higher sensitivity to echoes in LLM-style text or lower sensitivity to human-paraphrased echoes—would directly inflate the reported gap. The manuscript does not provide any test of source invariance. Please add a stratified human validation of a(s,C) separately for human and LLM source segments, or otherwise demonstrate that the entailment judgments are not biased by the origin of the segment.","section":"§3 Eq. (2) and §6 (Comparison with Other Similarity Metrics)"},{"comment":"The drop-ratio analysis uses the hyperparameters λ=0.9 and θ=0.5 without any sensitivity or robustness check. The reported 7–9 percentage-point differences between human and LLM stories in Table 3 are not accompanied by confidence intervals or a test of whether the ordering holds for a range of λ and θ. I would like to see a small sensitivity analysis (e.g., λ∈{0.8,0.9,0.95}, θ∈{0.3,0.5,0.7}) to confirm that the qualitative result is not an artifact of these particular thresholds.","section":"§5 (Table 3) and §4B"},{"comment":"The limitations section candidly notes that substituting a less capable model for entailment judgment could lead to bias, but it does not address the possibility that GPT-4 itself exhibits a source-dependent bias in this task. The 20-pair validation in Section 6 has low inter-annotator agreement (κ=0.33) and is too small to detect such a calibration offset. As a result, the main quantitative claim of the paper is not yet shielded from the plausible scorer-bias artifact described above.","section":"§7 (Limitations)"}],"minor_comments":[{"comment":"The phrase \"the first automatic metric\" is an overclaim given that other diversity metrics exist and are compared in Section 6; suggest softening to \"an automatic metric\".","section":"Abstract"},{"comment":"The GPT-4 model version is not specified (e.g., GPT-4-0613 or GPT-4-1106-preview); please state the exact model and API version for reproducibility.","section":"§4C"},{"comment":"The cross-model comparison uses only 20 stories per dataset; the text says \"Results on 20 Wiki and 20 WritingPrompts stories show...\" but the heatmap in Figure 5 is for 20 Wiki stories. Please report standard errors or confidence intervals for the cross-model differences.","section":"§5 (Cross-Model Scoring)"},{"comment":"The human surprise study uses 9 stories (159 segments) from LLaMA-3 only; the claim that SG correlates with human judgment would be strengthened by testing on GPT-4-generated and human-written stories as well.","section":"§5 (Sui Generis scores correlate with human judgment)"},{"comment":"The human judges were asked to give \"yes, no, or partially\" and the average was taken with \"partially\"=0.5; this coarse scale may compress variance. Please report agreement with a stricter criterion or use a finer scale.","section":"§6 (Comparison with Other Similarity Metrics)"},{"comment":"The heatmap uses triangular matrices for the lower and upper halves, but the colorbar and axis labels are not explained in the caption; please clarify the orientation of the two triangles and what the transpose represents.","section":"Figure 4"},{"comment":"The paraphrase generation process and manual verification are not described in the main text; please add a short description or a reference to the appendix.","section":"§5 (paraphrase robustness)"}],"recommendation":"major_revision","confidential_remarks":"The paper's contribution is timely and the metric is clearly defined, but the entailment validation is the weak link. In the revision, the authors should be asked to report the entailment model's performance stratified by source (human vs LLM) and to show sensitivity analyses for λ and θ. If these cannot be provided, the authors should temper the strength of the conclusions accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The Sui Generis score is a genuinely new way to measure narrative-level repetition in LLM outputs, and the human-vs-LLM gap it reports is likely real. The paper defines a metric that goes beyond lexical or embedding similarity, shows it correlates with human surprise (Spearman 0.55), and demonstrates a consistent gap across two datasets and two LLMs. The drop-ratio observation about pacing is a nice secondary result. The method is computationally explicit and code is promised on GitHub.\n\nThe soft spots are real but not fatal. The entailment function a(s,C) is the load-bearing component, and its human validation is thin: 20 pairs, Fleiss kappa 0.33, and no stratification by whether the source segment is human-written or LLM-generated. That matters because the comparison in Figure 3 mixes source types—human segments with LLM continuations versus LLM segments with LLM continuations. If GPT-4's entailment judgments are influenced by surface-form similarity, it could systematically under-detect echoes in human segments and over-detect them in LLM segments, inflating the headline gap. The Limitations section acknowledges that using a weaker model for both generation and judgment could cause bias, but it does not address source-dependence for GPT-4 itself. Also, Table 3's drop-ratio numbers come without error bars or significance tests, and the cross-model comparison uses only 20 stories per dataset. The paper says code is available but doesn't appear to link the data or human annotations.\n\nThese are addressable, not disqualifying. The core finding aligns with other work on LLM homogenization, and the metric's correlation with human surprise gives it some independent validity. But before I'd trust the quantitative gap, I'd want entailment validated on a stratified sample by source and score range, plus sensitivity analyses for K, lambda, and the GPT-4 scoring settings.\n\nThis deserves peer review—the idea and the result merit scrutiny. I'd send it out but ask for that strengthened validation before acceptance. If the authors close the source-bias gap, this becomes a useful evaluation tool for the LLM-diversity community.","headline":"A promising new metric for plot-level diversity, but the headline human-vs-LLM gap leans on an entailment judgment that hasn't been validated for source bias.","tokens_in":13203,"tokens_out":2039,"would_cite":false,"duration_ms":21116,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-generated stories reuse the same plot elements far more often than human-written ones, and a new automatic metric measures that echo quantitatively.","keywords":["Sui Generis score","plot diversity","LLM story generation","narrative uniqueness","collective creativity","echo score","surprise evaluation","text diversity metrics"],"falsifier":"Run the Sui Generis pipeline on a fresh set of prompts while replacing the GPT-4 entailment judge with human judgers on several hundred segments. If the average score gap between human- and LLM-written stories disappears or reverses, or if the automatic score's correlation with human surprise ratings becomes non-significant, the paper's central claim about echo-prone LLM plots would be falsified.","tokens_in":12157,"feed_emoji":"🔄","tokens_out":7884,"duration_ms":67298,"temperature":0.7,"pith_summary":"This paper argues that when a large language model is asked to continue the same story prompt many times, its continuations reuse the same narrative beats—plot elements that reappear across generations, often in different words and orders. To make that claim quantitative, the authors introduce the Sui Generis score, which measures how often a given story segment shows up, in spirit, in alternative continuations of the same prompt. Across 100 stories from two datasets, human-written segments score far higher (more unique) than segments produced by GPT-4 or LLaMA-3, and the effect persists when one model is used to score another model's output. A human study finds that the score's ranking of segments matches human surprise judgments moderately well (Spearman rho about 0.55), even though the score is computed automatically.","feed_headline":"AI stories echo the same plot beats again and again","feed_subtitle":"A new score rates each segment by how often it recurs in alternative AI continuations, and it tracks human surprise.","key_machinery":"The load-bearing machinery is the Sui Generis score $SG_i = -(\\sum_{j=1}^{i-1} \\lambda_j \\log p_{i,j}) /(\\sum_{j=1}^{i-1} \\lambda_j)$, where $\\lambda_j$ is an exponentially decaying weight controlled by a constant $\\lambda<1$ (the experiments use $\\lambda=0.9$). Here $p_{i,j}$ is an echo probability: over $K=20$ alternative continuations sampled after truncating the story at segment $j$, $p_{i,j}$ is the fraction in which a prompted GPT-4 entailment function $a(s_i, C_j^k)$ judges the plot of segment $i$ to be present. The earlier in the story an echo can be induced, the more heavily it is penalized, so a segment the LLM would spontaneously produce from a short prefix counts as less unique. This construction operates at the narrative level, independent of surface wording, and it also yields the drop ratio between consecutive segments that measures pacing.","core_discovery":"The central discovery is that LLM-generated stories are largely assembled from plot elements that the same model—and even a different LLM—would readily generate again under the same prompt, while human-written stories contain turns that fall almost entirely outside the LLM's distribution of possible continuations. The paper formalizes this via the Sui Generis score, defined for a segment $s_i$ as a prefix-weighted average of negative log probabilities that $s_i$'s plot appears in alternative continuations $C_j$ sampled from the model after truncating the story at earlier positions $j < i$. The metric is fully automatic, using a prompt-based entailment judgment by GPT-4 for whether a plot segment is echoed. It reveals not only lower average scores for LLM output, but also a 'drop ratio' reflecting the rushed, unresolved pacing noted in prior qualitative studies.","pith_inferences":["By analogy with the paper's narrative test, the same prefix-truncation-and-echo procedure could quantify diversity in other sequential creative domains—music, animation, dialogue—where an entailment or similarity function can be defined; the authors note this only in the conclusion.","The finding implies that an individual user who sees just one LLM output may overestimate its novelty; the 'echo' is only visible when sampling many alternatives. This could be tested directly by measuring whether users' novelty ratings drop after they view multiple generations.","A practical extension would be to use the Sui Generis score as an inference-time reward in a rejection-sampling loop with a smaller entailment model, checking whether the diversity gains found with GPT-4 persist when the judge is weaker and cheaper.","The moderate correlation with human surprise suggests the score could serve as a proxy for 'interestingness' in co-writing tools, flagging segments that deserve human attention."],"forward_implications":["If the claim holds, the Sui Generis score gives a cheap, automatic way to measure whether an LLM is producing genuinely diverse plots, without needing human raters or external corpora.","The drop-ratio result links the metric to narrative pacing: LLM stories show abrupt peaks followed by plunges in uniqueness, matching expert descriptions of rushed, unresolved endings.","Because the echoes are cross-model, simply switching from one LLM to another will not restore narrative diversity; the tendency appears shared.","Prompt choice matters: stories generated under more evocative prompts score higher for both humans and LLMs, so prompt engineering can partly counteract homogenization.","The score can be used as a generation-time filter, selecting the most unique among many sampled continuations, at the cost of additional LLM calls."],"supporting_citations":[{"why":"Provides the expert-writer judgments that LLM stories are less creative and rush their pacing; the paper's drop-ratio analysis extends that qualitative finding.","marker":"[7]"},{"why":"Reports that AI assistance raises individual creativity but lowers collective diversity, the qualitative effect this paper aims to quantify at the narrative level.","marker":"[10]"},{"why":"Studies content diversity when writing with LLMs and proposes diversity metrics; the paper contrasts its narrative-level score with those surface metrics.","marker":"[11]"},{"why":"Supplies the standardized diversity metrics (compression, self-BLEU, n-gram, homogenization, embedding) against which the entailment judgment is compared.","marker":"[18]"},{"why":"Introduces the WritingPrompts dataset used as one of the two testbeds for story generation.","marker":"[21]"},{"why":"GPT-4 technical report; the model is used both for story generation and as the entailment judge.","marker":"[22]"},{"why":"LLaMA-3 technical report; the second generation model used to test cross-model echoes.","marker":"[23]"},{"why":"Supports the choice of sampling temperature by showing that temperature has weak influence on LLM creativity.","marker":"[24]"}],"fun_headline_variants":["AI stories recycle plot beats, new score shows","Sui Generis score reveals LLMs' repetitive plots","LLM plots echo across generations, metric finds","GPT-4 and LLaMA-3 echo same story twists","Human plots stay unique, AI rehashes its own"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole score depends on GPT-4's prompt-based entailment judgment being accurate and unbiased for both human- and LLM-written segments, yet that judgment was validated on only 20 pairs with fair inter-annotator agreement (Fleiss kappa 0.33), and the same model family is used for generation and scoring.","fun_headline_variants_meta":{"raw":{"variants":["AI stories recycle plot beats, new score shows","Sui Generis score reveals LLMs' repetitive plots","LLM plots echo across generations, metric finds","GPT-4 and LLaMA-3 echo same story twists","Human plots stay unique, AI rehashes its own"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000149,"raw_usage":{"total_tokens":1177,"prompt_tokens":912,"completion_tokens":265,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":186}},"tokens_in":528,"tokens_out":265,"duration_ms":3177,"temperature":1.0,"reasoning_tokens":186,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:54:23.460229+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Sui Generis pipeline on a fresh set of prompts while replacing the GPT-4 entailment judge with human judgers on several hundred segments. If the average score gap between human- and LLM-written stories disappears or reverses, or if the automatic score's correlation with human surprise ratings becomes non-significant, the paper's central claim about echo-prone LLM plots would be falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the expert-writer judgments that LLM stories are less creative and rush their pacing; the paper's drop-ratio analysis extends that qualitative finding."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reports that AI assistance raises individual creativity but lowers collective diversity, the qualitative effect this paper aims to quantify at the narrative level."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Studies content diversity when writing with LLMs and proposes diversity metrics; the paper contrasts its narrative-level score with those surface metrics."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the standardized diversity metrics (compression, self-BLEU, n-gram, homogenization, embedding) against which the entailment judgment is compared."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the WritingPrompts dataset used as one of the two testbeds for story generation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GPT-4 technical report; the model is used both for story generation and as the entailment judge."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LLaMA-3 technical report; the second generation model used to test cross-model echoes."},{"cited_title":"(Association for Computing Machinery, New Y ork, NY , USA), p","cited_arxiv_id":null,"evidence_quote":"Supports the choice of sampling temperature by showing that temperature has weak influence on LLM creativity."}],"review_version":1}