{"id":"c4b5fcf6-f640-449f-821a-d5f6e9f37ebf","arxiv_id":"2505.23166","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Retell, a method that runs LDA on language models' abstractive retellings of literary passages, produces more theme-level topics than LDA alone or direct LM topic labeling.","lead":"This paper proposes a topic modeling method for literature: ask a small language model to retell each passage in abstract terms, then run LDA on those retellings. The authors find the resulting topics align better with human judgments and expert annotations than plain LDA or directly asking the model for topic labels, at a cost humanities researchers can afford.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No control for training-data overlap between LMs and the Goodreads/SparkNotes/LitCharts sources used to create gold labels; the measured advantage of Retell may reflect memorized study-guide themes rather than abstractive retelling.","rationale":"The reader's weakest assumption concerned the reliability of gold labels created by assuming preceding context matches the tag on a quote. That is a valid measurement-noise concern, but it applies symmetrically to all methods and does not directly implicate the mechanism behind Retell's advantage. A more load-bearing concern is that the evaluation's gold standard is drawn from the same public sources that likely shaped the LMs' outputs, so high relatedness may be an artifact of training-data memorization rather than evidence that abstractive retelling surfaces latent themes. The paper itself supplies evidence of book-level leakage in Section 6.2.2, making this more than a hypothetical. This concern directly threatens the central claim's interpretation and its generality to new or unseen literary corpora. The reader's verdict of CONDITIONAL remains appropriate: the paper is honest, reproducible in principle, and presents a useful method, but it should be conditioned on additional experiments that control for pretraining contamination. I therefore do not change the verdict, but I identify a different weakest assumption than the reader did.","tokens_in":26646,"tokens_out":8098,"duration_ms":90730,"concrete_test":"Construct a test set of book passages from texts published after the training cutoff of the open LMs (e.g., 2024-2025 novels or preprints) and obtain theme labels via independent human annotation, not from online study guides. Run the full Retell pipeline (same prompts, LMs, LDA settings) against default LDA and TopicGPT-lite on this held-out set, and compare crowd-rated relatedness to gold labels. If Retell's advantage over baselines shrinks or disappears, the central claim must be qualified as dependent on LMs' memorization of the specific books. As an auxiliary check, measure n-gram overlap between each LM retelling and the corresponding SparkNotes/LitCharts/Goodreads summary of the same book; if overlap strongly predicts the relatedness ratings, contamination is likely driving the results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that LDA on LMs' abstractive retellings yields more precise and informative topics than LDA on original text or directly eliciting topic labels. The evaluation in Section 5 relies on gold labels scraped from Goodreads, SparkNotes, and LitCharts (Section 4.1). These are popular public websites almost certainly included in the pretraining corpora of the tested LMs (Llama 3.1 8B, Phi-3.5-mini, Gemma 2 2B, GPT-4o mini). The LM may therefore reproduce thematic language and plot knowledge from the very study guides that define the evaluation's ground truth, inflating Retell's apparent quality without demonstrating genuine abstraction from the passage. The paper's own error analysis (Section 6.2.2) provides direct evidence of this leakage: retellings sometimes include book-level context not present in the passage (e.g., a character's Iranian background), which the authors attribute to the LM's pretraining familiarity with the book. If such memorization is common, Retell's advantage over baselines could stem from retrieving known summaries rather than 'telling what passages show.' This would not necessarily invalidate the method for well-known literary works, but it would undermine the claimed mechanism and its generality to unpublished, obscure, or post-cutoff texts, which are common in cultural analytics. The paper does not control for pretraining contamination in any experiment, leaving the central claim's real basis unverified.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Retell, a topic modeling pipeline that applies LDA to large language models' abstractive retellings of literary passages rather than to the original text. It compares Retell across four LMs and three retelling verbs against two baselines: LDA on the original passages and TopicGPT-lite, a simplified version of TopicGPT that directly elicits topic labels. Evaluation proceeds through crowdworker ratings of whether a method's most prominent topic is related to scraped literary-theme labels, a passage-level topic relevance/intrusion study, and a case study on racial/cultural identity passages in books taught in U.S. high schools. The paper reports that Retell outperforms both baselines on theme relatedness and identifies more race-relevant passages in the case study, and it releases code.","tokens_in":26924,"tokens_out":6847,"duration_ms":70655,"significance":"If the result holds, Retell offers a low-cost, easy-to-use method for humanities researchers who want to surface cross-cutting themes from literary text with smaller LMs. The paper is strong on transparency: it releases code, tests multiple models and retelling verbs, includes human annotation in two evaluation modes, and accompanies the case study with a detailed codebook and explicit limitations. Its main weakness is that the empirical evidence for the stated mechanism is undercut by the lack of any control for pretraining contamination from the very study-guide sources used as gold labels, and by the absence of uncertainty measures in the headline evaluation. The method is promising but the central claim, as currently worded, is broader than the evidence supports.","major_comments":[{"comment":"The evaluation's gold labels are scraped from Goodreads, SparkNotes, and LitCharts (§4.1), all of which are plausible components of the pretraining data for the tested LMs. The paper provides no control for this overlap, so the observed advantage of Retell over default LDA and TopicGPT-lite could arise from the LM reproducing memorized study-guide themes rather than from abstractive retelling of the passage. The paper's own error analysis in §6.2.2 documents exactly this mechanism: a retelling includes a character's Iranian background that is absent from the passage, which the authors attribute to the LM's familiarity with the book. This is load-bearing because the central claim is about the method's ability to translate surface form into higher-level concepts; without a control using post-cutoff or obscure texts, or a test that removes identifying book-level information, the mechanism and the method's generality to unpublished or niche works remain unverified.","section":"§4.1, §6.2.2"},{"comment":"The headline result in Table 2 (and the extended version in Table 8) is reported as rating proportions without any uncertainty quantification. No confidence intervals, standard errors, or significance tests are given for the comparisons across methods or across values of k. This makes the central claim that Retell consistently outperforms baselines difficult to evaluate, especially since some differences (e.g., certain Retell variants versus TopicGPT-lite) are numerically small. The authors should report bootstrap confidence intervals over the 60 passage sets and appropriate paired tests, and apply the same rigor to the precision/recall comparisons in Appendix C.1.","section":"Table 2, Table 8"},{"comment":"The abstract and Section 5 state the central claim without the resource-efficiency qualifier that the evidence actually supports. Appendix C.4 shows that with GPT-4o, TopicGPT-lite achieves similar crowd ratings to Retell-describe and higher precision and recall (Table 10). The conclusion's phrasing 'especially for resource-efficient, small LMs' is accurate, but the abstract and Section 5 should be revised so that the general claim is scoped to smaller LMs, or additional evidence should be provided for larger LMs.","section":"Abstract, §5, Appendix C.4"},{"comment":"The evaluation assumes that the paragraph context preceding a Goodreads, SparkNotes, or LitCharts quote does not deviate far from the tag or theme attached to the quote. This assumption is stated but not validated, despite a quote-matching rate of only 68.0 and the inclusion of up to 250 tokens of preceding context. If the surrounding passage differs thematically from the quote, the gold labels are noisy for every method, and all comparisons in Table 2 inherit that noise. The authors should either validate the assumption on a sample (e.g., human ratings of whether the full passage matches the quote's theme) or restrict evaluation to passages where the quote is representative of the whole window. The manual recoding of scraped tags/themes into 27 topics also lacks inter-annotator reliability, which is important because this recoding defines the gold set.","section":"§4.1"}],"minor_comments":[{"comment":"The column structure of Table 2 is difficult to parse; the 'Rating /k' row lists many numbers without visually separating model/k combinations, and the repeated '✓', '?', and '✗' rows make it hard to map ratings to methods. A grouped layout or a separate table per model would improve readability.","section":"Table 2"},{"comment":"The note that ratings 'differ significantly for each method' is ambiguous; please specify which pairwise comparisons are significant (e.g., Topic 1 versus Intruder) and whether any multiple-comparison correction was applied.","section":"Table 3"},{"comment":"The phrase 'different instructive verbs' appears to be a slip for 'retelling verbs' or 'instruction verbs'; the current wording is imprecise.","section":"§5"},{"comment":"The absolute precision values in Table 9 are low (e.g., 0.128 for Retell-summarize at k=50); the authors should report the base rate of positive passage pairs so that readers can judge the effect size, and should comment on whether precision at these levels is practically useful for exploratory literary analysis.","section":"Appendix C.1, Table 9"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's empirical CS/cultural-analytics niche and is written with commendable transparency. The most serious issue is the lack of any control for pretraining contamination from the study-guide sources used as gold labels; a single control experiment using contemporary or obscure texts, or a test that masks book-level identifiers, would substantially raise confidence in the central claim. The abstract's overgeneralization relative to the Appendix C.4 results should also be corrected before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Retell is a clean, useful piece of work: it shows that running LDA on LM retellings of literary passages gives more useful topics than running LDA on the original text or asking an LM to list topics directly, and it does this with small, cheap models. That is a real contribution for cultural analytics and digital humanities, where researchers often can't afford GPT-4-class APIs. The evaluation is more thorough than most: crowd ratings of topic relatedness against scraped study-guide labels, a passage-level intrusion test with human annotators, and a case study on race in ELA books with expert-annotated passages. The paper is also unusually honest about limitations, and it ships code.\n\nThe biggest soft spot is the one the stress-test note flags: the gold labels come from Goodreads, SparkNotes, and LitCharts, and the tested LMs were almost certainly trained on those sites. The paper's own error analysis shows LMs injecting book-level knowledge into retellings (the Iranian-background example). That means part of Retell's advantage over LDA could come from the LM regurgitating themes it has memorized from study guides, not from abstractive retelling of the passage. The paper doesn't test for this. It's not fatal—the passage-level human evaluation and the case study are less contaminated—but it does mean the central mechanistic claim is weaker than the title suggests, and the method's generality to obscure or post-cutoff texts is unverified.\n\nTwo smaller issues: Table 2 has no confidence intervals or significance tests, just proportions over 60 passage sets, so some of the differences may be noise. And TopicGPT-lite is a modified version of TopicGPT, so the comparison is not against the original method. Neither is a dealbreaker, but they should be noted.\n\nIn summary, this is a solid empirical paper with a useful contribution. It deserves serious peer review, but reviewers should push for a contamination analysis (e.g., hand-check retellings for verbatim overlap with study guides, or test on newly written or unpublished texts) and for basic inferential statistics. I'd bring it to our reading group and would cite it if I were working on literary topic modeling.","headline":"Useful, honest method paper on LM retellings for literary topic modeling, but the study-guide contamination worry is real and should be addressed.","tokens_in":27471,"tokens_out":2791,"would_cite":true,"duration_ms":28894,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Running LDA on language models' abstractive retellings of literary passages yields more precise and informative topics than running LDA on the original text or asking the models to list topics directly.","keywords":["topic modeling","abstractive retelling","literary analysis","language models","latent Dirichlet allocation","cultural analytics","racial identity","distant reading"],"falsifier":"Build a labeled dataset in which each passage's surrounding paragraphs are manually checked to share the theme of the quote's tag, then rerun the comparison of Retell, default LDA, and TopicGPT-lite; if Retell's advantage shrinks or disappears on this clean set, the reported gains are an artifact of label-context alignment.","tokens_in":26445,"feed_emoji":"📚","tokens_out":6894,"duration_ms":62078,"temperature":0.7,"pith_summary":"Literary passages 'show' rather than 'tell': they express themes through sensory details, dialogue, and action, so bag-of-words topic models like LDA see surface words ('black', 'hair', 'dress') instead of the underlying ideas ('race', 'identity', 'representation'). This paper proposes Retell, which prompts a small, resource-efficient language model to retell each passage in abstractive terms, then runs LDA on those retellings. Across reader-tagged and study-guide-themed passages, Retell's most prominent topics are rated more related to the gold themes than topics from LDA alone or from directly asking language models for topic labels. In a case study on racial and cultural identity in high-school English books, Retell's topics align with expert annotations and recover more passages of interest than the baselines, suggesting a low-cost path for cultural analytics at scale.","feed_headline":"AI retellings beat plain LDA at finding literary themes","feed_subtitle":"A one-prompt method translates a novel's sensory prose into abstract concepts, giving scholars a low-cost way to surface themes.","key_machinery":"The machinery is abstractive retelling: a single prompt tells a language model to 'describe', 'summarize', or 'paraphrase' a 250-token book passage 'for a literary scholar analyzing narrative content,' and LDA is fit to the resulting one-paragraph retellings after light preprocessing (lowercasing, removing character names, filtering frequent and rare words). The prompt turns 'showing' into 'telling'—low-level sensory detail into high-level conceptual exposition—which is exactly the representation LDA's bag-of-words assumptions need. The verb choice is the control: describe and summarize encourage abstraction and outperform paraphrase, which stays closer to the surface form.","core_discovery":"The paper's central claim is that the bottleneck in topic modeling literature is not the topic model but the representation it sees: narrative prose encodes themes implicitly, and LDA's bag-of-words assumptions fail on that surface form. Retell removes the bottleneck by having an instruction-tuned language model 'tell' what a passage 'shows'—with describe, summarize, or paraphrase prompts—and then fitting LDA to those retellings. Across four small LMs, Retell's most prominent topics are judged by crowdworkers to be 'very related' to gold themes far more often than default LDA's topics are (averaging 0.59–0.60 across models versus much lower baselines), and more informative than TopicGPT-lite's often over-broad labels such as 'human nature' or 'life'. In a case study on racial/cultural identity in U.S. high-school English books, the probability of Retell's race-related topics rises for passages that human annotators coded as mentioning or discussing race, and combining two relevant topics recovers more such passages than default LDA's single topic without losing precision.","pith_inferences":["If the gain comes from abstraction, Retell could be combined with other lexical or embedding methods (e.g., clustering or keyword extraction) beyond LDA, though the paper only tests LDA.","The error analysis hints that retellings can leak book-level knowledge from pretraining, so Retell may work better on canonical or widely discussed books than on obscure ones; a comparison across familiarity would test this.","The method's premise suggests it will help most in genres dense with sensory detail (literary fiction) and less in expository prose, where paraphrase adds little abstraction; a genre-by-genre evaluation would quantify this.","For cultural analytics, Retell's topic probabilities could serve as a preliminary screening filter to direct close reading toward candidate passages, reducing the 'needle-in-haystack' annotation burden the authors describe."],"forward_implications":["Scholars working with literary corpora can surface cross-cutting themes with a single LM prompt followed by LDA, without large API budgets.","Retell topics are more interpretable because they use abstractive terms (e.g., 'financial', 'racial') rather than surface words (e.g., 'money', 'black'), making them easier to relate to human notions of theme.","The method can screen collections for passages of interest: in the race case study, combining two relevant topics recalled more expert-annotated passages than default LDA without sacrificing precision.","Because topic granularity k is set after retellings are produced, researchers can adjust k without rerunning the LM.","Direct LM topic labeling with small models tends to collapse into a few overly broad labels; Retell avoids this by keeping LDA's distributional structure."],"supporting_citations":[{"why":"Supplies the LDA model that Retell runs on.","marker":"Blei et al., 2003"},{"why":"The TopicGPT framework that Retell is compared against and adapted into TopicGPT-lite.","marker":"Pham et al., 2024"},{"why":"Provides the topic-intrusion evaluation design used for passage-level human ratings.","marker":"Chang et al., 2009"},{"why":"Supplies the LDA preprocessing recipe and authorless-modeling concerns.","marker":"Thompson and Mimno, 2018"},{"why":"Motivates the need for stable, interpretable topic models and frames the stability discussion.","marker":"Hoyle et al., 2022"},{"why":"Provides the high-school ELA book list and racial-representation dataset for the case study.","marker":"Lucy et al., 2025"},{"why":"Supplies the definition of racial/ethnic categories used to build the case-study annotation codebook.","marker":"Algee-Hewitt et al., 2020"},{"why":"Provides the teacher-reported book lists that make up part of the ELA corpus.","marker":"Levine et al., 2021"}],"fun_headline_variants":["Retell: LM retellings sharpen topic modeling for novels","Turning prose into concepts: better literary topics via LM retellings","Show vs tell: LM retellings improve theme detection in books","Literary themes via LM retellings outperform bag-of-words"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the paragraph context preceding a reader-tagged or study-guide-tagged quote is about the same theme as the tag attached to the quote; if the surrounding passage is about something else, the gold labels used to score every method are unreliable and all comparisons inherit that noise.","fun_headline_variants_meta":{"raw":{"variants":["Retell: LM retellings sharpen topic modeling for novels","Turning prose into concepts: better literary topics via LM retellings","Show vs tell: LM retellings improve theme detection in books","Literary themes via LM retellings outperform bag-of-words"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000208,"raw_usage":{"total_tokens":1396,"prompt_tokens":931,"completion_tokens":465,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":403}},"tokens_in":547,"tokens_out":465,"duration_ms":4460,"temperature":1.0,"reasoning_tokens":403,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:51:27.376089+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a labeled dataset in which each passage's surrounding paragraphs are manually checked to share the theme of the quote's tag, then rerun the comparison of Retell, default LDA, and TopicGPT-lite; if Retell's advantage shrinks or disappears on this clean set, the reported gains are an artifact of label-context alignment.","supporting_citations":[],"review_version":1}