Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

CreativityPrism: A Cross-Domain Evaluation Framework for Large Language Model Creativity

T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that LLM creativity is fragmented: a model's strong performance in one creative dimension or domain rarely predicts strong performance in another, so a single score cannot measure machine creativity.

desk verdict A useful benchmark assembly with a load-bearing fragmentation claim that the current numbers don't yet support. read the letter →

arxiv 2510.20091 v3 pith:QNEF4GVH submitted 2025-10-23 cs.CL cs.AI

Zhaoyi Joey Hou , Bowei Alvin Zhang , Yining Lu , Bhiman Kumar Baghel , Anneliese Brei , Ximing Lu , Meng Jiang , Faeze Brahman
show 4 more authors
Snigdha Chaturvedi Haw-Shiuan Chang Daniel Khashabi Xiang Lorraine Li
This is my paper · ORCID
classification cs.CLcs.AI
keywords creativityevaluationlargelanguagemodelsbenchmarkdivergentthinkingcreativewritinglogicalreasoningnoveltydiversity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to build a scalable, cross-domain way to measure creativity in large language models. It proposes a framework with nine tasks spanning divergent thinking, creative writing, and logical reasoning, and scores each model on three dimensions—quality, novelty, and diversity. Evaluated on 17 models, the framework shows that proprietary models lead open models by about 10–15% overall but not in divergent thinking. The central finding is that creativity does not transfer: quality and diversity correlate with each other, while novelty correlates weakly or even negatively (e.g., story surprise vs. coding divergence at r = -0.25). The authors conclude that meaningful assessment of LLM creativity requires multiple tasks, domains, and dimensions rather than any single aggregate.

What carries the argument

CreativityPrism, a benchmark and analysis suite. It organizes nine existing tasks (e.g., Alternative Uses Test, creative short-story generation, constrained code generation, creative math) into three domains and classifies twenty task-specific metrics into three dimensions: quality (does the output satisfy task requirements), novelty (is it rare compared to existing content), and diversity (how varied are multiple outputs). The analysis machinery is a correlation matrix over model performance vectors: for each metric, the normalized scores of all 17 models are stacked into a vector, and Pearson correlations between metric pairs reveal which dimensions travel together. That matrix is the load

What would settle it

Collect human creativity ratings on a sample of outputs from all nine tasks (especially the one with no human annotations at all), replace the AI-judge scores with human scores, and recompute the cross-metric correlation matrix; if the weak and negative novelty correlations disappear, the fragmentation claim would collapse.

Watch

Extended reading notes

Core claim

The central claim, stated on the paper's own terms, is that creativity in LLMs is multidimensional and fragmented. High performance in one creative dimension or domain rarely generalizes to others; in particular, novelty metrics often show weak or negative correlations with other metrics, because 'novelty' means different things in different tasks—being surprising in a short story is not the same as solving a coding problem in an unprecedented way. From this, the paper argues that a holistic benchmark spanning three domains and three creativity dimensions is necessary, and that an 'overall' creativity score should be treated only as a coarse comparison aid, not as a measure of a single under

Load-bearing premise

The rankings and the fragmentation conclusion rest on the assumption that the AI judge's creativity scores align with human judgments of creativity, an alignment that is only thinly validated for some of the nine tasks.

Editorial extensions

If this is right

  • A single-task or single-domain evaluation of LLM creativity will misreport model capability, because strong performance in one domain does not predict performance in another.
  • Novelty should be measured and reported per task, since novelty metrics across tasks share little variance and can even be negatively correlated.
  • Models strong on quality tend to be strong on diversity, so diversity can serve as a rough proxy for quality, but neither predicts novelty.
  • Frontier proprietary models lead in creative writing and logical reasoning by roughly 0.10–0.15 normalized points, yet show no clear advantage in divergent thinking, suggesting those domains are undertrained.
  • The benchmark's three-dimension scores—quality, novelty, diversity—should be reported separately; the overall score is only a coarse summary.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: because the open–proprietary gap is smallest in divergent thinking, post-training with divergent-thinking tasks might be where open models can catch up most cheaply.
  • The weak novelty dimension suggests that 'novelty' is not one capability but a family of task-specific behaviors; a model could be trained to inflate one novelty metric without gaining another.
  • If human ratings later show the AI judges are biased, the specific rankings would shift, but the fragmentation structure might survive only if the bias is uniform across tasks; checking this with per-task human data is a natural next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces CreativityPrism, a benchmark and analysis framework for evaluating LLM creativity. It consolidates nine tasks in three domains (divergent thinking, creative writing, logical reasoning) and organizes twenty metrics into three dimensions (quality, novelty, diversity). The authors evaluate 17 proprietary and open-weight LLMs, aggregate scores by dimension and domain, and report a correlation analysis across metrics. The central claims are that proprietary models outperform open models in creative writing and logical reasoning but not in divergent thinking, and that creativity is fragmented: high performance on one dimension or domain does not reliably transfer, especially for novelty metrics, which show weak or negative correlations with other metrics. The paper also validates automatic LLM judges for six of the nine tasks, with varying levels of supporting evidence.

Significance. The benchmark addresses a real and timely gap: most LLM creativity evaluations are task-specific, single-domain, or rely on expensive human evaluation. CreativityPrism is useful as a structured consolidation of existing tasks and metrics, and the three-dimensional taxonomy is a sensible organizing principle. The paper also makes an explicit effort to check LLM-as-a-judge reliability, which is more than many benchmark papers do. If the results hold, the fragmentation claim would be an important caution against reporting a single creativity score. However, the two main supports for that claim — the LLM-judge validation and the cross-metric correlation analysis — are currently not strong enough to carry the conclusions, so the significance is conditional on the revisions described below.

major comments (3)
  1. [§5.3, Figures 3–4] The central fragmentation conclusion is a statistical claim about correlations among model performance vectors, but the paper provides no uncertainty quantification. For M=17 models, the standard error of a correlation under the null is about 1/sqrt(17−3) ≈ 0.27, so the headline example (Creative Short Story 'Surprise' vs NeoCoder 'Divergence@0', r=−0.25) has a 95% interval of roughly (−0.66, +0.26). That interval includes zero, and many other correlations in the 17×17 matrix are likely consistent with zero after accounting for multiple comparisons. In addition, several rows are from the same model family (Qwen, DeepSeek, Claude, GPT), so the effective independent sample size is smaller than 17. The paper should report confidence intervals or significance tests (ideally with a multiple-comparison adjustment) before claiming that novelty metrics are weakly or negatively correlated across
  2. [§4.2, Appendix D, Appendix E.2.5, E.8] The LLM-as-a-judge validation is too thin to support the ranking and correlation structure. For TTCT the paper states 'we have no human annotations at all' and instead reports Pearson correlations between Qwen2.5-72B and GPT-4.1 (0.50–0.69). That measures agreement between two LLM judges, not validity against human judgment, and the lower values are not obviously acceptable for a metric used to rank 17 models. For CS4, the human–LLM correlation is 0.55 on 15 stories; this is a small and modest alignment. For TTCW, the appendix (E.1.4) mentions a correlation threshold of 0.2 while the main text (D) reports accuracy values; the two descriptions are not consistent. These issues affect six of nine tasks, so they are load-bearing for the benchmark's conclusions; the manuscript's own acknowledgement that more human annotation is being collected confirms the validation is incomplete.
  3. [Abstract vs. §5.2 and Table 2] The abstract states that frontier LLMs 'offer no significant advantage in divergent thinking,' but §5.2 claims 'more than 10% in each domain' and Table 2 shows Claude3-Sonnet at 0.833 in divergent thinking versus the best open-weight model (Qwen2.5-72B) at 0.731, a gap of about 0.10 (roughly 14% relative). If 'no significant advantage' is meant statistically, no significance test is reported; if it is meant practically, the table does not support it. The claim should be reconciled with the reported numbers and, if retained, supported by an appropriate test.
minor comments (5)
  1. [Abstract] The phrase 'by a .10 (or 15%) lead' is unclear and appears garbled. Please use consistent absolute or relative numbers and specify which models are being compared.
  2. [Table 3 / Appendix E] Table 3 lists 19 models (including OLMo2-13B-SFT and OLMo2-13B-DPO), while Table 2 reports 17 models, and Appendix E.1.5/E.9.5 include Claude3-Opus, which does not appear in Table 2. The model inventory should be harmonized so that every reported result has a corresponding model entry.
  3. [Table 3] DeepSeek-R1 and DeepSeek-V3 are labeled as 'Gemini' in the Family column; this is a typo.
  4. [§5.2] The 'more than 20% in each dimension' and 'more than 10% in each domain' claims should state whether they are absolute or relative gaps and give the exact best-model comparisons from Table 2.
  5. [E.4.6] The table header in the Creative Short Story results uses 'Surprisal' and 'N-gram Diversity,' while the text refers to a 'novelty score'; please unify the metric names.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's conclusions are empirical measurements from a constructed benchmark, not derivations that assume their own outcomes.

full rationale

The paper is an empirical benchmark construction, not a formal derivation. Its central claims—that proprietary models lead on writing/reasoning but not divergent thinking, and that novelty metrics correlate weakly across domains—are computed from measured model scores and correlation matrices, not entailed by any definition or fitted parameter. The taxonomy (quality/novelty/diversity) is a conceptual grouping of existing metrics; it does not by construction force low cross-metric correlations, and the empirical finding could in principle have been the opposite. The self-cited tasks (NeoCoder, Creativity Index) come from prior published work with independent human validation and code, so they serve as external evidence rather than a self-referential load-bearing chain. The TTCW metric-selection step (keeping only metrics with human–judge correlation > 0.2) is a post-hoc reliability filter, but it selects evaluation questions based on human agreement, not based on the paper's target conclusion, so it is not circular. The TTCT judge validation via correlation with another LLM judge, and the small human sample for CS4, raise validity and uncertainty concerns, but under-determination or statistical fragility is not circularity. The highlighted r=-0.25 lacks confidence intervals, but that is a robustness/statistical reporting issue, not a reduction of a prediction to an input. No equation or fitted parameter is renamed as a prediction, and no uniqueness claim is imported from the authors' prior work. The benchmark's conclusions are self-contained empirical measurements, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The benchmark rests on domain assumptions about what creativity is and which tasks measure it, plus hand-set thresholds for accepting LLM-judge agreement. No physical or mathematical free parameters are fitted; the free parameters listed are design choices that affect the results.

free parameters (3)
  • TTCW metric inclusion threshold = correlation > 0.2
    E.1.4: only TTCW questions where the LLM-judge achieves correlation > 0.2 with human votes are included; this post-hoc threshold shapes the TTCW quality scores.
  • LLM-judge agreement acceptance level = lowered for story creativity
    Appendix D: 'we have a lower acceptance threshold when we are considering what kind of human-LLM agreement level is acceptable'; this hand-set tolerance legitimizes weak validations.
  • Task selection criterion = 9 tasks with automatic metrics
    §3: tasks selected 'primarily based on the availability of automatic evaluation metrics'; this excludes tasks that require human judgment and biases the benchmark toward measurable dimensions.
assumptions (3)
  • domain assumption Creativity decomposes into quality, novelty, and diversity
    §3 states this as the organizing taxonomy; it is borrowed from psychology but is a modeling choice the entire benchmark rests on.
  • domain assumption LLM-as-a-judge judgments are valid proxies for human creativity judgments
    Assumed in §4.2 for six of nine tasks; for TTCT there is no human annotation and only cross-LLM correlation is used.
  • standard math Known min/max bounds make min-max normalization across metrics meaningful
    Used in §4.2 and B.3; assumes equal scaling of Likert, ratios, and n-gram proportions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CreativityPrism: A Cross-Domain Evaluation Framework for Large Language Model Creativity." pith.science (2026). https://pith.science/paper/QNEF4GVH

@misc{pith2026251020091,
  author       = {Pith},
  title        = {Pith review of: CreativityPrism: A Cross-Domain Evaluation Framework for Large Language Model Creativity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QNEF4GVH}},
  note         = {Machine review of arXiv:2510.20091}
}
read the original abstract

Creativity is often seen as a hallmark of human intelligence. While large language models(LLMs) are increasingly perceived as generating creative text, there is still no cross-domain and scalable framework to evaluate their creativity across diverse scenarios. Existing methods of LLM creativity evaluation either heavily rely on humans, limiting speed and scalability, or are fragmented across different domains and different definitions of creativity. To address this gap, we propose CreativityPrism, an evaluation and analysis framework that consolidates eight tasks from three domains: divergent thinking, creative writing, and logical reasoning, into a taxonomy of creativity that emphasizes three dimensions: quality, novelty, and diversity of LLM generations. The framework is designed to be scalable with reliable automatic evaluation judges that have been validated against human annotations. We evaluate 17 state-of-the-art (SoTA) LLMs on CreativityPrism and find that while frontier-scale LLMs dominate creative writing and logical reasoning tasks by a .10 (or 15%) lead over locally-deployable open models, they offer no significant advantage in divergent thinking, a domain much less explored in existing post-training regimes. Our analysis also shows that high performance in one creative dimension or domain rarely generalizes to others; specifically, novelty metrics often show weak or negative correlations with other metrics. This fragmentation confirms that a cross-domain, multi-dimensional framework like CreativityPrism is essential for any meaningful assessment of LLM creativity.

Figures

Figures reproduced from arXiv: 2510.20091 by the authors.

Figure 1
Figure 1. Overview of CREATIVITYPRISM. We evaluated 17 LLMs across nine datasets spanning three domains: divergent thinking, creative writing, and logical reasoning. Each LLM was prompted to complete the tasks, and their outputs were evaluated using task-specific metrics. However, these metrics are diverse and difficult to interpret holistically in terms of machine creativity. To address this, we organize the metrics into thr… view at source ↗
Figure 2
Figure 2. (a) Performance v.s. Day since LLM release date. The line represents best fit linear [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Models’ performance correlations, grouped by task and domain; C-Index refers to Creativity [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Left: models’ performance correlations, grouped by creativity dimensions; Right: individual [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Overall performances. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]
Figure 6
Figure 6. Figure 6: Inter-metric correlation (grouped by creativity taxonomy). [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]
Figure 7
Figure 7. Figure 7: Performance on quality dimension [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: Performance on novelty dimension [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: Performance on diversity dimension 22 [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Performance on creative writing tasks [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: Performance on divergent thinking tasks [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Performance on logical reasoning dimension [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Where Models Converge and Humans Diverge: A Coverage Framework for Distributional Pluralism in Open-Ended Generation

    cs.CL 2026-08 conditional novelty 6.0 of 10

    LLM outputs are usually plausible but cover far less of the human response space than matched human samples do, with the biggest gap at the periphery of the human distribution.

Reference graph

Works this paper leans on

124 extracted references · 1 canonical work pages · cited by 1 Pith paper

  1. [1]

    What do educators need to know about the torrance tests of creative thinking: A comprehensive review

    Ahmed M Abdulla Alabbasi, Sue Hyeon Paek, Daehyun Kim, and Bonnie Cramond. What do educators need to know about the torrance tests of creative thinking: A comprehensive review. Front. Psychol., 13:1000385, October 2022

    2022

  2. [2]

    Claude 3 model family, 2024

    Anthropic. Claude 3 model family, 2024. URL https://www.anthropic.com/news/ claude-3-family. Accessed: 2025-04-30. 10

    2024

  3. [3]

    CS4: Measuring the creativity of large language models automatically by controlling the number of story-writing constraints.arXiv [cs.CL], October 2024

    Anirudh Atmakuru, Jatin Nainani, Rohith Siddhartha Reddy Bheemreddy, Anirudh Lakkaraju, Zonghai Yao, Hamed Zamani, and Haw-Shiuan Chang. CS4: Measuring the creativity of large language models automatically by controlling the number of story-writing constraints.arXiv [cs.CL], October 2024

    2024

  4. [4]

    Divergent creativity in humans and large language models.arXiv [cs.CL], May 2024

    Antoine Bellemare-Pepin, François Lespinasse, Philipp Thölke, Yann Harel, Kory Mathewson, Jay A Olson, Yoshua Bengio, and Karim Jerbi. Divergent creativity in humans and large language models.arXiv [cs.CL], May 2024

    2024

  5. [5]

    The MIT Press, June 1994

    Margaret A Boden, editor.Dimensions of Creativity. The MIT Press, June 1994

    1994

  6. [6]

    The crowdless future? generative ai and creative problem-solving.Organization Science, 35(5):1589–1607, 2024

    Léonard Boussioux, Jacqueline N Lane, Miaomiao Zhang, Vladimir Jacimovic, and Karim R Lakhani. The crowdless future? generative ai and creative problem-solving.Organization Science, 35(5):1589–1607, 2024

    2024

  7. [7]

    Evaluation of text generation: A survey

    Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. Evaluation of text generation: A survey. arXiv [cs.CL], June 2020

    2020

  8. [8]

    Art or artifice? large language models and the false promise of creativity

    Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien-Sheng Wu. Art or artifice? large language models and the false promise of creativity. InProceedings of the CHI Conference on Human Factors in Computing Systems, volume 70, pages 1–34, New York, NY , USA, May 2024. ACM

    2024

Show all 124 references
  1. [9]

    Creativity support in the age of large language models: An empirical study involving professional writers

    Tuhin Chakrabarty, Vishakh Padmakumar, Faeze Brahman, and Smaranda Muresan. Creativity support in the age of large language models: An empirical study involving professional writers. InCreativity and Cognition, New York, NY , USA, June 2024. ACM

    2024

  2. [10]

    creativity

    Honghua Chen and Nai Ding. Probing the “creativity” of large language models: Can models produce divergent semantic association? InThe 2023 Conference on Empirical Methods in Natural Language Processing, December 2023

    2023

  3. [11]

    DeepSeek-V3 technical report

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei L...

    2024

  4. [12]

    DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.arXiv [cs.CL], January 2025

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z F Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Fen...

    2025

  5. [13]

    The definition and measurement of creativity: What do we know?J

    Jaafar El-Murad and Douglas C West. The definition and measurement of creativity: What do we know?J. Advert. Res., 44(02):188–201, June 2004

    2004

  6. [14]

    Creation-MMBench: Assessing context-aware creative intelligence in MLLM.arXiv [cs.CV], March 2025

    Xinyu Fang, Zhijian Chen, Kai Lan, Lixin Ma, Shengyuan Ding, Yingji Liang, Xiangyu Zhao, Farong Wen, Zicheng Zhang, Guofeng Zhang, Haodong Duan, Kai Chen, and Dahua Lin. Creation-MMBench: Assessing context-aware creative intelligence in MLLM.arXiv [cs.CV], March 2025

    2025

  7. [15]

    The MIT Press, October 1992

    Ronald A Finke, Thomas B Ward, and Steven M Smith.Creative Cognition: Theory, research, and applications. The MIT Press, October 1992

    1992

  8. [16]

    Creativity and machine learning: A survey.ACM Comput

    Giorgio Franceschelli and Mirco Musolesi. Creativity and machine learning: A survey.ACM Comput. Surv., 56(11), June 2024. ISSN 0360-0300. doi: 10.1145/3664595. URL https: //doi.org/10.1145/3664595

    2024 doi

  9. [17]

    Pushing GPT’s creativity to its limits: Alternative uses and torrance tests

    Fabricio Goes, Marco V olpe, Piotr Sawicki, Marek Grzes, and Jacob Watson. Pushing GPT’s creativity to its limits: Alternative uses and torrance tests. In14th International Conference on Computational Creativity 2023, 2023

    2023

  10. [18]

    Gemini 1.5 and 2.0: Next-gen multimodal models, 2024

    Google DeepMind. Gemini 1.5 and 2.0: Next-gen multimodal models, 2024. URL https: //deepmind.google/technologies/gemini/. Accessed: 2025-04-30

    2024

  11. [19]

    The llama 3 herd of models.arXiv [cs.AI], July 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ah- mad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, A...

    2024

  12. [20]

    OLMo: Accelerating the science of language models

    Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tusha...

    2024

  13. [21]

    Alternate uses, June 2012

    J P Guilford, Paul R Christensen, Philip R Merrifield, and Robert C Wilson. Alternate uses, June 2012. Title of the publication associated with this dataset: PsycTESTS Dataset

    2012

  14. [22]

    A confederacy of models: A comprehensive evaluation of LLMs on creative writing

    Carlos Gómez-Rodríguez and Paul Williams. A confederacy of models: A comprehensive evaluation of LLMs on creative writing. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 14504–14528, Stroudsburg, PA, USA, December 2023. Association for Computati...

    2023

  15. [23]

    Pun generation with surprise

    He He, Nanyun Peng, and Percy Liang. Pun generation with surprise. In Jill Burstein, Christy Doran, and Thamar Solorio, editors,Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum...

    2019

  16. [24]

    What shapes a creative machine mind? comprehensively benchmarking creativity in foundation models.arXiv [cs.AI], October 2025

    Zicong He, Boxuan Zhang, Weihao Liu, Ruixiang Tang, and Lu Cheng. What shapes a creative machine mind? comprehensively benchmarking creativity in foundation models.arXiv [cs.AI], October 2025

    2025

  17. [25]

    Holyoak and R.G

    K.J. Holyoak and R.G. Morrison.The Cambridge Handbook of Thinking and Reasoning. Cam- bridge Handbooks in Psychology. Cambridge University Press, 2005. ISBN 9780521824170. URLhttps://books.google.com/books?id=znbkHaC8QeMC. 14

    2005

  18. [26]

    A scientific creativity test for secondary school students.Int

    Weiping Hu and Philip Adey. A scientific creativity test for secondary school students.Int. J. Sci. Educ., 24(4):389–403, April 2002

    2002

  19. [27]

    Qwen2.5-coder technical report.arXiv [cs.CL], September 2024

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou...

    2024

  20. [28]

    Creativity in ai: Progresses and challenges, 2024

    Mete Ismayilzada, Debjit Paul, Antoine Bosselut, and Lonneke van der Plas. Creativity in ai: Progresses and challenges, 2024. URLhttps://arxiv.org/abs/2410.17218

    2024 arXiv

  21. [29]

    Evaluating creative short story generation in humans and large language models.arXiv [cs.CL], November 2024

    Mete Ismayilzada, Claire Stevenson, and Lonneke van der Plas. Evaluating creative short story generation in humans and large language models.arXiv [cs.CL], November 2024

    2024

  22. [30]

    Evaluating creative short story generation in humans and large language models.arXiv preprint arXiv:2411.02316, 2024

    Mete Ismayilzada, Claire Stevenson, and Lonneke van der Plas. Evaluating creative short story generation in humans and large language models.arXiv preprint arXiv:2411.02316, 2024

    2024 arXiv

  23. [31]

    LLM output homogenization is task dependent.arXiv [cs.CL], September 2025

    Shomik Jain, Jack Lanchantin, Maximilian Nickel, Karen Ullrich, Ashia Wilson, and Jamelle Watson-Daniels. LLM output homogenization is task dependent.arXiv [cs.CL], September 2025

    2025

  24. [32]

    Mistral 7B.arXiv [cs.CL], October 2023

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas W...

    2023

  25. [33]

    Mixtral of experts.arXiv [cs.LG], January 2024

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne ...

    2024

  26. [34]

    Trust or escalate: LLM judges with provable guarantees for human agreement

    Jaehun Jung, Faeze Brahman, and Yejin Choi. Trust or escalate: LLM judges with provable guarantees for human agreement. InThe Thirteenth International Conference on Learning Representations, October 2024

    2024

  27. [35]

    Gemini 2.5: Our most intelligent AI model

    Koray Kavukcuoglu. Gemini 2.5: Our most intelligent AI model. https://blog.google/ technology/google-deepmind/gemini-model-thinking-updates-march-2025/ , March 2025. Accessed: 2025-4-30

    2025

  28. [36]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. InProceedings of the ACM SIGOPS 29th Symposium on Operating System...

    2023

  29. [37]

    From generation to judgment: Opportunities and challenges of llm-as-a-judge, 2024

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu. From generation to judgment: Opportunities and challenges of llm-as-a-judge, 2024

    2024

  30. [38]

    Llms-as-judges: A comprehensive survey on llm-based evaluation methods, 2024

    Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. Llms-as-judges: A comprehensive survey on llm-based evaluation methods, 2024. URL https://arxiv.org/abs/2412.05579

    2024 arXiv

  31. [39]

    Rethinking creativity evaluation: A critical analysis of existing creativity evaluations.arXiv [cs.CL], August 2025

    Li-Chun Lu, Miri Liu, Pin-Chun Lu, Yufei Tian, Shao-Hua Sun, and Nanyun Peng. Rethinking creativity evaluation: A critical analysis of existing creativity evaluations.arXiv [cs.CL], August 2025

    2025

  32. [40]

    AI as humanity’s salieri: Quantifying linguistic creativity of language models via systematic attribution of machine text against web text

    Ximing Lu, Melanie Sclar, Skyler Hallinan, Niloofar Mireshghallah, Jiacheng Liu, Seungju Han, Allyson Ettinger, Liwei Jiang, Khyathi Chandu, Nouha Dziri, and Yejin Choi. AI as humanity’s salieri: Quantifying linguistic creativity of language models via systematic attribution o...

    2024

  33. [41]

    Ai as humanity’s salieri: Quantifying linguistic creativity of language models via systematic attribution of machine text against web text.CoRR, abs/2410.04265, 2024

    Ximing Lu, Melanie Sclar, Skyler Hallinan, Niloofar Mireshghallah, Jiacheng Liu, Seungju Han, Allyson Ettinger, Liwei Jiang, Khyathi Raghavi Chandu, Nouha Dziri, and Yejin Choi. Ai as humanity’s salieri: Quantifying linguistic creativity of language models via systematic attri...

    2024 doi arXiv

  34. [42]

    Benchmarking language model creativity: A case study on code generation

    Yining Lu, Dixuan Wang, Tianjian Li, Dongwei Jiang, Sanjeev Khudanpur, Meng Jiang, and Daniel Khashabi. Benchmarking language model creativity: A case study on code generation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for ...

    2025

  35. [43]

    AidanBench: Evaluating novel idea generation on open-ended questions

    Aidan McLaughlin, Anuja Uppuluri, and James Campbell. AidanBench: Evaluating novel idea generation on open-ended questions. InLanguage Gamification - NeurIPS 2024 Workshop, December 2024

    2024

  36. [44]

    Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 english words

    Saif Mohammad. Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 english words. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 174–184, Stroudsburg, PA, USA,

  37. [45]

    Lane, Leonard Boussioux, Charles Ayoubi, Ying Hao Chen, Camila Lin, Rebecca Spens, Pooja Wagh, and Pei-Hsin Wang

    Jacqueline N. Lane, Leonard Boussioux, Charles Ayoubi, Ying Hao Chen, Camila Lin, Rebecca Spens, Pooja Wagh, and Pei-Hsin Wang. The narrative AI advantage? a field experiment on generative AI-augmented evaluations of early-stage innovations.Social Science Research Network, August 2024

    2024

  38. [46]

    Nam- ing unrelated words predicts creativity.Proc

    Jay A Olson, Johnny Nahas, Denis Chmoulevitch, Simon J Cropper, and Margaret E Webb. Nam- ing unrelated words predicts creativity.Proc. Natl. Acad. Sci. U. S. A., 118(25):e2022340118, June 2021

    2021

  39. [47]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024. URL https://openai.com/research/gpt-4. Ac- cessed: 2025-04-30

    2024

  40. [48]

    Beyond semantic distance: Automated scoring of divergent thinking greatly improves with large language models.Think

    Peter Organisciak, Selcuk Acar, Denis Dumas, and Kelly Berthiaume. Beyond semantic distance: Automated scoring of divergent thinking greatly improves with large language models.Think. Skills Creat., 49(101356):101356, September 2023

    2023

  41. [49]

    Does writing with language models reduce content diversity? InThe Twelfth International Conference on Learning Representations, October 2023

    Vishakh Padmakumar and He He. Does writing with language models reduce content diversity? InThe Twelfth International Conference on Learning Representations, October 2023

    2023

  42. [50]

    GloVe: Global vectors for word representation

    Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans, editors,Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 153...

    2014 doi

  43. [51]

    GloVe: Global vectors for word representation

    Jeffrey Pennington, Richard Socher, and Christopher D Manning. GloVe: Global vectors for word representation. InProceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, October 2014

    2014

  44. [52]

    Qwen2.5 technical report.arXiv [cs.CL], December 2024

    Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le ...

    2024

  45. [53]

    A meta-analysis of when and how advertising creativity works.J

    Sara Rosengren, Martin Eisend, Scott Koslow, and Micael Dahlen. A meta-analysis of when and how advertising creativity works.J. Mark., 84(6):39–56, November 2020

    2020

  46. [54]

    Approaches to measuring creativity: A systematic literature review.Creativity

    Sameh Said-Metwaly, Wim Van den Noortgate, and Eva Kyndt. Approaches to measuring creativity: A systematic literature review.Creativity. Theories – Research - Applications, 4(2): 238–275, December 2017. 16

    2017

  47. [55]

    Can LLMs generate novel research ideas? a large-scale human study with 100+ NLP researchers

    Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can LLMs generate novel research ideas? a large-scale human study with 100+ NLP researchers. InThe Thirteenth International Con- ference on Learning Representations, 2025. URL https://openreview.net/forum?id= M23dTGWCZy

    2025

  48. [56]

    Modeling the determinants and effects of creativity in advertising.Mark

    Robert E Smith, Scott B MacKenzie, Xiaojing Yang, Laura M Buchholz, and William K Darley. Modeling the determinants and effects of creativity in advertising.Mark. Sci., 26(6):819–833, November 2007

    2007

  49. [57]

    An investment theory of creativity and its development.Human Development, 34(1):1–31, June 1991

    R Sternberg and T Lubart. An investment theory of creativity and its development.Human Development, 34(1):1–31, June 1991

    1991

  50. [58]

    Judgebench: A benchmark for evaluating LLM-based judges

    Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Yuan Tang, Alejandro Cuadron, Chen- guang Wang, Raluca Popa, and Ion Stoica. Judgebench: A benchmark for evaluating LLM-based judges. InThe Thirteenth International Conference on Learning Representations, 2025. URL https://ope...

    2025

  51. [59]

    Yufei Tian, Tenghao Huang, Miri Liu, Derek Jiang, Alexander Spangher, Muhao Chen, Jonathan May, and Nanyun Peng. Are large language models capable of generating human-level nar- ratives? InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, ...

    2024

  52. [60]

    Yufei Tian, Abhilasha Ravichander, Lianhui Qin, Ronan Le Bras, Raja Marjieh, Nanyun Peng, Yejin Choi, Thomas Griffiths, and Faeze Brahman. MacGyver: Are large language models creative problem solvers? InProceedings of the 2024 Conference of the North American Chapter of the As...

    2024

  53. [61]

    Yufei Tian, Abhilasha Ravichander, Lianhui Qin, Ronan Le Bras, Raja Marjieh, Nanyun Peng, Yejin Choi, Thomas Griffiths, and Faeze Brahman. MacGyver: Are large language models creative problem solvers? In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the ...

    2024

  54. [62]

    Assessing the creativity of LLMs in proposing novel solutions to mathematical problems.arXiv [cs.CL], October 2024

    Junyi Ye, Jingyi Gu, Xinyun Zhao, Wenpeng Yin, and Guiling Wang. Assessing the creativity of LLMs in proposing novel solutions to mathematical problems.arXiv [cs.CL], October 2024

    2024

  55. [63]

    BERTScore: Evaluating text generation with BERT.arXiv [cs.CL], April 2019

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. BERTScore: Evaluating text generation with BERT.arXiv [cs.CL], April 2019

    2019

  56. [64]

    NoveltyBench: Evaluating creativity and diversity in language models

    Yiming Zhang, Harshita Diddee, Susan Holm, Hanchen Liu, Xinyue Liu, Vinay Samuel, Barry Wang, and Daphne Ippolito. NoveltyBench: Evaluating creativity and diversity in language models. InSecond Conference on Language Modeling, August 2025

    2025

  57. [65]

    Assessing and understanding creativity in large language models.Mach

    Yunpu Zhao, Rui Zhang, Wenyi Li, and Ling Li. Assessing and understanding creativity in large language models.Mach. Intell. Res., pages 1–20, April 2025

    2025

  58. [66]

    Judging LLM-as-a-judge with MT-bench and chatbot arena.arXiv [cs.CL], June 2023

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena.arXiv [cs.CL], June 2023

    2023

  59. [67]

    non-novel

    Shanshan Zhong, Zhongzhan Huang, Shanghua Gao, Wushao Wen, Liang Lin, Marinka Zitnik, and Pan Zhou. Let’s think outside the box: Exploring leap-of-thought in large language models with creative humor generation. In2024 IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2024

  60. [69]

    The protagonist suffers physical discomfort when overwhelmed by emotions (nausea, shaking, etc.)

  61. [70]

    The protagonist is challenged by the need to engage in public spaces

  62. [71]

    The unknown man realizes that the protagonist can feel his emotions

  63. [72]

    The protagonist uses humor and sarcasm to cope with her situation

  64. [73]

    The protagonist is an introverted character

  65. [74]

    The story includes communication via text messages

  66. [75]

    The story is set in a Starbucks on Michigan in Chicago a week before Christmas

  67. [76]

    The protagonist is forced to leave the meeting early due to being overwhelmed

  68. [77]

    The protagonist desires to live a more normal life despite her unique condition

  69. [78]

    There exists a vaccine for controlling intuition

  70. [79]

    The protagonist devises coping strategies for managing her anxiety in public places. 12. Scientists are working to find a solution for people who can’t use the intuition vaccine

  71. [80]

    Tiffany threatens the protagonist to meet her

  72. [81]

    The protagonist struggles with accepting her condition

  73. [82]

    The protagonist must grapple with the thoughts and feelings of others in the Starbucks

  74. [83]

    The man looks at the protagonist with both desire and love

  75. [84]

    The protagonist is physically attractive

  76. [85]

    The protagonist encounters an unknown man who causes powerful and unique emotions

  77. [86]

    There is societal disapproval for people whose intuitions cannot be controlled by the vaccine

  78. [87]

    The protagonist has a heightened intuition

  79. [88]

    The setting should be during the holiday season

  80. [89]

    The protagonist feels other people’s emotions intensely

  81. [90]

    The protagonist struggles with disentangling their own feelings from others’

  82. [91]

    Tiffany is a strong-willed and passionate character

  83. [92]

    "crazy cat lady

    The protagonist reluctantly acknowledges being a potential ""crazy cat lady""

  84. [93]

    Puberty is identified as a critical time for the progression of intuition powers

  85. [94]

    The protagonist and Tiffany were inseparable until puberty

  86. [95]

    The protagonist has personal hygiene items (travel mouthwash) handy

  87. [96]

    This vaccine doesn’t work for the protagonist due to a genetic mutation

  88. [97]

    Characters should express understanding of the protagonist’s predicament

  89. [98]

    The protagonist’s primary means of communication with the outside world is through the internet

  90. [99]

    The protagonist experiences other’s thoughts as if they were their own

  91. [100]

    The protagonist’s intuition is uncontrollable due to a genetic mutation

  92. [101]

    The protagonist’s coping mechanisms do not always successfully block out other people’s emotions

  93. [102]

    The protagonist uses strategies to block out the feelings of others, such as counting letters on the menu board

  94. [103]

    The protagonist finds solace in the idea of drinking coffee

  95. [104]

    Include a hint of romance in the story

  96. [105]

    There is societal pressure to control intuition with the vaccine

  97. [106]

    yes") only if it is being completely satisfied in the story. For no satisfaction/partial satisfaction, mark a

    The protagonist prefers isolation to manage their heightened intuition." 28 E.2.3 Experiment Configurations • Temperature: 0.75 • Max Token: 4096 • Top-p: 1 E.2.4 Inference Prompt Inference Prompt User: Write a story in less than 500 words about {story theme} Base Story: {base...

  98. [107]

    Week 18 aboard the Depth Reaver, Circa 2023

    Start the story with the sentence: "Week 18 aboard the Depth Reaver, Circa 2023"

    2023

  99. [108]

    Yes - The story is 302 words long, meeting the constraint of being less than 377 words

    Include a revelation of an unexpected large-scale phenomenon observed in space." Output 1. Yes - The story is 302 words long, meeting the constraint of being less than 377 words

  100. [109]

    Week 18 aboard the Depth Reaver, Circa 2023

    Yes - The story starts with the exact sentence: "Week 18 aboard the Depth Reaver, Circa 2023"

    2023

  101. [110]

    Yes - The revelation of the moon cracking open to reveal a colossal human face qualifies as an unexpected large-scale phenomenon observed in space. Number of constraints satisfied: 3 {other examples} Input Story: {story to be evaluated} Constraints: {constraints} Output 29 Eva...

  102. [111]

    Grammar: Which story has better writing and grammar comparitively?

  103. [112]

    Coherence: Which story has a better logical flow and the writing fits together with respect to the plot?

  104. [113]

    A" or "B

    Likability: Which story do you find more enjoyable to read? You will be given two Stories - Story A and Story B. Add a rating out of 5 for each category, specify which story you prefer for each metric by responding with just the letter "A" or "B" followed by a hyphen and one l...

  105. [117]

    Then, please provide a very brief reason for your evaluation based on the criteria above

    If the new solution is significantly simpler or more complex than the others, it can be regarded as essentially novel, even if they lead to the same result. Given the following mathematical problem: {problem} Reference solutions: {reference_solutions} New solution: {new_soluti...

  106. [118]

    Even if the final results are the same, if the intermediate steps or processes involved in reaching those solutions vary significantly, the new solution can be considered novel

  107. [119]

    If the new solution relies on different assumptions or conditions, it should be considered novel

  108. [120]

    In such cases, they are considered distinct

    A solution might generalize to a broader class of problems, while another solution might be specific to certain conditions. In such cases, they are considered distinct

  109. [121]

    If the new solution is significantly simpler or more complex than the others, it can be regarded as essentially novel, even if they lead to the same result. Given the following mathematical problem: {problem} Reference solutions: {reference_solutions} New solution: {new_soluti...

  110. [122]

    **Galaxy** 10

    **Sonnet** 9. **Galaxy** 10. **Bureaucracy** E.7.4 Model Performance Model DAT Score Mistral-7B 0.7908 Qwen2.5-7B 0.6907 OLMo2-7B 0.8058 Llama3.1-8B 0.8208 OLMo2-13B 0.8133 Mistral-24B 0.6004 Qwen2.5-32B 0.6919 Mixtral-8x7B 0.8298 Llama3.3-70B 0.6940 Qwen2.5-72B 0.7747 Claude3...

  111. [123]

    Fluency refers to the ability to generate a large quantity of ideas or solutions to a given problem

    Fluency. Fluency refers to the ability to generate a large quantity of ideas or solutions to a given problem. This measure isn’t concerned with the quality or uniqueness of the ideas, but rather the sheer volume. The more ideas one can produce, the higher the fluency

  112. [124]

    Flexibility is the capacity to shift one’s thinking and to produce a wide range of ideas from different categories or perspectives

    Flexibility. Flexibility is the capacity to shift one’s thinking and to produce a wide range of ideas from different categories or perspectives. It involves being able to think outside of the box and to switch from one type of idea to another

  113. [125]

    Originality refers to the ability to come up with unique or novel ideas that differ from the norm

    Originality. Originality refers to the ability to come up with unique or novel ideas that differ from the norm. It’s not just about producing many ideas (fluency), but also about producing ideas that are different from what others might typically think of

  114. [126]

    Elaboration is the ability to expand upon or add detail to ideas

    Elaboration. Elaboration is the ability to expand upon or add detail to ideas. It involves taking a simple idea and building upon it, adding complexity and depth. Elaboration isn’t just about creating more, but about deepening what is there

  115. [127]

    Finally, you will provide an overall score between 1 and 5, with 5 being the highest. You should only give the score, format like: Fluency: 3 Question: {Question} Answer: {Answer} Model Elaboration Flexibility Fluency Originality Mistral-7B 0.7861 0.7757 0.7660 0.7279 Qwen2.5-...

  116. [2018]

    Association for Computational Linguistics

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.