Pith. sign in

REVIEW 3 major objections 6 minor 27 references

LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces LitBench, the first standardized benchmark and paired dataset for creative-writing verification, and claims that a small trained reward model reaches 78% human agreement, beating all zero-shot LLM judges tested.

desk verdict Useful benchmark, honest limitations, but the headline accuracy numbers measure agreement with a filtered upvote proxy, not independent human taste. read the letter →

arxiv 2507.00769 v1 pith:NVTZIWXW submitted 2025-07-01 cs.CL cs.AI

Daniel Fein , Sebastian Russo , Violet Xiang , Kabir Jolly , Rafael Rafailov , Nick Haber This is my paper
classification cs.CLcs.AI
keywords creativewritingevaluationLLM-as-a-judgerewardmodelsBradley-TerrypreferencemodelgenerativeRedditupvotesashumanchain-of-thoughtreasoningr/WritingPrompts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to make creative-writing evaluation measurable, the way math and code verification already are. It builds LitBench, a benchmark of 2,480 paired story comparisons from r/WritingPrompts whose preferred story is inferred from upvotes after heavy filtering, plus a 43,827-pair training corpus. It claims that a small fine-tuned Bradley-Terry reward model reaches 78% agreement with these human-preference labels, beating every zero-shot LLM judge tested, including the strongest, Claude-3.7-Sonnet, at 73%. The paper also reports that chain-of-thought reasoning hurts generative reward models on this task, and that an online study on newly generated stories confirms the trained verifier's rankings transfer. This matters because reliable automated judgment is the missing ingredient for steering creative-writing generation the way verifiers accelerated math and code.

What carries the argument

The central object is LitBench's label-construction pipeline, which converts noisy Reddit upvote counts into pairwise 'chosen versus rejected' story labels by removing low-engagement stories, dropping pairs with small upvote gaps, requiring that the chosen story be published later, and balancing the length distribution so preference is not confounded with length. This pipeline produces the training signal for a Bradley-Terry discriminative reward model (a linear head on a Llama-8B backbone) that scores each story independently and is trained so the chosen story scores higher; the benchmark then measures how well any judge, trained or zero-shot, reproduces those labels.

What would settle it

Take a random sample of LitBench test pairs, have a panel of expert readers rate each pair directly, and compare those ratings to the upvote-derived 'chosen' labels. If human pairwise agreement with the upvote labels is near chance, or if the 78%-accurate verifier agrees with the upvote labels but not with the direct human ratings, then the benchmark measures agreement with a voting proxy rather than with human literary taste.

Watch

Extended reading notes

Core claim

The central claim is that creative writing has convergent human preferences, those preferences can be captured at scale from Reddit upvotes, and a verifier trained on such preferences can be evaluated and used. LitBench supplies the evaluation: 2,480 pairwise comparisons built from stories posted after January 2023, filtered for engagement, paired only when the higher-upvoted story was published later, and pruned on length so that 'chosen' stories are not simply the longer ones. On that test set, a Bradley-Terry reward model fine-tuned on LitBench's 43,827-pair training corpus reaches 78% accuracy, and the best zero-shot judge, Claude-3.7-Sonnet, reaches 73%. The paper also reports that adding chain-of-thought reasoning degrades generative reward models for this task (72% versus 78%), and that an online human study on 64 newly generated stories confirms the trained verifier ranks quality: humans chose the reward-model-preferred story 57% of the time versus 41% for the rejected story.

Load-bearing premise

The load-bearing premise is that upvote counts on r/WritingPrompts, after the curation filters, indicate human preference for creative writing quality; the paper never directly validates the benchmark's labels against fresh human pairwise ratings, only a trained verifier built from them.

Editorial extensions

If this is right

  • Small open-source verifiers (1B-8B) trained on LitBench can match or beat large proprietary LLM judges on creative-writing evaluation at a fraction of the cost.
  • Zero-shot LLM judges should not be treated as reliable for story quality: only frontier proprietary models in this comparison clear roughly 70% agreement, while smaller models hover near chance.
  • Chain-of-thought reasoning is not automatically helpful for judging narratives; in this domain it lowered generative-reward-model accuracy, so its use should be tested rather than assumed.
  • A verifier that scores well on LitBench generalizes to newly generated stories, so the benchmark can serve as a reward signal for steering creative-writing generators.
  • Because the test set is drawn from post-2023 stories, it provides a genuinely zero-shot evaluation for models with earlier training cutoffs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the upvote-preference assumption holds, the same curation recipe could be applied to other writing communities, poetry, or serialized fiction to build additional verifier datasets without fresh human annotation.
  • Because the paper's own limitation notes that Reddit demographics skew male, educated, and middle-aged, the 78% figure may overstate alignment with other reader populations; measuring verifier accuracy on ratings stratified by reader demographics would test this.
  • The explanation-text statistics, which found plot discussion most predictive of correct verdicts, suggest that rubric-guided prompts might outperform free-form reasoning for LLM judges, a testable extension of the paper's prompt-optimization results.
  • The 41% disagreement with the rejected story in the online study implies pairwise upvote labels leave substantial room for richer supervision, such as rubric-based ratings or expert critiques, before automatic rewards can fully capture literary taste.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. LitBench is a new benchmark and training dataset for evaluating automated judges of creative writing. The authors collect story pairs from Reddit's r/WritingPrompts, constructing labels from upvote counts with several curation filters (engagement threshold, upvote differential, temporal ordering, and length-balancing). The test set contains 2,480 pairs and the training set 43,827 pairs. The paper benchmarks zero-shot LLM judges, trains Bradley-Terry and generative reward models, and conducts an online human study on 64 newly generated stories. Headline results are that a fine-tuned Llama-8B Bradley-Terry reward model reaches 78% agreement with the labels, surpassing the best zero-shot judge (Claude-3.7-Sonnet at 73%), and that chain-of-thought distillation hurts generative reward model accuracy. The authors release the dataset and models.

Significance. If the benchmark labels are a valid proxy for human creative-writing preferences, LitBench would be a valuable resource: it is the first standardized pairwise creative-writing evaluation set of its scale, it ships with a training corpus, and the public release of code and models supports reproducibility. The paper also makes two useful empirical observations: small fine-tuned reward models can beat much larger zero-shot judges, and chain-of-thought supervision harms generative reward models in this domain. The ablations on curation filters (Figure 7) are a good-faith attempt to justify design choices. However, the central interpretation of the accuracy numbers depends on an assumption that is explicitly acknowledged but not directly tested: that upvotes, after filtering, encode human preference for writing quality. Because the test labels are never validated against independent human pairwise judgments, the headline 'human agreement' figures are currently agreement with a proxy.

major comments (3)
  1. [§3.2, §5, §7] The test set is called 'human-labeled' in the abstract and throughout, but the labels are derived solely from upvote counts (minimum 10 upvotes, ≥25% upvote differential, higher-upvote story published later). Section 7 explicitly concedes that upvotes may encode exposure, demographic skew, altruism, and other factors rather than literary quality. The online human study in Section 5 validates a trained reward model on 64 newly generated stories, not on the 2,480 test pairs. Consequently, the 73% and 78% accuracy figures in Figure 4 are agreements with the upvote proxy, not with independent human judgments. This is load-bearing for the benchmark's core interpretation. The authors should directly validate a random sample of test pairs with human pairwise ratings (reporting agreement, e.g., accuracy or Cohen's kappa) or, failing that, reword all 'human-labeled' and 'human agreement' claims to say 'upvote-preference proxy.'
  2. [Figure 4, Table A.1, Figure 8] The paper reports accuracy and preference percentages without any uncertainty quantification. With 2,480 test pairs, a difference between 78% and 73% may be significant, but the reader cannot tell; with only 64 generated pairs in the human study, the 57% versus 41% preference result needs a confidence interval or a significance test (e.g., a binomial test per pair or a mixed-effects model). Without this, the claims that trained models 'outperform' zero-shot judges and that the human study 'confirms' alignment are not statistically grounded. Please add error bars or confidence intervals to the key accuracy figures and report the outcome of a significance test for the human study.
  3. [§3.3, §4] The claim that the test set enables 'true zero-shot evaluation' is undermined for models with training cutoffs after January 2023. The test stories are all posted after January 2, 2023, and several evaluated models, such as GPT-4.1 and Claude-3.7-Sonnet, are likely to have been trained on data from that period. The paper should either restrict the zero-shot claim to models with confirmed earlier cutoffs, or include a leakage analysis (e.g., checking whether judge accuracy varies by story date or by model training cutoff).
minor comments (6)
  1. [§1] In the sentence 'In domains where human where ground truth are usually collected from human raters, LLM judges are often used,' the phrase 'where human where ground truth' appears to be a typo; it should read 'where ground truth labels are usually collected from human raters.'
  2. [§2.1] The paper repeatedly refers to 'ROGUE' metrics; the standard acronym is ROUGE. Please correct this throughout, including the related work discussion.
  3. [§3.2] 'This entire process is performed independently for both our benchmark and training dataset' is redundant with the preceding paragraph; consider removing or condensing. Also, 'reduce the affect of noise' should be 'reduce the effect of noise.'
  4. [§5 and Figure 8] The text says the best LLM judge (Claude-3.7-Sonnet) 'performs at chance' in the human study, but the details of how Claude-3.7-Sonnet was evaluated on the generated stories are not given. Please specify the evaluation protocol and the exact chance-level accuracy, or remove the comparison if it is not central.
  5. [Appendix A.2] There is a missing reference in the motivation paragraph: 'prompts enable (a) introduction of criteria ... (CITE).' Please fill in the citation.
  6. [General] The paper contains minor typographical errors, including 'GemRM-CoT' for 'GenRM-CoT', 'Across across' in Section 6, and inconsistent hyphenation of 'Bradley-Terry'/'Bradley Terry.' A careful proofreading pass is recommended.

Circularity Check

2 steps flagged · score 4.0 of 10

Several 'human agreement' results are agreement with a Reddit-upvote proxy that also defines the benchmark labels, and the curation ablation validates the filters against a test set built with those same filters; the online human study provides partial independent support.

  1. self definitional [Section 3.2 Quality Control and Section 5 'Validating Data Filtration Methodology']
    "To form pairs, we carry out two steps to ensure that true preferences are being captured, and then one step to address length bias. Initially, we exclude pairs with marginal differences in up-votes, filtering out those with an upvote difference less than 25%. Next, following the methodology of [Ethayarajh et al., 2022b], we only create pairs where the higher-upvote story is also published later, mitigating temporal bias from varying exposure durations."

    The test set on which the curation process is 'confirmed' was built with the very same filters being validated: the chosen story must have at least a 25% upvote margin and must also have been published later. A Bradley-Terry model trained on timestamp- and upvote-paired data is therefore predicting a target that already encodes that pairing rule, while the ablated models are penalized for missing a rule that was used to define the labels. The accuracy gap in Figure 7 is forced in part by construction, so the experiment cannot by itself show that the filters recover true human preference.

  2. renaming known result [Appendix A.3 Prompt Templates and Section 5 'BT and Generative Reward Models Outperform Zero-Shot LLMs']
    "Your task is to predict which response would receive more upvotes from the Reddit community. [...] The best Bradley–Terry reward model (Llama-8B) fine-tuned on LitBench training set achieves 78% human agreement."

    LitBench's 'chosen' label is defined in Section 3.2 as the higher-upvote story after filtering, and the judge prompt explicitly asks models to predict upvotes. Reporting the resulting accuracy as 'human agreement' renames upvote-prediction accuracy as agreement with human preferences; the target variable, the prompt, and the metric all refer to the same upvote proxy. The Limitations acknowledge the underlying assumption, and the online human study supplies some external support, but the headline zero-shot and reward-model accuracies on the 2,480 test pairs are not direct measurements against independent human pairwise judgments.

full rationale

The clearest circularity is local: the ablations that claim to confirm the curation filters evaluate on a test set that was itself produced by those same filters, so the measured benefit of timestamp and margin filtering is partly built into the labels. The headline accuracy numbers are a separate concern: they are agreement with a Reddit-upvote proxy rather than with independently collected pairwise human ratings, and the paper openly labels this as an assumption in Section 7. I do not see a load-bearing self-citation chain: citations to SHP, GenRMs, and Reddit-demographics work are external or methodological and do not by themselves force the results. The online human study with 46 crowdworkers and 64 new GPT-generated stories gives genuine, independent evidence that a LitBench-trained reward model's rankings transfer to fresh text, which prevents the benchmark from being definitionally circular. However, that study does not validate the 2,480 test labels themselves, so the benchmark's core 'human agreement' numbers remain measurements against the same proxy used to construct the labels. Overall, there is one locally circular validation plus a proxy-labeling conflated with human agreement, but meaningful independent content exists elsewhere, warranting a score of 4.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No new physical or theoretical entities are introduced; LitBench is a dataset and benchmark artifact. The free parameters are curation thresholds chosen by hand that define the benchmark and its label quality, none of which are given sensitivity analyses.

free parameters (5)
  • min_upvotes = 10 upvotes
    Hand-chosen engagement threshold in Section 3.2; affects which stories enter the benchmark and the label quality.
  • upvote_differential = 25% minimum gap
    Hand-chosen threshold in Section 3.2 to exclude near-tie pairs; no sensitivity analysis reported.
  • max_tokens = 2048 tokens
    Hand-chosen cutoff in Section 3.2 following Chung et al. 2025; excludes overly long stories.
  • min_words = 50 words
    Hand-chosen lower bound in Section 3.2 to exclude fragments; qualitative choice.
  • length_balance_buckets = 100 histogram buckets
    Hand-chosen bucket count for length-balancing step in Section 3.2; affects how aggressively length bias is removed.
assumptions (4)
  • domain assumption Reddit upvotes encode reliable human preferences for creative writing quality after curation.
    Invoked throughout Section 3.2 and explicit in Section 7 Limitations as an assumption inherited from Ethayarajh et al. 2023.
  • domain assumption Pairing higher-upvote stories with later publication time removes exposure bias.
    Section 3.2, 'we only create pairs where the higher-upvote story is also published later'; no direct test of residual exposure effects.
  • domain assumption Length balancing removes the main confound; residual non-length confounds are negligible.
    Section 3.2 and Figure 2; no test for residual confounds such as topic or genre popularity.
  • domain assumption Human creative preferences converge sufficiently to make upvote aggregates meaningful.
    Section 2.2 cites Amabile and canon formation to motivate that convergent preferences exist; this grounds the use of upvotes as quality labels.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing." pith.science (2026). https://pith.science/paper/NVTZIWXW

@misc{pith2026250700769,
  author       = {Pith},
  title        = {Pith review of: LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NVTZIWXW}},
  note         = {Machine review of arXiv:2507.00769}
}
read the original abstract

Evaluating creative writing generated by large language models (LLMs) remains challenging because open-ended narratives lack ground truths. Without performant automated evaluation methods, off-the-shelf (OTS) language models are employed as zero-shot judges, yet their reliability is unclear in this context. In pursuit of robust evaluation for creative writing, we introduce LitBench, the first standardized benchmark and paired dataset for creative writing verification, comprising a held-out test set of 2,480 debiased, human-labeled story comparisons drawn from Reddit and a 43,827-pair training corpus of human preference labels. Using LitBench, we (i) benchmark zero-shot LLM judges, (ii) train Bradley Terry and generative reward models, and (iii) conduct an online human study to validate reward model rankings on newly LLM-generated stories. Our benchmark identifies Claude-3.7-Sonnet as the strongest off-the-shelf judge, reaching 73% agreement with human preferences; among trained reward models, Bradley-Terry and Generative reward models both attain an accuracy of 78%, outperforming all off-the-shelf judges. An online human study further confirms that our trained reward models consistently align with human preferences in novel LLM-generated stories. We release LitBench and reward models at https://huggingface.co/collections/SAA-Lab/litbench-68267b5da3aafe58f9e43461, providing a vetted resource for reliable, automated evaluation and optimization of creative writing systems.

Figures

Figures reproduced from arXiv: 2507.00769 by the authors.

Figure 1
Figure 1. Preprocessing methodology for dataset creation. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Length bias mitigation. 3.3 Final Dataset Description LitBench consists of 2,480 pairwise comparisons composed of 3,543 total stories. These stories have an average length of 550 words, and story length is right-skewed, with a tail of longer stories. The data is exclusively sourced from after January 2nd, 2023. This guarantees the data have no overlap with our training dataset, as well as enabling true zero-shot eva… view at source ↗
Figure 3
Figure 3. Distributions of word count, date, and upvotes for the LitBench test- and train-set. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Trained verifiers outperform zero-shot LLM-judges on LitBench. Claude3.7-Sonnet is [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Qualities of explanation text that impact verdict accuracy. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Human agreement scaling is inconsistent with model size for different types of RMs. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Naive upvote pairing and naive timestamp and upvote pairing saturate at lower accuracy [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Human preference alignment with reward model on generated writing pairs. [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 9 canonical work pages

  1. [8]

    Fabio Duarte

    URL https://arxiv.org/abs/2504.18116. Fabio Duarte. Reddit user age, gender, & demographics (2025). https://explodingtopics.com/ blog/reddit-users, May

    arXiv 2025

  2. [9]

    Michele Elam

    Accessed: 2025-05-06. Michele Elam. Poetry will not optimize; or, what is literature to ai? American literature, 95(2): 281–303,

    work page 2025

  3. [10]

    Understanding Dataset Difficulty with $\mathcal{V}$-Usable Information

    Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. arXiv preprint arXiv:2110.08420 , 2022a. URL https://arxiv.org/ abs/2110.08420. Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song,...

    work page Pith review arXiv

  4. [11]

    Hierarchical neural story generation

    Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833, 2018a. URL https://arxiv.org/abs/1805.04833. Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation, 2018b. URL https://arxiv.org/abs/1805.04833. Benjamin Feuer, Micah Goldblum, Teresa Datta, Sanjana Nambiar, Raz Besalel...

    arXiv

  5. [12]

    Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al

    URL https://arxiv.org/abs/2409.15268. Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985,

    arXiv

  6. [13]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874,

    arXiv

  7. [15]

    org/abs/2504.15784

    URL https://arxiv. org/abs/2504.15784. Yang Liu, Jonas Schneider, Jonathan Raiman, Ian Tenney, Nitish Gupta, Diya Raghu, Douwe Kiela, and Lazaros Polymenakos. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634,

    arXiv

  8. [16]

    Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak

    URL https://arxiv.org/abs/2303.16634. Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. arXiv preprint arXiv:2410.12832,

    arXiv

Show all 27 references
  1. [17]

    Danielle S McNamara, Scott A Crossley, and Philip M McCarthy

    URL https://deepblue.lib.umich.edu/handle/2027.42/133407. Danielle S McNamara, Scott A Crossley, and Philip M McCarthy. Linguistic features of writing quality. Written communication, 27(1):57–86,

    2027

  2. [20]

    URL https://store.macmillanlearning.com/us/product/ The-Practice-of-Creative-Writing/p/1319215955

    ISBN 9781319215958. URL https://store.macmillanlearning.com/us/product/ The-Practice-of-Creative-Writing/p/1319215955 . Guijin Son, Hyunwoo Ko, Hoyoung Lee, Yewon Kim, and Seunghyeok Hong. Llm-as-a-judge and reward model: What they can and cannot do,

  3. [23]

    org/abs/2305.17926

    URL https://arxiv. org/abs/2305.17926. Hui Wei, Shenghua He, Tian Xia, Fei Liu, Andy Wong, Jingyang Lin, and Mei Han. Systematic evaluation of llm-as-a-judge in llm alignment tasks: Explainable metrics and diverse prompt templates,

    arXiv

  4. [24]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H

    URL https://arxiv.org/abs/2408.13006. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V . Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903,

    arXiv

  5. [25]

    Sara Cushing Weigle

    URL https://arxiv.org/abs/2201.11903. Sara Cushing Weigle. Assessing Writing . Cambridge University Press, Cambridge,

    arXiv

  6. [27]

    Generative verifiers: Reward modeling as next-token prediction

    Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408.15240,

    arXiv

  7. [29]

    URL https://arxiv.org/abs/2306.05685. 16

    arXiv

  8. [1960]

    Ralph Allan Bradley and Milton E Terry

    URL https://harpers.org/archive/1960/10/ the-cults-of-research-and-creativity/ . Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345,

    1960

  9. [1961]

    Heather Sellers

    URL https: //www.jstor.org/stable/20342603. Heather Sellers. The Practice of Creative Writing: A Guide for Students . Bed- ford/St. Martin’s (Macmillan Learning), New York, NY , 4 edition,

  10. [1982]

    Sher Badshah and Hassan Sajjad

    doi: 10.1037/0022-3514.43.5.997. Sher Badshah and Hassan Sajjad. Reference-guided verdict: Llms-as-judges in automatic evaluation of free-form text,

    doi

  11. [2002]

    15 Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V Chawla, and Xiangliang Zhang

    doi: 10.1017/CBO9780511732997. 15 Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V Chawla, and Xiangliang Zhang. Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2...

    arXiv

  12. [2016]

    Danial Alihosseini, Ehsan Montahaei, and Mahdieh Soleymani Baghshah

    Accessed: 2025-05-06. Danial Alihosseini, Ehsan Montahaei, and Mahdieh Soleymani Baghshah. Jointly measuring diversity and quality in text generation models. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation , pages 90–98, Minne...

    2025

  13. [2019]

    doi: 10.18653/v1/W19-2311

    Association for Computational Linguistics. doi: 10.18653/v1/W19-2311. URL https://aclanthology. org/W19-2311. Teresa M. Amabile. Social psychology of creativity: A consensual assessment technique. Journal of Personality and Social Psychology, 43(5):997–1013,

    doi

  14. [2020]

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui

    URL https://arxiv.org/abs/2009.01325. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators,

    2009 arXiv

  15. [2021]

    Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770,

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770,

    arXiv

  16. [2022]

    Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang

    URL https://arxiv.org/abs/2203.02155. Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym.arXiv preprint arXiv:2412.21139,

    arXiv

  17. [2023]

    URL https: //doi.org/10.1111/exsy.13292

    doi: 10.1111/exsy.13292. URL https: //doi.org/10.1111/exsy.13292. 13 Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien-Sheng Wu. Art or artifice? large language models and the false promise of creativity. In Proceedings of the 2024 CHI Conference...

    doi

  18. [2024]

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al

    URL https://arxiv.org/abs/2408.09235. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073,

  19. [2025]

    URL https: //arxiv.org/abs/2503.17126. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. a...

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.