Pith. sign in

REVIEW 5 cited by

NoveltyBench: Evaluating Language Models for Humanlike Diversity

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2504.05228 v4 pith:KR52H6MG submitted 2025-04-07 cs.CL

Yiming Zhang , Harshita Diddee , Susan Holm , Hanchen Liu , Xinyue Liu , Vinay Samuel , Barry Wang , Daphne Ippolito This is my paper · ORCID
classification cs.CL
keywords diversitymodelslanguagenoveltybenchbenchmarkscurrentdiverseelicit
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Language models have demonstrated remarkable capabilities on standard benchmarks, yet they struggle increasingly from mode collapse, the inability to generate diverse and novel outputs. Our work introduces NoveltyBench, a benchmark specifically designed to evaluate the ability of language models to produce multiple distinct and high-quality outputs. NoveltyBench utilizes prompts curated to elicit diverse answers and filtered real-world user queries. Evaluating 20 leading language models, we find that current state-of-the-art systems generate significantly less diversity than human writers. Notably, larger models within a family often exhibit less diversity than their smaller counterparts, challenging the notion that capability on standard benchmarks translates directly to generative utility. While prompting strategies like in-context regeneration can elicit diversity, our findings highlight a fundamental lack of distributional diversity in current models, reducing their utility for users seeking varied responses and suggesting the need for new training and evaluation paradigms that prioritize diversity alongside quality.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Improving LLMs via Validator-to-Generator Alignment

    cs.CL 2026-07 conditional novelty 7.0 of 10

    Frequency-corrected rank alignment of an LLM generator to its own validator improves generator AUROC and G-V Pearson correlation by up to 27 points while preserving validator quality.

  2. Beyond One Output: Visualizing and Comparing Distributions of Language Model Generations

    cs.AI 2026-04 conditional novelty 7.0 of 10

    GROVE visualizes distributions of language model generations as overlapping paths through a text graph, with user studies showing that graph summaries aid structural judgments like diversity assessment while raw outpu...

  3. Structured Output Collapses Answer Diversity Across 44 Language Models

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Requesting JSON instead of plain chat measurably reduces answer diversity across 44 LLMs, concentrating answers onto the field's modal choice.

  4. The One-Word Census: Answer-Choice Conformity Across 44 Language Models

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Forty-four language models asked to name one thing per category converge on the same modal answers far more than people do, with newest flagships most conformist and persona-tuned models most divergent.

  5. Avoidance Decoding for Diverse Multi-Branch Story Generation

    cs.CL 2025-09 conditional novelty 5.0 of 10

    Avoidance Decoding penalizes token choices that resemble previously generated story branches, using a hybrid concept-level and narrative-level similarity penalty, and reports large diversity gains across several LLMs.

Pith tools