{"work":{"id":"1ef2ec4a-db7b-42b2-9416-0f1bb628c3c8","openalex_id":"https://openalex.org/W2938704169","doi":"10.48550/arxiv.1904.09751","arxiv_id":"1904.09751","raw_key":null,"title":"The Curious Case of Neural Text Degeneration","authors":null,"authors_text":"Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi","year":2019,"venue":"cs.CL","abstract":"Despite considerable advancements with deep neural language models, the enigma of neural text degeneration persists when these models are tested as text generators. The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, using likelihood as a decoding objective leads to text that is bland and strangely repetitive.\n  In this paper, we reveal surprising distributional differences between human text and machine text. In addition, we find that decoding strategies alone can dramatically effect the quality of machine text, even when generated from exactly the same neural language model. Our findings motivate Nucleus Sampling, a simple but effective method to draw the best out of neural generation. By sampling text from the dynamic nucleus of the probability distribution, which allows for diversity while effectively truncating the less reliable tail of the distribution, the resulting text better demonstrates the quality of human text, yielding enhanced diversity without sacrificing fluency and coherence.","external_url":"https://arxiv.org/abs/1904.09751","cited_by_count":1112,"metadata_source":"pith","metadata_fetched_at":"2026-08-05T02:28:24.338817+00:00","pith_arxiv_id":"1904.09751","created_at":"2026-05-09T20:37:32.161269+00:00","updated_at":"2026-08-05T02:28:24.338817+00:00","title_quality_ok":true,"display_title":"The Curious Case of Neural Text Degeneration","render_title":"The Curious Case of Neural Text Degeneration"},"hub":{"state":{"work_id":"1ef2ec4a-db7b-42b2-9416-0f1bb628c3c8","tier":"super_hub","tier_reason":"100+ Pith inbound or 10,000+ external citations","pith_inbound_count":112,"external_cited_by_count":1112,"distinct_field_count":19,"first_pith_cited_at":"2019-05-19T23:57:23+00:00","last_pith_cited_at":"2026-07-08T15:50:54+00:00","author_build_status":"needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-08-23T23:49:26.624988+00:00","tier_text":"super_hub"},"tier":"super_hub","role_counts":[{"context_role":"background","n":15},{"context_role":"method","n":3},{"context_role":"other","n":1}],"polarity_counts":[{"context_polarity":"background","n":15},{"context_polarity":"use_method","n":3},{"context_polarity":"unclear","n":1}],"runs":{"ask_index":{"job_type":"ask_index","status":"succeeded","result":{"title":"The Curious Case of Neural Text Degeneration","claims":[{"claim_text":"Despite considerable advancements with deep neural language models, the enigma of neural text degeneration persists when these models are tested as text generators. The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, using likelihood as a decoding objective leads to text that is bland and strangely repetitive.\n  In this paper, we reveal surprising distributional differences between human text and machine text. In addition, we find that decoding strategies alon","claim_type":"abstract","evidence_strength":"source_metadata"},{"claim_text":"InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1689-1701, 2019. [33] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020. [34] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text","claim_type":"background","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"Third, we analyse the frequency of discourse connectives in each explanation. We focus on connec- tives because empirical work reports that explicit markers can reduce reading time, improve memory for clauses, and increase comprehension accuracy [45], which are relevant for XAI explanations. We detect connectives using the Lexicon of English Discourse Connectives Das et al. [56]. The lexicon contains 142 unique connective types and groups them into four relation classes. We define Connectives Ra","claim_type":"method","confidence":0.85,"evidence_strength":"citation_context"},{"claim_text":"The four signals below provide a deterministic decomposition of \"the model wastes its budget\"; each both terminates the loop early and is reported per-task as an independent quality measurement (Section 4.5). These signals draw on three prior-art lines: cycle detection in iterative search [ 12, 3, 8], stall generations in evolutionary algorithms [ 7], and degeneration in LLM text generation [ 10]. Since iterative LLM refinement is not guaranteed to converge [16, 21], we apply four explicit non-c","claim_type":"background","confidence":0.85,"evidence_strength":"citation_context"},{"claim_text":"Jiang, M. Liu, X. Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. InCVPR, 2022. [50] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. InICLR, 2021. [51] J. R. Hermans, G. Spanakis, and R. Möckel. Accumulated gradient normalization. InACML, 2017. [52] A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi. The curious case of neural text degeneration.arXiv preprint arXiv:1904.09751, 2019. [5","claim_type":"background","confidence":0.8,"evidence_strength":"citation_context"},{"claim_text":"calculated the recall of our prototype on a clone pair level, which is comparable to CCD tools. To interpret the data it is important to keep in mind that we cre- ated our search patterns manually using only reference implementations from the internet. The CCD tools on 2Since the runtime of CCD tools can be quite long BigCloneEval recommends to only consider pairs with a minimal token length of 50 [44]. We additionally report our results with a minimum token limit of 0, since many Fibonacci and ","claim_type":"background","confidence":0.8,"evidence_strength":"citation_context"},{"claim_text":"The human-annotated preference data involves comparing several candidate model responses using a 5-scale rating system. The prompts for generating preference data cover all general visual understanding abilities, and maintain a balanced scale across all abilities. We utilize the current top-performing in-house models to randomly sample responses through nucleus sampling [47]. To ensure the diversity of responses, we apply filtering techniques-such as editing distance, semantic similarity, and le","claim_type":"method","confidence":0.8,"evidence_strength":"citation_context"}],"why_cited":"Pith tracks The Curious Case of Neural Text Degeneration because it crossed a citation-hub threshold. Current citing contexts most often use it as background evidence (15 contexts).","role_counts":[{"n":15,"context_role":"background"},{"n":3,"context_role":"method"},{"n":1,"context_role":"other"}]},"error":null,"updated_at":"2026-07-03T05:53:26.630882+00:00"},"author_expand":{"job_type":"author_expand","status":"succeeded","result":{"authors_linked":[{"id":"dee8d433-2639-40cc-a1f9-711f8d3afdd2","orcid":null,"display_name":"Ari Holtzman"},{"id":"0996fd47-fbfc-4e1f-856a-c1ab48013a2e","orcid":null,"display_name":"Jan Buys"},{"id":"87eb9e1b-8792-4f90-a224-e826af402405","orcid":null,"display_name":"Li Du"},{"id":"905352e3-dc28-4d29-b41d-217450c2f1a2","orcid":null,"display_name":"Maxwell Forbes"},{"id":"346b7b6d-4a6f-40a5-8b42-db5c7ee77c4b","orcid":null,"display_name":"Yejin Choi"}]},"error":null,"updated_at":"2026-07-03T05:53:26.627113+00:00"},"context_extract":{"job_type":"context_extract","status":"succeeded","result":{"enqueued_papers":25},"error":null,"updated_at":"2026-05-14T17:59:57.953448+00:00"},"graph_features":{"job_type":"graph_features","status":"succeeded","result":{"co_cited":[{"title":"LLaMA: Open and Efficient Foundation Language Models","work_id":"c018fc23-6f3f-4035-9d02-28a2173b2b9d","shared_citers":11},{"title":"The Llama 3 Herd of Models","work_id":"1549a635-88af-4ac1-acfe-51ae7bb53345","shared_citers":10},{"title":"GPT-4 Technical Report","work_id":"b928e041-6991-4c08-8c81-0359e4097c7b","shared_citers":9},{"title":"Qwen3 Technical Report","work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e","shared_citers":8},{"title":"DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning","work_id":"e6b75ad5-2877-4168-97c8-710407094d20","shared_citers":7},{"title":"Evaluating Large Language Models Trained on Code","work_id":"042493e9-b26f-4b4e-bbde-382072ca9b08","shared_citers":7},{"title":"Llama 2: Open Foundation and Fine-Tuned Chat Models","work_id":"68a5177f-d644-44c1-bd4f-4e5278c22f5d","shared_citers":7},{"title":"Proximal Policy Optimization Algorithms","work_id":"240c67fe-d14d-4520-91c1-38a4e272ca19","shared_citers":7},{"title":"DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models","work_id":"c5006563-f3ec-438a-9e35-b7b484f34828","shared_citers":6},{"title":"Fine-Tuning Language Models from Human Preferences","work_id":"4f54aad1-f3b6-404f-b9c7-e21ba0a33b99","shared_citers":6},{"title":"Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge","work_id":"28ea1282-d657-4c61-a83c-f1249be6d6b1","shared_citers":6},{"title":"Training Verifiers to Solve Math Word Problems","work_id":"acab1aa8-b4d6-40e0-a3ee-25341701dca2","shared_citers":6},{"title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","work_id":"ed240a10-5b19-406c-baa5-30803f465785","shared_citers":5},{"title":"Hierarchical neural story generation.CoRR, abs/1805.04833","work_id":"5e06c7f2-e43a-4c73-b898-2848e095ff14","shared_citers":5},{"title":"Language Models are Few-Shot Learners","work_id":"214732c0-2edd-44a0-af9e-28184a2b8279","shared_citers":5},{"title":"Measuring Massive Multitask Language Understanding","work_id":"e87ec49a-544b-4ec8-8991-75298c64ff5e","shared_citers":5},{"title":"Training Compute-Optimal Large Language Models","work_id":"b2faf28d-86b7-429c-bc42-469458efc246","shared_citers":5},{"title":"Attention Is All You Need","work_id":"baafb5a2-5272-43bc-932f-09fa9ffe5316","shared_citers":4},{"title":"Decoupled Weight Decay Regularization","work_id":"07ef7360-d385-4033-83f7-8384a6325204","shared_citers":4},{"title":"DeepSeek-V3 Technical Report","work_id":"57d2791d-2219-4c31-a077-afc04b12a75c","shared_citers":4},{"title":"HellaSwag: Can a Machine Really Finish Your Sentence?","work_id":"79f44c0c-96f4-4edb-bc50-a3c9d6b85936","shared_citers":4},{"title":"Measuring Mathematical Problem Solving With the MATH Dataset","work_id":"50652ac6-fb7c-4675-a2c2-159c241feb17","shared_citers":4},{"title":"Qwen2.5 Technical Report","work_id":"d8432992-4980-4a81-85c7-9fa2c2b87f85","shared_citers":4},{"title":"Scaling Laws for Neural Language Models","work_id":"b7dd8749-9c45-4977-ab9b-64478dce1ae8","shared_citers":4}],"time_series":[{"n":1,"year":2019},{"n":1,"year":2020},{"n":1,"year":2021},{"n":4,"year":2024},{"n":2,"year":2025},{"n":28,"year":2026}],"dependency_candidates":[]},"error":null,"updated_at":"2026-05-14T17:59:32.245350+00:00"},"identity_refresh":{"job_type":"identity_refresh","status":"succeeded","result":{"items":[{"title":"Qwen3 Technical Report","outcome":"unchanged","work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e","resolver":"local_arxiv","confidence":0.98,"old_work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e"}],"counts":{"fixed":0,"merged":0,"unchanged":1,"quarantined":0,"needs_external_resolution":0},"errors":[],"attempted":1},"error":null,"updated_at":"2026-05-14T18:00:02.489561+00:00"},"role_polarity":{"job_type":"role_polarity","status":"succeeded","result":{"title":"The Curious Case of Neural Text Degeneration","claims":[{"claim_text":"Despite considerable advancements with deep neural language models, the enigma of neural text degeneration persists when these models are tested as text generators. The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, using likelihood as a decoding objective leads to text that is bland and strangely repetitive.\n  In this paper, we reveal surprising distributional differences between human text and machine text. In addition, we find that decoding strategies alon","claim_type":"abstract","evidence_strength":"source_metadata"},{"claim_text":"InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1689-1701, 2019. [33] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020. [34] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text","claim_type":"background","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"Third, we analyse the frequency of discourse connectives in each explanation. We focus on connec- tives because empirical work reports that explicit markers can reduce reading time, improve memory for clauses, and increase comprehension accuracy [45], which are relevant for XAI explanations. We detect connectives using the Lexicon of English Discourse Connectives Das et al. [56]. The lexicon contains 142 unique connective types and groups them into four relation classes. We define Connectives Ra","claim_type":"method","confidence":0.85,"evidence_strength":"citation_context"},{"claim_text":"The four signals below provide a deterministic decomposition of \"the model wastes its budget\"; each both terminates the loop early and is reported per-task as an independent quality measurement (Section 4.5). These signals draw on three prior-art lines: cycle detection in iterative search [ 12, 3, 8], stall generations in evolutionary algorithms [ 7], and degeneration in LLM text generation [ 10]. Since iterative LLM refinement is not guaranteed to converge [16, 21], we apply four explicit non-c","claim_type":"background","confidence":0.85,"evidence_strength":"citation_context"},{"claim_text":"Jiang, M. Liu, X. Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. InCVPR, 2022. [50] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. InICLR, 2021. [51] J. R. Hermans, G. Spanakis, and R. Möckel. Accumulated gradient normalization. InACML, 2017. [52] A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi. The curious case of neural text degeneration.arXiv preprint arXiv:1904.09751, 2019. [5","claim_type":"background","confidence":0.8,"evidence_strength":"citation_context"},{"claim_text":"calculated the recall of our prototype on a clone pair level, which is comparable to CCD tools. To interpret the data it is important to keep in mind that we cre- ated our search patterns manually using only reference implementations from the internet. The CCD tools on 2Since the runtime of CCD tools can be quite long BigCloneEval recommends to only consider pairs with a minimal token length of 50 [44]. We additionally report our results with a minimum token limit of 0, since many Fibonacci and ","claim_type":"background","confidence":0.8,"evidence_strength":"citation_context"},{"claim_text":"The human-annotated preference data involves comparing several candidate model responses using a 5-scale rating system. The prompts for generating preference data cover all general visual understanding abilities, and maintain a balanced scale across all abilities. We utilize the current top-performing in-house models to randomly sample responses through nucleus sampling [47]. To ensure the diversity of responses, we apply filtering techniques-such as editing distance, semantic similarity, and le","claim_type":"method","confidence":0.8,"evidence_strength":"citation_context"}],"why_cited":"Pith tracks The Curious Case of Neural Text Degeneration because it crossed a citation-hub threshold. Current citing contexts most often use it as background evidence (15 contexts).","role_counts":[{"n":15,"context_role":"background"},{"n":3,"context_role":"method"},{"n":1,"context_role":"other"}]},"error":null,"updated_at":"2026-07-03T05:53:26.633366+00:00"},"summary_claims":{"job_type":"summary_claims","status":"succeeded","result":{"title":"The Curious Case of Neural Text Degeneration","claims":[{"claim_text":"Despite considerable advancements with deep neural language models, the enigma of neural text degeneration persists when these models are tested as text generators. The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, using likelihood as a decoding objective leads to text that is bland and strangely repetitive.\n  In this paper, we reveal surprising distributional differences between human text and machine text. In addition, we find that decoding strategies alon","claim_type":"abstract","evidence_strength":"source_metadata"}],"why_cited":"Pith tracks The Curious Case of Neural Text Degeneration because it crossed a citation-hub threshold.","role_counts":[]},"error":null,"updated_at":"2026-05-14T17:59:27.366443+00:00"}},"summary":{"title":"The Curious Case of Neural Text Degeneration","claims":[{"claim_text":"Despite considerable advancements with deep neural language models, the enigma of neural text degeneration persists when these models are tested as text generators. The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, using likelihood as a decoding objective leads to text that is bland and strangely repetitive.\n  In this paper, we reveal surprising distributional differences between human text and machine text. In addition, we find that decoding strategies alon","claim_type":"abstract","evidence_strength":"source_metadata"}],"why_cited":"Pith tracks The Curious Case of Neural Text Degeneration because it crossed a citation-hub threshold.","role_counts":[]},"graph":{"co_cited":[{"title":"LLaMA: Open and Efficient Foundation Language Models","work_id":"c018fc23-6f3f-4035-9d02-28a2173b2b9d","shared_citers":11},{"title":"The Llama 3 Herd of Models","work_id":"1549a635-88af-4ac1-acfe-51ae7bb53345","shared_citers":10},{"title":"GPT-4 Technical Report","work_id":"b928e041-6991-4c08-8c81-0359e4097c7b","shared_citers":9},{"title":"Qwen3 Technical Report","work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e","shared_citers":8},{"title":"DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning","work_id":"e6b75ad5-2877-4168-97c8-710407094d20","shared_citers":7},{"title":"Evaluating Large Language Models Trained on Code","work_id":"042493e9-b26f-4b4e-bbde-382072ca9b08","shared_citers":7},{"title":"Llama 2: Open Foundation and Fine-Tuned Chat Models","work_id":"68a5177f-d644-44c1-bd4f-4e5278c22f5d","shared_citers":7},{"title":"Proximal Policy Optimization Algorithms","work_id":"240c67fe-d14d-4520-91c1-38a4e272ca19","shared_citers":7},{"title":"DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models","work_id":"c5006563-f3ec-438a-9e35-b7b484f34828","shared_citers":6},{"title":"Fine-Tuning Language Models from Human Preferences","work_id":"4f54aad1-f3b6-404f-b9c7-e21ba0a33b99","shared_citers":6},{"title":"Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge","work_id":"28ea1282-d657-4c61-a83c-f1249be6d6b1","shared_citers":6},{"title":"Training Verifiers to Solve Math Word Problems","work_id":"acab1aa8-b4d6-40e0-a3ee-25341701dca2","shared_citers":6},{"title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","work_id":"ed240a10-5b19-406c-baa5-30803f465785","shared_citers":5},{"title":"Hierarchical neural story generation.CoRR, abs/1805.04833","work_id":"5e06c7f2-e43a-4c73-b898-2848e095ff14","shared_citers":5},{"title":"Language Models are Few-Shot Learners","work_id":"214732c0-2edd-44a0-af9e-28184a2b8279","shared_citers":5},{"title":"Measuring Massive Multitask Language Understanding","work_id":"e87ec49a-544b-4ec8-8991-75298c64ff5e","shared_citers":5},{"title":"Training Compute-Optimal Large Language Models","work_id":"b2faf28d-86b7-429c-bc42-469458efc246","shared_citers":5},{"title":"Attention Is All You Need","work_id":"baafb5a2-5272-43bc-932f-09fa9ffe5316","shared_citers":4},{"title":"Decoupled Weight Decay Regularization","work_id":"07ef7360-d385-4033-83f7-8384a6325204","shared_citers":4},{"title":"DeepSeek-V3 Technical Report","work_id":"57d2791d-2219-4c31-a077-afc04b12a75c","shared_citers":4},{"title":"HellaSwag: Can a Machine Really Finish Your Sentence?","work_id":"79f44c0c-96f4-4edb-bc50-a3c9d6b85936","shared_citers":4},{"title":"Measuring Mathematical Problem Solving With the MATH Dataset","work_id":"50652ac6-fb7c-4675-a2c2-159c241feb17","shared_citers":4},{"title":"Qwen2.5 Technical Report","work_id":"d8432992-4980-4a81-85c7-9fa2c2b87f85","shared_citers":4},{"title":"Scaling Laws for Neural Language Models","work_id":"b7dd8749-9c45-4977-ab9b-64478dce1ae8","shared_citers":4}],"time_series":[{"n":1,"year":2019},{"n":1,"year":2020},{"n":1,"year":2021},{"n":4,"year":2024},{"n":2,"year":2025},{"n":28,"year":2026}],"dependency_candidates":[]},"authors":[{"id":"dee8d433-2639-40cc-a1f9-711f8d3afdd2","orcid":null,"display_name":"Ari Holtzman","source":"manual","import_confidence":0.72},{"id":"0996fd47-fbfc-4e1f-856a-c1ab48013a2e","orcid":null,"display_name":"Jan Buys","source":"manual","import_confidence":0.72},{"id":"87eb9e1b-8792-4f90-a224-e826af402405","orcid":null,"display_name":"Li Du","source":"manual","import_confidence":0.72},{"id":"905352e3-dc28-4d29-b41d-217450c2f1a2","orcid":null,"display_name":"Maxwell Forbes","source":"manual","import_confidence":0.72},{"id":"346b7b6d-4a6f-40a5-8b42-db5c7ee77c4b","orcid":null,"display_name":"Yejin Choi","source":"manual","import_confidence":0.72}]}}