Pith. sign in

REVIEW 3 major objections 8 minor 2 cited by

LIFEBench: Evaluating Length Instruction Following in Large Language Models

T0 review · 3 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Almost no LLM reaches its advertised max output, 26-model test finds

desk verdict Solid bilingual benchmark for length instruction following; qualitative findings hold, but the leaderboard rests on hand-set scoring constants that need a sensitivity check. read the letter →

arxiv 2505.16234 v2 pith:AXCPT52C submitted 2025-05-22 cs.CL cs.AI

Wei Zhang , Zhenhong Zhou , Kun Wang , Junfeng Fang , Yuanhe Zhang , Rui Wang , Ge Zhang , Xavier Li
show 4 more authors
Li Sun Lingjuan Lyu Yang Liu Sen Su
This is my paper
classification cs.CLcs.AI
keywords lengthinstructionfollowinglong-textgenerationlargelanguagemodelsbenchmarkcontrolbilingualevaluationoutputawarenessreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LIFEBench is a benchmark designed to test one seemingly simple skill: whether large language models can produce output of a requested length, from 16 words to 8,192 words, in English and Chinese, across question answering, summarization, reasoning, and creative generation. The paper's central claim is that current models follow short length instructions acceptably but deteriorate sharply once targets grow, and that none can be relied on for exact-length outputs at long constraints. Its second claim is that almost all of the 26 tested models fall well short of the maximum output lengths their vendors advertise, despite long-context capabilities. A sympathetic reader would care because length control is not an abstract nicety: report drafting, summarization, and creative writing all depend on the model delivering what the prompt asked for. The paper also offers a diagnosis — models lack awareness of how many words they have actually produced, and sometimes quit or refuse early — and shows that reasoning models, which check length during their intermediate steps, currently do best but still fail at long targets.

What carries the argument

The load-bearing object is the Length Score (LS), an exponential penalty curve defined for each control method; for Equal To it is $LS = 100 e^{k_1 LD}$ when $LD<0$ and $LS = 100 e^{-k_2 LD}$ when $LD\ge 0$, with $k_1=5$ and $k_2=2$, where $LD=(L_{\text{output}}-L_{\text{constraint}})/L_{\text{constraint}}$ is the Length Deviation. The curve maps raw overshoot or undershoot to a bounded 0–100 score, punishing under-generation more harshly than over-generation, and every leaderboard number in the paper is computed through it. The supporting diagnostic machinery is the length-awareness experiment, in which models are asked to append their actual word count; the paper uses the gap between reported and true length to argue that word-count unawareness is a proximal cause of instruction failure.

What would settle it

Recompute the Equal To leaderboard with a symmetric penalty curve ($k_1=k_2=2$) and with a linear penalty $LS=100(1-|LD|)$; if the model ordering changes materially, or if the number of models scoring above 60 changes substantially, the paper's headline quantitative claim is an artifact of the chosen curve rather than a fact about the underlying outputs.

Watch

Extended reading notes

Core claim

The paper establishes, on its own terms, that length instruction following is a measurable capability in which current LLMs are broadly deficient. On LIFEBench's 10,800 instances, 23 of 26 models score below 60 on the Equal To Length Score; for constraints of 4,096 words or more, no model consistently exceeds a score of 40. Under extended "At Least" prompts up to 32,768 words, only the Claude and Qwen families reach the requested length even in their longest decile, and most models stop far short of their vendor-claimed maximum output. The authors attribute this to three interacting causes: models cannot accurately self-report how long their own outputs are (reported counts anchor to the instructed length), long inputs further degrade compliance, and some models adopt lazy strategies such as premature termination or outright refusal. Reasoning models like o3-mini improve on this by drafting, counting, and revising during intermediate reasoning, reaching a Length Score of 75.4, yet the same mechanism still breaks down at the longest constraints.

Load-bearing premise

The whole leaderboard rests on one hand-picked scoring curve — the Length Score with $k_1=5$ and $k_2=2$ — and the paper does not show whether the model rankings, or the claim that most models score below 60, survive under a different but equally reasonable penalty curve.

Editorial extensions

If this is right

  • If the paper's results hold, a user who asks for a 10,000-word output should expect a much shorter response or a refusal, and vendors' maximum output length claims should not be treated as achievable targets.
  • Long-context window size and length-instruction following appear to be separate capabilities; expanding context alone will not make models hit a requested length, so training and evaluation must target length control explicitly.
  • Length following is task- and language-dependent: summarization and Chinese prompts degrade scores substantially, so any practical system needs to set expectations per task and language rather than assume a single 'length control' skill.
  • Reasoning models' self-checking behavior (draft, count, revise) is currently the most effective mitigation, but it breaks down past a few thousand words, so inference-time verification alone is not a complete solution.
  • Benchmarks that measure only output quality and not length compliance are missing a failure mode that is visible in LIFEBench; length metrics should join quality metrics in standard LLM evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference, not the paper's claim: the Length Score's hand-chosen asymmetry ($k_1=5$, $k_2=2$) means the leaderboard order is partly determined by the scoring curve; a symmetric curve would compress the gaps, so the 'most models below 60' headline needs a robustness check before it is treated as a stable fact.
  • My inference, not the paper's claim: the length-awareness result suggests a direct training signal — reward models for accurately self-reporting how many words they generated, or for placing a marker every 100 words — that could improve length following; the paper does not test this.
  • My inference, not the paper's claim: because the benchmark scores word count only, a model could game it by padding with repetitive or irrelevant text; combining the Length Score with a quality filter or a repetition detector would give a more complete picture of length instruction following.
  • My inference, not the paper's claim: the finding that Chinese prompts trigger over-generation hints at tokenization or pretraining-corpus biases; a natural next experiment is to see whether the effect disappears when output length is measured in tokens instead of characters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. The paper introduces LIFEBENCH, a benchmark for evaluating length instruction following in LLMs, with 10,800 instances across four NLG task categories, three control methods (Equal To, At Most, At Least), ten length constraints (16 to 8192 words), and English/Chinese data. The authors evaluate 26 models and report that most models fail on long length constraints, that almost all models fall short of their vendor-claimed maximum output lengths, and that reasoning models outperform long-text-specific models. The benchmark is released with code, data, and a dataset card. The central qualitative finding—that precise length following degrades sharply at longer constraints—is supported by raw output word counts and per-constraint tables, while the specific leaderboard numbers and the '23/26 below 60' claim depend on the hand-set penalty constants in the Length Score metric.

Significance. If the quantitative claims are made robust, LIFEBENCH would be a useful community resource: it covers a wider range of lengths and tasks than most prior instruction-following benchmarks, provides bilingual coverage, and includes diagnostic experiments (length awareness, input-length sensitivity, lazy-strategy prevalence) that go beyond simple pass/fail measurement. The paper ships code, data, and a dataset card, and its qualitative conclusion that long length instructions are unreliable is supported by multiple independent evidence streams, including mean output word counts and At Most/At Least success rates. The length-awareness analysis and the distinction between true generation limits and lazy refusal are valuable contributions.

major comments (3)
  1. [§4.3, Eq. (3); Table 1; Figure 1] The entire leaderboard and the headline claim that '23 out of 26 models score below 60' are computed through the Length Score with hand-set constants k1=5 and k2=2, but the paper provides no sensitivity analysis. Because LS is a nonlinear, asymmetric transform of LD, and because per-sample scores are averaged after the transform, different but equally reasonable (k1,k2) choices (e.g., symmetric k1=k2, or reversed asymmetry k1=2,k2=5) can reorder models with different over- vs. under-generation profiles and change the fraction of models below any threshold. This is load-bearing because o3-mini's top ranking and the 'below 60' statement are headline contributions. I ask the authors to report a grid sensitivity analysis over (k1,k2) (covering symmetric and reversed-asymmetry cases) and to state whether the main qualitative claims and the specific leaderboard order are stable.
  2. [§5.1, Table 1] Model-level LS scores are reported as point estimates without confidence intervals or significance tests. Since the benchmark has only 360 unique content units, each repeated across 30 constraint/control combinations, the 10,800 instances are not independent; the effective sample size for model ranking is much smaller. Claims such as o3-mini (75.4) outperforming Claude-3.7-Sonnet-Thinking (61.3) and Gemini-2.5-Pro (60.0) need bootstrap CIs or per-model standard errors to be credible, particularly for models separated by small margins. The authors should add such uncertainty quantification to the leaderboard.
  3. [§5.1, Evaluation Protocol] Empty outputs are excluded from the final results ('approximately 0.1% of instances with empty outputs per model'). Since empty generations and refusals are later characterized as a 'lazy strategy' failure mode (Section 5.3, Appendix H), their exclusion likely biases LS upward for the exact long-constraint cases where models fail. The fraction is small, but the authors should demonstrate robustness by either including empty outputs as LD=-100% in a re-analysis or reporting the exclusion rate per model and per constraint and showing that it does not affect any conclusion.
minor comments (8)
  1. [Table 1] The 'Params' column lists Llama-3.1-70B-Instruct as '8B'; it should be '70B'.
  2. [Table 10] The API entries for Qwen3-32B-Thinking and Qwen3-235B-A22B-Thinking say 'deepseek-reasoner1', which appears to be a copy-paste error from the DeepSeek-R1 row.
  3. [Appendix C.1 and C.2] The headings 'Lable Task' should read 'Label Task'.
  4. [Appendix D.2.1] There is a typo: 'legnth' should be 'length'.
  5. [Figure 8 and several figure captions] Garbled Unicode sequences (e.g., '/uni00000014/...') appear in the rendered text of Figure 8 and elsewhere; the figures and captions should be regenerated with clean font embedding.
  6. [§5.3, paragraph on sensitivity to input length] The text says 'LongWriter-Llama3.1-8B exhibiting the largest Length Deviation reduction (↓ 13.6)', but Figure 6 plots Length Score; the metric name should be corrected to Length Score in that sentence.
  7. [Appendix D.2.2] The section claims to 'demonstrate the rationality of LS and its hyperparameter settings', but it only justifies the choice qualitatively; adding the sensitivity analysis requested above would substantiate this claim.
  8. [Section 3] In the data statistics paragraph, 'LENGTHBENCH' should be 'LIFEBENCH'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LIFEBench's central claims are empirical measurements of model output lengths, and no load-bearing step reduces to its own inputs.

full rationale

This is an empirical benchmark paper rather than a derivational one. The central claims—that most models follow short length instructions but fail long ones, that almost all models fall short of vendor-claimed maximum output lengths, and that reasoning models outperform long-text-enhanced models—are supported by direct measurements of generated word counts against specified constraints. The Length Deviation (LD) metric is a direct ratio of observed output length to instructed length, and Length Score (LS) is a monotone, bounded transformation of LD with hand-chosen constants k1=5 and k2=2; no parameter is fitted to any subset of the data, and no prediction is derived from fitted values. The leaderboard numbers are computed from the chosen scoring curve, but this is a design choice about how to aggregate deviations, not a case where an input is defined in terms of the output or where a fitted quantity is renamed as a prediction. The paper's own raw word-count plots and At Most/At Least tables independently corroborate the qualitative conclusion that long-constraint adherence is poor. Self-citations are not load-bearing: the only noticeable self-citations are to a broad survey of LLM safety and prior length-control works, and the main evaluation does not depend on any such cited result being assumed true. The arbitrary character of k1 and k2 is a robustness/validity concern, not circularity, because alternative scoring curves would change rankings but would not make the underlying measurements self-referential. No step in the derivation chain is equivalent by construction to its inputs, so the circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claims rest on a hand-defined Length Score metric and curated benchmark data. The only fitted or hand-chosen numbers are the scoring constants k1 and k2, which drive the leaderboard and all comparative statements.

free parameters (2)
  • k1 (under-generation penalty constant) = 5
    Hand-chosen in the Length Score definition (Eq. 3, Section 4.3) to penalize under-generation more heavily than over-generation; all model rankings depend on this curve.
  • k2 (over-generation penalty constant) = 2
    Hand-chosen in Eq. 3 to give slower score decay for over-length outputs; no robustness analysis is provided despite this shaping every reported Length Score.
assumptions (3)
  • domain assumption Chinese output length measured in characters, English in words, and the two summed for mixed text gives a comparable length measure across languages.
    Appendix D.2.1 defines the counting rule; language comparisons such as the Chinese over-generation finding depend on this comparability.
  • domain assumption The four selected task categories (QA, Summarization, Reasoning, Creative Generation) represent the flexible-length NLG tasks where length instruction following is meaningful.
    Section 3 and Appendix B.1 justify this via the Flexible-Length Paradigm; if other task types behave differently, benchmark conclusions may not generalize.
  • domain assumption GPT-4o-generated reasoning problems are valid, unbiased flexible-length reasoning items after manual review.
    Reasoning data are AI-generated (Section 3, Appendix B.2); the paper reports manual quality checks but does not independently validate neutrality or difficulty.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LIFEBench: Evaluating Length Instruction Following in Large Language Models." pith.science (2026). https://pith.science/paper/AXCPT52C

@misc{pith2026250516234,
  author       = {Pith},
  title        = {Pith review of: LIFEBench: Evaluating Length Instruction Following in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AXCPT52C}},
  note         = {Machine review of arXiv:2505.16234}
}
read the original abstract

While large language models (LLMs) can solve PhD-level reasoning problems over long context inputs, they still struggle with a seemingly simpler task: following explicit length instructions-e.g., write a 10,000-word novel. Additionally, models often generate far too short outputs, terminate prematurely, or even refuse the request. Existing benchmarks focus primarily on evaluating generations quality, but often overlook whether the generations meet length constraints. To this end, we introduce Length Instruction Following Evaluation Benchmark (LIFEBench) to comprehensively evaluate LLMs' ability to follow length instructions across diverse tasks and a wide range of specified lengths. LIFEBench consists of 10,800 instances across 4 task categories in both English and Chinese, covering length constraints ranging from 16 to 8192 words. We evaluate 26 widely-used LLMs and find that most models reasonably follow short-length instructions but deteriorate sharply beyond a certain threshold. Surprisingly, almost all models fail to reach the vendor-claimed maximum output lengths in practice, as further confirmed by our evaluations extending up to 32K words. Even long-context LLMs, despite their extended input-output windows, counterintuitively fail to improve length-instructions following. Notably, Reasoning LLMs outperform even specialized long-text generation models, achieving state-of-the-art length following. Overall, LIFEBench uncovers fundamental limitations in current LLMs' length instructions following ability, offering critical insights for future progress.

Figures

Figures reproduced from arXiv: 2505.16234 by the authors.

Figure 1
Figure 1. Overview of LIFEBENCH. Top Left. LIFEBENCH data types. Top Right. Length Instruction Following Leaderboard based on LIFEBENCH. Bottom. LIFEBENCH evaluates length generalization capabilities across bilingual content (Chinese/English), incorporating three distinct control methodologies and ten granular length constraints. • RQ1: To what extent do current LLMs accurately follow explicit length instructions, and what fa… view at source ↗
Figure 2
Figure 2. Left. Detailed task categories distribution in LIFEBENCH, spanning more than 20 diverse subtypes. Right. Input length distribution for each task category. • Question Answering (QA) Task: The QA task requires the model to generate answers based on given questions. The answer length can vary flexibly depending on the level of detail required. We collect questions from six representative QA datasets [53, 1, 54, 118, 74… view at source ↗
Figure 3
Figure 3. Left. Length Score across all length constraints under the Equal To control method. Impact of task type (Middle) and language (Right) on the Length Score, separated by model type [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (27 more)
Figure 5
Figure 5. Figure 5: Length Awareness Experiment: The real output word count and self-reported word count are shown for cases where the length constraint exceeds 128, with both axes in log scale. “Real Fit” and “Reported Fit” represent the least squares regression results for real and self…
Figure 6
Figure 6. Figure 6: Impact of long input on the Length Score [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 8
Figure 8. Figure 8: Visualization of Length Score for different control methods. score rapidly approaches zero, achieving values such as 0.7 and 0.2, respectively. This indicates that distinctions among significantly poor cases are appropriately minimized, aligning the metric’s behavior w…
Figure 9
Figure 9. Figure 9: Output word count for all length constraints under the [PITH_FULL_IMAGE:figures/full_fig_p043_9.png]
Figure 10
Figure 10. Figure 10: Output word count for all length constraints under the [PITH_FULL_IMAGE:figures/full_fig_p043_10.png]
Figure 11
Figure 11. Figure 11: Output word count for all length constraints under the [PITH_FULL_IMAGE:figures/full_fig_p043_11.png]
Figure 12
Figure 12. Figure 12: Length Awareness Experiment Results of All Models: The real output word count and [PITH_FULL_IMAGE:figures/full_fig_p047_12.png]
Figure 13
Figure 13. Figure 13: Empirical Cumulative Distribution Function (ECDF) of marker positions in generated [PITH_FULL_IMAGE:figures/full_fig_p048_13.png]
Figure 14
Figure 14. Figure 14: Distribution of inter-marker intervals in outputs: Each box plot shows the distribution of [PITH_FULL_IMAGE:figures/full_fig_p049_14.png]
Figure 15
Figure 15. Figure 15: , [PITH_FULL_IMAGE:figures/full_fig_p051_15.png]
Figure 16
Figure 16. Figure 16: Length Score for Summarization across different length constraints under the Equal To control method. GPT-4o mini GPT-4o o1-mini o3-mini Claude-3.7-Sonnet Claude-3.7-Sonnet-Thinking Gemini-2.0-Flash Gemini-2.0-Flash-Thinking Gemini-2.5-Pro Doubao-1.5-Pro Doubao-1.5-Th…
Figure 17
Figure 17. Figure 17: Length Score for Reasoning across different length constraints under the Equal To control method. 52 [PITH_FULL_IMAGE:figures/full_fig_p052_17.png]
Figure 18
Figure 18. Figure 18: Length Score for Creative Generation across different length constraints under the Equal To control method. G.2 Input Length Analysis across task categories (as shown in [PITH_FULL_IMAGE:figures/full_fig_p053_18.png]
Figure 19
Figure 19. Figure 19: Length Score for Short Input across different length constraints under the Equal To control method. GPT-4o mini GPT-4o o1-mini o3-mini Claude-3.7-Sonnet Claude-3.7-Sonnet-Thinking Gemini-2.0-Flash Gemini-2.0-Flash-Thinking Gemini-2.5-Pro Doubao-1.5-Pro Doubao-1.5-Thin…
Figure 20
Figure 20. Figure 20: Length Score for Medium Input across different length constraints under the Equal To control method. 54 [PITH_FULL_IMAGE:figures/full_fig_p054_20.png]
Figure 21
Figure 21. Figure 21: Length Score for Long Input across different length constraints under the Equal To control method. G.3 Language Language bias may also contribute to the limitations observed in length instruction following. For example, one plausible hypothesis is that length instruct…
Figure 22
Figure 22. Figure 22: Length Score for English across different length constraints under the Equal To control method. GPT-4o mini GPT-4o o1-mini o3-mini Claude-3.7-Sonnet Claude-3.7-Sonnet-Thinking Gemini-2.0-Flash Gemini-2.0-Flash-Thinking Gemini-2.5-Pro Doubao-1.5-Pro Doubao-1.5-Thinking…
Figure 23
Figure 23. Figure 23: Length Score for Chinese across different length constraints under the Equal To control method. 56 [PITH_FULL_IMAGE:figures/full_fig_p056_23.png]
Figure 24
Figure 24. Figure 24: (Left) Average proportion of lazy strategies across all models under varying length constraints (Right) Proportion of lazy strategies exhibited by different models under varying length constraints. Results are reported for the case where the control method is set to E…
Figure 25
Figure 25. Figure 25: and [PITH_FULL_IMAGE:figures/full_fig_p067_25.png]
Figure 26
Figure 26. Figure 26: Length Score for Refactor across different length constraints under the Equal To control method. 68 [PITH_FULL_IMAGE:figures/full_fig_p068_26.png]
Figure 27
Figure 27. Figure 27: We compare 20 models with the increase of length instruction in terms of their textual [PITH_FULL_IMAGE:figures/full_fig_p071_27.png]
Figure 28
Figure 28. Figure 28: The horizontal axis is the actual length generated, and the vertical axis is the Average Score. [PITH_FULL_IMAGE:figures/full_fig_p073_28.png]
Figure 29
Figure 29. Figure 29: Under the four length constraints of [1024, 2048, 4096, 8192], different models generate average histograms of text quality. We marked the Mean, Max and Min reference lines of the bar in each sub-graph. The results are presented in [PITH_FULL_IMAGE:figures/full_fig_p…
Figure 30
Figure 30. Figure 30: Comparison of Length Score with the Arena Scores. P denotes the Pearson correlation coefficient, and S denotes the Spearman correlation coefficient. Reasoning models are indicated by a star marker, while non-reasoning models use a circle marker. “w/SC” stands for “wit…
Figure 31
Figure 31. Figure 31: This figure illustrates the relationship between the model’s actual output length and the [PITH_FULL_IMAGE:figures/full_fig_p077_31.png]
Figure 32
Figure 32. Figure 32: Output length distributions of base models. The gray dashed line denotes the length [PITH_FULL_IMAGE:figures/full_fig_p078_32.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Librarian Who Refused to Code: Model-Dependent Identity Enactment in LLM Code Generation

    cs.CL 2026-07 conditional novelty 7.0 of 10

    A controlled pre-registered experiment shows that biographical personas change code-generation behavior in one frontier model (including refusals to code) but not another, acting as model-dependent behavioral-policy biases.

  2. Length Value Model: Scalable Value Pretraining for Token-Level Length Modeling

    cs.CL 2026-04 unverdicted novelty 6.0 of 10

    LenVM trains a token-level value head to predict discounted remaining length, enabling length control and efficiency steering on LLMs and VLMs.

Reference graph

Works this paper leans on

129 extracted references · 47 canonical work pages · cited by 2 Pith papers

  1. [1]

    Abedi Firouzjaei

    H. Abedi Firouzjaei. Survival analysis for user disengagement prediction: question-and- answering communities’ case.Social Network Analysis and Mining, 12(1):86, 2022

    2022

  2. [2]

    Alzantot, Y

    M. Alzantot, Y . S. Sharma, A. Elgohary, B.-J. Ho, M. Srivastava, and K.-W. Chang. Generating natural language adversarial examples. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018

    2018

  3. [3]

    C. An, S. Gong, M. Zhong, X. Zhao, M. Li, J. Zhang, L. Kong, and X. Qiu. L-eval: Instituting standardized evaluation for long context language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14388–14411, 2024

    2024

  4. [4]

    Claude 3.7 Sonnet and Claude Code

    Anthropic. Claude 3.7 Sonnet and Claude Code. https://www.anthropic.com/news/ claude-3-7-sonnet, 2025

    2025

  5. [5]

    G. Attardi. Ag’s corpus of news articles. http://groups.di.unipi.it/~gulli/AG_ corpus_of_news_articles.html, 2004. Accessed: 2025-05-09

    2004

  6. [6]

    Y . Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3119–3137, 2024

    2024

  7. [7]

    Y . Bai, S. Tu, J. Zhang, H. Peng, X. Wang, X. Lv, S. Cao, J. Xu, L. Hou, Y . Dong, et al. Long- bench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv preprint arXiv:2412.15204, 2024

    arXiv 2024

  8. [8]

    Y . Bai, J. Zhang, X. Lv, L. Zheng, S. Zhu, L. Hou, Y . Dong, J. Tang, and J. Li. Longwriter: Unleashing 10,000+ word generation from long context LLMs. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

Show all 129 references
  1. [9]

    Bordes, Y .-L

    A. Bordes, Y .-L. Boureau, and J. Weston. Learning end-to-end goal-oriented dialog. In International Conference on Learning Representations, 2017

    2017

  2. [10]

    Bosselut, A

    A. Bosselut, A. Celikyilmaz, X. He, J. Gao, P.-S. Huang, and Y . Choi. Discourse-aware neural rewards for coherent text generation. InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,...

    2018

  3. [11]

    Butcher, M

    B. Butcher, M. O’Keefe, and J. Titchener. Precise length control for large language models. Natural Language Processing Journal, page 100143, 2025

    2025

  4. [12]

    Doubao-1.5-Pro

    ByteDance. Doubao-1.5-Pro. https://seed.bytedance.com/zh/special/doubao_1_ 5_pro/, 2025

    2025

  5. [13]

    Chang, X

    Y . Chang, X. Wang, J. Wang, Y . Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y . Wang, et al. A survey on evaluation of large language models.ACM transactions on intelligent systems and technology, 15(3):1–45, 2024

    2024

  6. [14]

    D. S. Chaplot. Albert q. jiang, alexandre sablayrolles, arthur mensch, chris bamford, devendra singh chaplot, diego de las casas, florian bressand, gianna lengyel, guillaume lample, lucile saulnier, lélio renard lavaud, marie-anne lachaux, pierre stock, teven le scao, thibaut ...

    arXiv

  7. [15]

    D. Chen, A. Fisch, J. Weston, and A. Bordes. Reading wikipedia to answer open-domain questions. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870–1879, 2017. 10

    2017

  8. [16]

    Chen and C

    D. Chen and C. D. Manning. A fast and accurate dependency parser using neural networks. InProceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 740–750, 2014

    2014

  9. [17]

    Y . Chen, B. Xu, Q. Wang, Y . Liu, and Z. Mao. Benchmarking large language models on controllable generation under diversified instructions. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17808–17816, 2024

    2024

  10. [18]

    Chiang, L

    W.-L. Chiang, L. Zheng, Y . Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez, and I. Stoica. Chatbot arena: An open platform for evaluating llms by human preference, 2024

    2024

  11. [19]

    W. S. Cho, P. Zhang, Y . Zhang, X. Li, M. Galley, C. Brockett, M. Wang, and J. Gao. Towards coherent and cohesive long-form text generation. InProceedings of the First Workshop on Narrative Understanding, pages 1–11, 2019

    2019

  12. [20]

    Cobbe, V

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    2021 arXiv

  13. [21]

    Cohan, F

    A. Cohan, F. Dernoncourt, D. S. Kim, T. Bui, S. Kim, W. Chang, and N. Goharian. A discourse- aware attention model for abstractive summarization of long documents. InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisti...

    2018

  14. [22]

    Collobert, J

    R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa. Natural language processing (almost) from scratch.Journal of Machine Learning Research, 12:2493–2537, 2011

    2011

  15. [23]

    M. R. Costa-jussà, P. Andrews, M. C. Meglioli, J. Chen, J. Chuang, D. Dale, C. Ropers, A. Mourachko, E. Sánchez, H. Schwenk, et al. Lcfo: Long context and long form output dataset and benchmarking.arXiv preprint arXiv:2412.08268, 2024

    2024 arXiv

  16. [24]

    Davidson, D

    T. Davidson, D. Warmsley, M. Macy, and I. Weber. Automated hate speech detection and the problem of offensive language. InProceedings of the international AAAI conference on web and social media, volume 11, pages 512–515, 2017

    2017

  17. [25]

    S. J. Delany, M. Buckley, and D. Greene. Sms spam filtering: Methods and data.Expert Systems with Applications, 39(10):9899–9908, 2012

    2012

  18. [26]

    L. Dong, J. Mallinson, S. Reddy, and M. Lapata. Learning to paraphrase for question answering. InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 875–886, 2017

    2017

  19. [27]

    Dubois, C

    Y . Dubois, C. X. Li, R. Taori, T. Zhang, I. Gulrajani, J. Ba, C. Guestrin, P. S. Liang, and T. B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback.Advances in Neural Information Processing Systems, 36:30039–30069, 2023

    2023

  20. [28]

    A. R. Fabbri, I. Li, T. She, S. Li, and D. Radev. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074–1084, 2019

    2019

  21. [29]

    A. Fan, D. Grangier, and M. Auli. Controllable abstractive summarization. InProceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 45–54, 2018

    2018

  22. [30]

    A. Fan, Y . Jernite, E. Perez, D. Grangier, J. Weston, and M. Auli. Eli5: Long form question answering. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558–3567, 2019

    2019

  23. [31]

    A. Fan, M. Lewis, and Y . Dauphin. Hierarchical neural story generation. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898, 2018. 11

    2018

  24. [32]

    Foundation

    W. Foundation. Wikimedia downloads. https://dumps.wikimedia.org. Accessed: 2025- 03-11

    2025

  25. [33]

    T. GLM, A. Zeng, B. Xu, B. Wang, C. Zhang, D. Yin, D. Zhang, D. Rojas, G. Feng, H. Zhao, et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools.arXiv preprint arXiv:2406.12793, 2024

    2024 arXiv

  26. [34]

    Gemini 2.0 Flash

    Google. Gemini 2.0 Flash. https://deepmind.google/technologies/gemini/flash/, 2024

    2024

  27. [35]

    Gemini 2.5 Pro

    Google. Gemini 2.5 Pro. https://deepmind.google/technologies/gemini/pro/, 2025

    2025

  28. [36]

    Grattafiori, A

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  29. [37]

    A. Gu, K. Goel, and C. Re. Efficiently modeling long sequences with structured state spaces. InInternational Conference on Learning Representations, 2022

    2022

  30. [38]

    J. Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, W. Li, Y . Shen, S. Ma, H. Liu, et al. A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594, 2024

    2024 arXiv

  31. [39]

    Y . Gu, W. Wang, X. Feng, W. Zhong, K. Zhu, L. Huang, T.-S. Chua, and B. Qin. Length controlled generation for black-box llms.arXiv preprint arXiv:2412.14656, 2024

    2024 arXiv

  32. [40]

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  33. [41]

    Y . He, D. Jin, C. Wang, C. Bi, K. Mandyam, H. Zhang, C. Zhu, N. Li, T. Xu, H. Lv, et al. Multi-if: Benchmarking llms on multi-turn and multilingual instructions following.arXiv preprint arXiv:2410.15553, 2024

    2024 arXiv

  34. [42]

    K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blun- som. Teaching machines to read and comprehend.Advances in neural information processing systems, 28, 2015

    2015

  35. [43]

    Hsieh, S

    C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, and B. Ginsburg. Ruler: What’s the real context size of your long-context language models? InFirst Conference on Language Modeling, 2024

    2024

  36. [44]

    Huang and K

    J. Huang and K. C.-C. Chang. Towards reasoning in large language models: A survey. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1049–1065, 2023

    2023

  37. [45]

    Huang, X

    P.-S. Huang, X. He, J. Gao, L. Deng, A. Acero, and L. Heck. Learning deep structured semantic models for web search using clickthrough data. InProceedings of the 22nd ACM international conference on Information & Knowledge Management, pages 2333–2338, 2013

    2013

  38. [46]

    Huang, K

    Y . Huang, K. Tang, M. Chen, and B. Wang. A comprehensive survey on evaluating large language model applications in the medical industry.arXiv preprint arXiv:2404.15777, 2024

    2024 arXiv

  39. [47]

    Jacovi, A

    A. Jacovi, A. Wang, C. Alberti, C. Tao, J. Lipovetz, K. Olszewska, L. Haas, M. Liu, N. Keating, A. Bloniarz, et al. The facts grounding leaderboard: Benchmarking llms’ ability to ground responses to long-form input.arXiv preprint arXiv:2501.03200, 2025

    2025 arXiv

  40. [48]

    Jaech, A

    A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  41. [49]

    Jhamtani, V

    H. Jhamtani, V . Gangal, E. Hovy, and E. Nyberg. Shakespearizing modern language using copy-enriched sequence to sequence models. InProceedings of the Workshop on Stylistic Variation, pages 10–19, 2017. 12

    2017

  42. [50]

    R. Jie, X. Meng, L. Shang, X. Jiang, and Q. Liu. Prompt-based length controlled generation with multiple control types. InFindings of the Association for Computational Linguistics ACL 2024, pages 1067–1085, 2024

    2024

  43. [51]

    webnovel_cn (revision 745338c), 2023

    Jun. webnovel_cn (revision 745338c), 2023

    2023

  44. [52]

    G. Kamradt. Needle in a haystack - pressure testing llms, 2023. Accessed: 2025-04-22

    2023

  45. [53]

    Q. E. Kim. Question pairs dataset. https://www.kaggle.com/datasets/quora/ question-pairs-dataset/data, 2017

    2017

  46. [54]

    Koupaee and W

    M. Koupaee and W. Y . Wang. Wikihow: A large scale text summarization dataset, 2018

    2018

  47. [55]

    Kry´sci´nski, N

    W. Kry´sci´nski, N. Rajani, D. Agarwal, C. Xiong, and D. Radev. Booksum: A collection of datasets for long-form narrative summarization. InFindings of the Association for Computa- tional Linguistics: EMNLP 2022, pages 6536–6558, 2022

    2022

  48. [56]

    Kuratov, A

    Y . Kuratov, A. Bulatov, P. Anokhin, I. Rodkin, D. Sorokin, A. Sorokin, and M. Burtsev. Babilong: Testing the limits of llms with long context reasoning-in-a-haystack.Advances in Neural Information Processing Systems, 37:106519–106554, 2024

    2024

  49. [57]

    Lample, M

    G. Lample, M. Ballesteros, S. Subramanian, K. Kawakami, and C. Dyer. Neural architectures for named entity recognition. InProceedings of NAACL-HLT, pages 260–270, 2016

    2016

  50. [58]

    J. Li, L. Zhang, Y . Li, Z. Liu, Y . Bai, R. Luo, L. Chen, and M. Yang. Ruler: A model-agnostic method to control generated length for large language models. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 3042–3059, 2024

    2024

  51. [59]

    T. Li, G. Zhang, Q. D. Do, X. Yue, and W. Chen. Long-context llms struggle with long in-context learning.arXiv preprint arXiv:2404.02060, 2024

    2024 arXiv

  52. [60]

    X. Li, H. Shi, R. Xu, and W. Xu. Ai awareness.arXiv preprint arXiv:2504.20084, 2025

    2025 arXiv

  53. [61]

    Z. Li, X. Jiang, L. Shang, and H. Li. Paraphrase generation with deep reinforcement learning. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3865–3878, 2018

    2018

  54. [62]

    Liang, H

    X. Liang, H. Wang, Y . Wang, S. Song, J. Yang, S. Niu, J. Hu, D. Liu, S. Yao, F. Xiong, et al. Controllable text generation for large language models: A survey.arXiv preprint arXiv:2408.12599, 2024

    2024 arXiv

  55. [63]

    Lightman, V

    H. Lightman, V . Kosaraju, Y . Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe. Let’s verify step by step. InThe Twelfth International Conference on Learning Representations, 2023

    2023

  56. [64]

    B. Y . Lin, A. Ravichander, X. Lu, N. Dziri, M. Sclar, K. Chandu, C. Bhagavatula, and Y . Choi. The unlocking spell on base LLMs: Rethinking alignment via in-context learning. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  57. [65]

    Z. Ling, K. Liu, K. Yan, Y . Yang, W. Lin, T.-H. Fan, L. Shen, Z. Du, and J. Chen. Longrea- son: A synthetic long-context reasoning benchmark via context expansion.arXiv preprint arXiv:2501.15089, 2025

    2025

  58. [66]

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  59. [67]

    J. Liu, D. Zhu, Z. Bai, Y . He, H. Liao, H. Que, Z. Wang, C. Zhang, G. Zhang, J. Zhang, et al. A comprehensive survey on long context language modeling.arXiv preprint arXiv:2503.17407, 2025

    2025

  60. [68]

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang. Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics, 12:157–173, 2024. 13

    2024

  61. [69]

    X. Liu, P. Dong, X. Hu, and X. Chu. Longgenbench: Long-context generation benchmark. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 865–883, 2024

    2024

  62. [70]

    Q. Luo, Y . Ye, S. Liang, Z. Zhang, Y . Qin, Y . Lu, Y . Wu, X. Cong, Y . Lin, Y . Zhang, et al. Repoagent: An llm-powered open-source framework for repository-level code documentation generation. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Pr...

    2024

  63. [71]

    Malaviya, S

    C. Malaviya, S. Lee, S. Chen, E. Sieber, M. Yatskar, and D. Roth. Expertqa: Expert-curated questions and attributed answers.arXiv preprint arXiv:2309.07852, 2023

    2023 arXiv

  64. [72]

    J. J. McAuley and J. Leskovec. From amateurs to connoisseurs: modeling the evolution of user expertise through online reviews. InProceedings of the 22nd international conference on World Wide Web, pages 897–908, 2013

    2013

  65. [73]

    Chinesenlpcorpus

    MOP-LIWU Community and MNBVC Team. Chinesenlpcorpus. https://github.com/ SophonPlus/ChineseNlpCorpus/tree/master?tab=readme-ov-file, 2018

    2018

  66. [74]

    Mnbvc: Massive never-ending bt vast chinese corpus.https://github.com/esbatmop/MNBVC, 2023

    MOP-LIWU Community and MNBVC Team. Mnbvc: Massive never-ending bt vast chinese corpus.https://github.com/esbatmop/MNBVC, 2023

    2023

  67. [75]

    Mostafazadeh, N

    N. Mostafazadeh, N. Chambers, X. He, D. Parikh, D. Batra, L. Vanderwende, P. Kohli, and J. Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computati...

    2016

  68. [76]

    Nallapati, B

    R. Nallapati, B. Zhou, C. dos Santos, C. Gulcehre, and B. Xiang. Abstractive text summa- rization using sequence-to-sequence rnns and beyond. InProceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290, 2016

    2016

  69. [77]

    GPT-4o mini: advancing cost-efficient intelligence

    OpenAI. GPT-4o mini: advancing cost-efficient intelligence. https://openai.com/index/ gpt-4o-mini-advancing-cost-efficient-intelligence/, 2024

    2024

  70. [78]

    Hello GPT-4o.https://openai.com/index/hello-gpt-4o/, 2024

    OpenAI. Hello GPT-4o.https://openai.com/index/hello-gpt-4o/, 2024

    2024

  71. [79]

    OpenAI o1-mini: Advancing cost-efficient reasoning

    OpenAI. OpenAI o1-mini: Advancing cost-efficient reasoning. https://openai.com/ index/openai-o1-mini-advancing-cost-efficient-reasoning/, 2024

    2024

  72. [80]

    OpenAI o3-mini: Pushing the frontier of cost-effective reasoning

    OpenAI. OpenAI o3-mini: Pushing the frontier of cost-effective reasoning. https://openai. com/index/openai-o3-mini/, 2025

    2025

  73. [81]

    S. J. Paech. Eq-bench creative writing benchmark v3. https://github.com/EQ-bench/ creative-writing-bench, 2025

    2025

  74. [82]

    P. Patil. The internet movie script dataset. https://www.kaggle.com/datasets/ prasad22/the-internet-movie-script-dataset, 2024

    2024

  75. [83]

    C. Pham, S. Sun, and M. Iyyer. Suri: Multi-constraint instruction following in long-form text generation. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 1722–1753, 2024

    2024

  76. [84]

    Y . Qin, K. Song, Y . Hu, W. Yao, S. Cho, X. Wang, X. Wu, F. Liu, P. Liu, and D. Yu. Infobench: Evaluating instruction following ability in large language models. InFindings of the Association for Computational Linguistics ACL 2024, pages 13025–13048, 2024

    2024

  77. [85]

    S. Quan, T. Tang, B. Yu, A. Yang, D. Liu, B. Gao, J. Tu, Y . Zhang, J. Zhou, and J. Lin. Language models can self-lengthen to generate long texts.arXiv preprint arXiv:2410.23933, 2024

    2024 arXiv

  78. [86]

    H. Que, F. Duan, L. He, Y . Mou, W. Zhou, J. Liu, W. Rong, Z. M. Wang, J. Yang, G. Zhang, et al. Hellobench: Evaluating long text generation capabilities of large language models.arXiv preprint arXiv:2409.16191, 2024. 14

    2024 arXiv

  79. [87]

    Radford and K

    A. Radford and K. Narasimhan. Improving language understanding by generative pre- training. https://cdn.openai.com/research-covers/language-unsupervised/ language_understanding_paper.pdf, 2018

    2018

  80. [88]

    Rafailov, A

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    2023

  81. [89]

    A. M. Rush, S. Chopra, and J. Weston. A neural attention model for sentence summarization. InConference on Empirical Methods in Natural Language Processing, EMNLP 2015, pages 379–389. Association for Computational Linguistics (ACL), 2015

    2015

  82. [90]

    H. Schmid. Part-of-speech tagging with neural networks. InProceedings of the 15th conference on Computational linguistics-Volume 1, pages 172–176, 1994

    1994

  83. [91]

    Sennrich, B

    R. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, 2016

    2016

  84. [92]

    Shaham, M

    U. Shaham, M. Ivgi, A. Efrat, J. Berant, and O. Levy. Zeroscrolls: A zero-shot benchmark for long text understanding. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 7977–7989, 2023

    2023

  85. [93]

    Socher, A

    R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y . Ng, and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. InProceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642, 2013

    2013

  86. [94]

    D. Srirag. Text classification on emails. https://www.kaggle.com/datasets/ dipankarsrirag/topic-modelling-on-emails, 2020

    2020

  87. [95]

    Sutskever, O

    I. Sutskever, O. Vinyals, and Q. V . Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27, 2014

    2014

  88. [96]

    Talmor, J

    A. Talmor, J. Herzig, N. Lourie, and J. Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volu...

    2019

  89. [97]

    H. Tan, Z. Guo, Z. Shi, L. Xu, Z. Liu, Y . Feng, X. Li, Y . Wang, L. Shang, Q. Liu, et al. Proxyqa: An alternative framework for evaluating long-form text generation with large language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Lingu...

    2024

  90. [98]

    Venkatraman, N

    S. Venkatraman, N. I. Tripto, and D. Lee. Collabstory: Multi-llm collaborative story generation and authorship analysis.arXiv preprint arXiv:2406.12665, 2024

    2024 arXiv

  91. [99]

    C. Wang, H. Duan, S. Zhang, D. Lin, and K. Chen. Ada-leval: Evaluating long-context llms with length-adaptable benchmarks. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1...

    2024

  92. [100]

    K. Wang, G. Zhang, Z. Zhou, J. Wu, M. Yu, S. Zhao, C. Yin, J. Fu, Y . Yan, H. Luo, et al. A comprehensive survey in llm (-agent) full stack safety: Data, training and deployment.arXiv preprint arXiv:2504.15585, 2025

    2025 arXiv

  93. [101]

    Y . Wang, D. Ma, and D. Cai. With greater text comes greater necessity: Inference-time training helps long text generation. InFirst Conference on Language Modeling, 2024

    2024

  94. [102]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou, et al. Chain-of- thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022. 15

    2022

  95. [103]

    B. Wen, P. Ke, X. Gu, L. Wu, H. Huang, J. Zhou, W. Li, B. Hu, W. Gao, J. Xu, et al. Benchmarking complex instruction-following with multiple constraints composition.Advances in Neural Information Processing Systems, 37:137610–137645, 2024

    2024

  96. [104]

    Wiseman, S

    S. Wiseman, S. M. Shieber, and A. M. Rush. Challenges in data-to-document generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2253–2263, 2017

    2017

  97. [105]

    Y . Wu, M. S. Hee, Z. Hu, and R. K.-W. Lee. Longgenbench: Benchmarking long-form generation in long context LLMs. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  98. [106]

    Y . Wu, J. Mei, M. Yan, C. Li, S. Lai, Y . Ren, Z. Wang, J. Zhang, M. Wu, Q. Jin, et al. Writing- bench: A comprehensive benchmark for generative writing.arXiv preprint arXiv:2503.05244, 2025

    2025

  99. [107]

    A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  100. [108]

    K. Yang, Y . Tian, N. Peng, and D. Klein. Re3: Generating longer stories with recursive reprompting and revision. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4393–4479, 2022

    2022

  101. [109]

    Z. Yang, F. Xiong, P. Li, and Y . Liu. Rethinking long context generation from the continual learning perspective. InProceedings of the 31st International Conference on Computational Linguistics, pages 1922–1933, 2025

    1922

  102. [110]

    S. Yao, H. Chen, A. W. Hanjie, R. Yang, and K. R. Narasimhan. COLLIE: Systematic construction of constrained text generation tasks. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  103. [111]

    X. Ye, F. Yin, Y . He, J. Zhang, H. Yen, T. Gao, G. Durrett, and D. Chen. Longproc: Benchmarking long-context language models on long procedural generation.arXiv preprint arXiv:2501.05414, 2025

    2025

  104. [112]

    P. Yuan, C. Tan, S. Feng, Y . Li, X. Wang, Y . Zhang, J. Shi, B. Pan, Y . Hu, and K. Li. From sub-ability diagnosis to human-aligned generation: Bridging the gap for text length control via markergen.arXiv preprint arXiv:2502.13544, 2025

    2025 arXiv

  105. [113]

    W. Yuan, I. Kulikov, P. Yu, K. Cho, S. Sukhbaatar, J. Weston, and J. Xu. Following length constraints in instructions.arXiv preprint arXiv:2406.17744, 2024

    2024 arXiv

  106. [114]

    Z. Zeng, J. Yu, T. Gao, Y . Meng, T. Goyal, and D. Chen. Evaluating large language models at evaluating instruction following. In12th International Conference on Learning Representa- tions, ICLR 2024, 2024

    2024

  107. [115]

    Zhang, R

    J. Zhang, R. Zhang, F. Kong, Z. Miao, Y . Ye, and Y . Zheng. Lost-in-the-middle in long- text generation: Synthetic dataset, evaluation framework, and mitigation.arXiv preprint arXiv:2503.06868, 2025

    2025 arXiv

  108. [116]

    Zhang, Y

    T. Zhang, Y . Shen, W. Luo, Y . Zhang, H. Liang, F. Yang, M. Lin, Y . Qiao, W. Chen, B. Cui, et al. Cfbench: A comprehensive constraints-following benchmark for llms.arXiv preprint arXiv:2408.01122, 2024

    2024

  109. [117]

    Zhang and M

    X. Zhang and M. Lapata. Chinese poetry generation with recurrent neural networks. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 670–680. Association for Computational Linguistics, 2014

    2014

  110. [118]

    Zhang, J

    X. Zhang, J. Zhao, and Y . LeCun. Character-level convolutional networks for text classification. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015

    2015

  111. [119]

    Zheng, W.-L

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023. 16

    2023

  112. [120]

    Zhong, D

    M. Zhong, D. Yin, T. Yu, A. Zaidi, M. Mutuma, R. Jha, A. Hassan, A. Celikyilmaz, Y . Liu, X. Qiu, et al. Qmsum: A new benchmark for query-based multi-domain meeting summarization. InProceedings of the 2021 Conference of the North American Chapter of the Association for Computa...

    2021

  113. [121]

    How to use git efficiently

    J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y . Luan, D. Zhou, and L. Hou. Instruction- following evaluation for large language models.arXiv preprint arXiv:2311.07911, 2023. 17 Appendices Appendix Table of Contents A Comparison of LIFEBENCHwith Existing Benchmarks 20 B Deta...

    2023 arXiv

  114. [122]

    We packed snorkels for the beach, but after traffic delays, found only high surfing waves

    arrived 9. with 10. snorkel 11. gear 12. to 13. find 14. only 15. surfing 16. waves That’s exactly 16 words and it captures the main idea of the original story: they planned for a beach day with snorkeling, but when they arrived, the conditions were only good for surfing, not ...

  115. [123]

    We packed snorkels for the perfect beach day, but after traffic delays, found only surfing waves

    packed 3. snorkels 4. for 5. the 6. perfect 7. beach 8. day 9. but 10. after 11. traffic 12. 62 delays 13. found 14. only 15. surfing 16. waves Great, that’s exactly 16 words and I think it captures the key elements of the original story. I’ll go with this: "We packed snorkels...

  116. [124]

    Well, I guess the universe had other plans for us today,

    Conflict/Problem: Reaction to not being able to snorkel as planned 6. Attempting solutions: 64 What they try to do instead 7. Complications: Additional challenges that arise 8. Resolution: How they adapt and what they learn 9. Conclusion: Reflections on the experience and retu...

  117. [125]

    Give a high score if the response generally follows the intent and subject of the instruction, even if not all minor aspects are covered

    Relevance: Evaluate how well the content stays on-topic and addresses the main theme of the user’s request, regardless of any word or length limits specified by the user. Give a high score if the response generally follows the intent and subject of the instruction, even if not...

  118. [126]

    Accuracy: Score from content that is completely accurate, with no factual errors or misleading information, to content with numerous errors and highly misleading information

  119. [127]

    Coherence: Score from a clear structure with smooth logical connections, to a disorganized structure with no coherence

  120. [128]

    Clarity: Score from clear, detailed, and easy-to-understand language, to confusing expres- sion with minimal details

  121. [129]

    Analysis

    Breadth and Depth: Score from content that is both broad and deep, providing a lot of information, to content that is seriously lacking in breadth and depth, with minimal information. Please evaluate the quality of the following response to a user’s request according to the ab...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.