REVIEW 12 cited by
LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Current long context large language models (LLMs) can process inputs up to 100,000 tokens, yet struggle to generate outputs exceeding even a modest length of 2,000 words. Through controlled experiments, we find that the model's effective generation length is inherently bounded by the sample it has seen during supervised fine-tuning (SFT). In other words, their output limitation is due to the scarcity of long-output examples in existing SFT datasets. To address this, we introduce AgentWrite, an agent-based pipeline that decomposes ultra-long generation tasks into subtasks, enabling off-the-shelf LLMs to generate coherent outputs exceeding 20,000 words. Leveraging AgentWrite, we construct LongWriter-6k, a dataset containing 6,000 SFT data with output lengths ranging from 2k to 32k words. By incorporating this dataset into model training, we successfully scale the output length of existing models to over 10,000 words while maintaining output quality. We also develop LongBench-Write, a comprehensive benchmark for evaluating ultra-long generation capabilities. Our 9B parameter model, further improved through DPO, achieves state-of-the-art performance on this benchmark, surpassing even much larger proprietary models. In general, our work demonstrates that existing long context LLM already possesses the potential for a larger output window--all you need is data with extended output during model alignment to unlock this capability. Our code & models are at: https://github.com/THUDM/LongWriter.
Forward citations
Cited by 12 Pith papers
-
HSS-Synth: Humanities and Social Sciences Data Synthesis for LLMs
HSS-Synth generates 230k instruction-tuning samples for 14 humanities/social-science fields and reports state-of-the-art fine-tuning results on 16 benchmarks.
-
Reverse-Engineered Reasoning for Open-Ended Generation
Given a high-quality output, the authors search for a thinking trace that minimizes that output's perplexity, then fine-tune Qwen3-8B on 20,000 such traces, reporting writing performance near GPT-4o and Claude 3.5.
-
LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation framework
LOOM-Scope is a framework that standardizes long-context LLM evaluation across 22 benchmarks and integrates a lightweight 12-benchmark suite, LOOMBench, for fast comprehensive assessment.
-
From General to Targeted Rewards: Surpassing GPT-4 in Open-Ended Long-Context Generation
ProxyReward trains long-form generation models by rewarding how well an AI judge can answer generated yes/no questions about the response, improving open-source models on ProxyQA.
-
SuperWriter: Reflection-Driven Long-Form Generation with Large Language Models
A 7B writing model trained on plan-write-refine thinking data with multi-stage preference optimization matches or beats several larger models on long-form generation benchmarks.
-
AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism
AdaDecode speeds up LLM generation by predicting tokens at early layers when confidence is high, running the skipped layers in parallel, and verifying the output exactly matches standard decoding.
-
FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Quality
A multi-LLM consensus pipeline turns 14,450 auto-generated candidate rubrics into 2,600 distinguishable gold rubrics that rank 10 financial deep-research systems from 58.58% to 22.23% pass rate.
-
GEIS: A Generation-Evaluation-Improvement Loop of Agent Skills for Long-Form Article Generation
A skill-based generate–evaluate–improve loop raises Wikipedia-style long-form article quality over fixed multi-agent pipelines and self-improves via permanent writing-rule patches.
-
STORYTELLER: An Enhanced Plot-Planning Framework for Coherent and Cohesive Story Generation
A plot-node and knowledge-graph guided LLM framework that generates long stories and reports high preference win rates over strong baselines, though with evaluation confounds.
-
HiCaM: A Hierarchical-Causal Modification Framework for Long-Form Text Modification
HiCaM combines an LLM-built hierarchical summary tree with a causal entity graph to guide long-document editing, reporting 56.8-72.5% win rates and up to 59.5% net win rates over direct LLM baselines as judged by GPT-4o.
-
MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Models
MiniLongBench, a 237-sample compression of LongBench, is claimed to reproduce model rankings with a 0.97 Spearman correlation at 4.5% of the evaluation cost.
-
DynamicBench: Evaluating Real-Time Report Generation in Large Language Models
DynamicBench is a proposed benchmark for real-time report generation, and the authors claim their retrieval-augmented system outperforms GPT-4o, but the evidence is inconsistent and the comparison is not apples-to-apples.
Discussion (0). Continue with ORCID to comment.