REVIEW 13 cited by
Effective Long-Context Scaling of Foundation Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We present a series of long-context LLMs that support effective context windows of up to 32,768 tokens. Our model series are built through continual pretraining from Llama 2 with longer training sequences and on a dataset where long texts are upsampled. We perform extensive evaluation on language modeling, synthetic context probing tasks, and a wide range of research benchmarks. On research benchmarks, our models achieve consistent improvements on most regular tasks and significant improvements on long-context tasks over Llama 2. Notably, with a cost-effective instruction tuning procedure that does not require human-annotated long instruction data, the 70B variant can already surpass gpt-3.5-turbo-16k's overall performance on a suite of long-context tasks. Alongside these results, we provide an in-depth analysis on the individual components of our method. We delve into Llama's position encodings and discuss its limitation in modeling long dependencies. We also examine the impact of various design choices in the pretraining process, including the data mix and the training curriculum of sequence lengths -- our ablation experiments suggest that having abundant long texts in the pretrain dataset is not the key to achieving strong performance, and we empirically verify that long context continual pretraining is more efficient and similarly effective compared to pretraining from scratch with long sequences.
Forward citations
Cited by 13 Pith papers
-
MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent
MemAgent uses multi-conversation RL to train a memory agent that reads text in segments and overwrites memory, extrapolating from 8K training to 3.5M token QA with under 5% loss and 95%+ on 512K RULER.
-
From General to Targeted Rewards: Surpassing GPT-4 in Open-Ended Long-Context Generation
ProxyReward trains long-form generation models by rewarding how well an AI judge can answer generated yes/no questions about the response, improving open-source models on ProxyQA.
-
Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning
Enhancing a model's long-context ability before supervised fine-tuning improves math reasoning accuracy, including on short-input benchmarks.
-
Domyn-Small: A European 10B Reasoning Language Model
Domyn-Small is a 10B reasoning LLM that claims to deliver roughly one-third the inference tokens of Qwen3.5-9B at competitive accuracy, though results are marked as preliminary.
-
StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understanding
A training-free, query-agnostic KV cache compression method for streaming video MLLMs, using chat-template attention as a saliency proxy, matches or beats prior streaming methods at a fixed 6K memory budget.
-
100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?
100-LongBench and LongScore evaluate LLMs at controlled context lengths and rank models by relative performance drop from a short-context baseline, not by raw accuracy.
-
SELF: Self-Extend the Context Length With Logistic Growth Function
SELF replaces Self-Extend's fixed token grouping with a logistic-growth grouping schedule, giving mixed but sometimes large gains on long-context benchmarks.
-
LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions
By prompting an aligned LLM with a document and the special token that precedes a user query, LongMagpie synthesizes long-context instruction data that outperforms prior datasets when used to fine-tune Llama-3-8B.
-
SciGPT: A Large Language Model for Scientific Literature Understanding and Knowledge Discovery
SciGPT, a fine-tuned Qwen3 model for scientific literature, is reported to outperform GPT-4 on a new ScienceBench benchmark, but the evaluation is unreliable due to missing artifacts and contradictory numbers.
-
HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Models
HoPE replaces RoPE's sine/cosine rotations with hyperbolic functions plus an exponential damping term to enforce monotonic attention decay, but the claimed consistent superiority and the 'RoPE as special case' theorem...
-
AgentScope 1.0: A Developer-Centric Framework for Building Agentic Applications
AgentScope 1.0 packages the components needed to build, evaluate, and deploy LLM agent applications into one developer framework.
-
An Overview of Algorithms for Contactless Cardiac Feature Extraction from Radar Signals: Advances and Challenges
The paper claims to be the first review of signal-processing algorithms for radar-based cardiac monitoring, with a new taxonomy and public dataset listings.
-
GigaChat Family: Efficient Russian Language Modeling Through Mixture of Experts Architecture
A from-scratch Russian MoE LLM family is released with competitive benchmark scores, but key efficiency and novelty claims are not fully evidenced.
Discussion (0). Continue with ORCID to comment.