Pith. sign in

REVIEW 6 cited by

Training-Free Long-Context Scaling of Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.17463 v2 pith:5OKDIZC7 submitted 2024-02-27 cs.CL

Chenxin An , Fei Huang , Jun Zhang , Shansan Gong , Xipeng Qiu , Chang Zhou , Lingpeng Kong This is my paper · ORCID
classification cs.CL
keywords modelsattentiontokenschunklanguagelargelong-contextperformance
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The ability of Large Language Models (LLMs) to process and generate coherent text is markedly weakened when the number of input tokens exceeds their pretraining length. Given the expensive overhead of finetuning large-scale models with longer sequences, we propose Dual Chunk Attention (DCA), which enables Llama2 70B to support context windows of more than 100k tokens without continual training. By decomposing the attention computation for long sequences into chunk-based modules, DCA manages to effectively capture the relative positional information of tokens within the same chunk (Intra-Chunk) and across distinct chunks (Inter-Chunk), as well as integrates seamlessly with Flash Attention. In addition to its impressive extrapolation capability, DCA achieves performance on practical long-context tasks that is comparable to or even better than that of finetuned models. When compared with proprietary models, our training-free 70B model attains 94% of the performance of gpt-3.5-16k, indicating it is a viable open-source alternative. All code and data used in this work are released at \url{https://github.com/HKUNLP/ChunkLlama}.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE

    cs.LG 2026-07 accept novelty 6.0 of 10

    Jet-Long is a tuning-free bifocal RoPE method that dynamically sets remote group size from sequence length, recovering the base model within the pretrained window and beating prior zero-shot extenders on RULER, HELMET...

  2. MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent

    cs.CL 2025-07 unverdicted novelty 6.0 of 10

    MemAgent uses multi-conversation RL to train a memory agent that reads text in segments and overwrites memory, extrapolating from 8K training to 3.5M token QA with under 5% loss and 95%+ on 512K RULER.

  3. SuperWriter: Reflection-Driven Long-Form Generation with Large Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A 7B writing model trained on plan-write-refine thinking data with multi-stage preference optimization matches or beats several larger models on long-form generation benchmarks.

  4. Training Long-Context LLMs Efficiently via Chunk-wise Optimization

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Chunk-wise backpropagation with a sparse, rescaled variant fine-tunes an 8B model on 16K tokens using a single RTX 3090 GPU.

  5. AgentScope 1.0: A Developer-Centric Framework for Building Agentic Applications

    cs.AI 2025-08 unverdicted novelty 4.0 of 10

    AgentScope 1.0 packages the components needed to build, evaluate, and deploy LLM agent applications into one developer framework.

  6. Dynamic Chunking and Selection for Reading Comprehension of Ultra-Long Context in Large Language Models

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Dynamic chunking plus question-aware chunk selection improves long-context QA, but the headline numbers are partly inflated by choosing hyperparameters on the test benchmarks.

Pith tools