Pith. sign in

REVIEW 13 cited by

LongNet: Scaling Transformers to 1,000,000,000 Tokens

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2307.02486 v2 pith:FV6BMPRK submitted 2023-07-05 cs.CL cs.LG

Jiayu Ding , Shuming Ma , Li Dong , Xingxing Zhang , Shaohan Huang , Wenhui Wang , Nanning Zheng , Furu Wei This is my paper · ORCID
classification cs.CLcs.LG
keywords sequencelongnetattentionlengthsequencestokenscomplexitydilated
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Scaling sequence length has become a critical demand in the era of large language models. However, existing methods struggle with either computational complexity or model expressivity, rendering the maximum sequence length restricted. To address this issue, we introduce LongNet, a Transformer variant that can scale sequence length to more than 1 billion tokens, without sacrificing the performance on shorter sequences. Specifically, we propose dilated attention, which expands the attentive field exponentially as the distance grows. LongNet has significant advantages: 1) it has a linear computation complexity and a logarithm dependency between any two tokens in a sequence; 2) it can be served as a distributed trainer for extremely long sequences; 3) its dilated attention is a drop-in replacement for standard attention, which can be seamlessly integrated with the existing Transformer-based optimization. Experiments results demonstrate that LongNet yields strong performance on both long-sequence modeling and general language tasks. Our work opens up new possibilities for modeling very long sequences, e.g., treating a whole corpus or even the entire Internet as a sequence.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 35 citations worldwide. Full citation record

  1. Do Multiple Instance Learning Models Transfer?

    cs.CV 2025-06 conditional novelty 7.0 of 10

    Pretrained multiple instance learning models transfer across organs and tasks in computational pathology, and pancancer pretraining can rival slide foundation models with far less data.

  2. From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving

    cs.DC 2026-06 accept novelty 6.5 of 10

    KV-cache serving systems concentrate into five archetypes under a four-axis taxonomy, with ownership explaining residual distributed design variance and seven measurement gaps blocking next steps.

  3. AGNFormer I: Reconstruction of AGN spectra using a probabilistic transformer model

    astro-ph.GA 2026-07 conditional novelty 6.0 of 10

    An uncertainty-aware transformer reconstructs masked AGN broad lines and spectral halves with 4-16% flux errors and beats eleven purpose-built Lyα-reconstruction algorithms on a blind benchmark.

  4. MOOZY: A Patient-First Foundation Model for Computational Pathology

    cs.CV 2026-03 conditional novelty 6.0 of 10

    Patient-level pretraining with a case transformer and multi-task public supervision yields transferable WSI embeddings that beat larger slide-centric models on held-out pathology tasks.

  5. HiCI: Hierarchical Construction-Integration for Long-Context Attention

    cs.CL 2026-03 unverdicted novelty 6.0 of 10

    HiCI is a new hierarchical attention module that extends LLaMA-2 context from 4K to 100K tokens using under 5.5% extra parameters and improves results on retrieval and code tasks.

  6. RAT+: Train Dense, Infer Sparse -- Recurrence Augmented Attention for Dilated Inference

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    RAT+ pretrains a single dense recurrent-augmented attention model that supports flexible dilated sparse inference after short adaptation, matching dense accuracy at moderate dilation and losing only 1-3 points at high...

  7. RED-PIM: Reducing Data Movement for Transformers using Processing-in-Memory

    cs.LG 2026-07 conditional novelty 5.0 of 10

    RED-PIM restructures attention to use a compact d×d intermediate matrix, reducing PIM inter-bank data movement from O(N²) to O(N) and claiming up to 99.99% latency reduction in simulation.

  8. GigaPath-Flash and GigaTIME-Flash: Efficient Pathology Foundation Models for Whole-Slide and Tumor Microenvironment Analysis

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Distilled, Apache-2.0-licensed GigaPath-Flash and GigaTIME-Flash models deliver most of the original models' accuracy at a fraction of the compute and memory.

  9. SAC-MIL: Spatial-Aware Correlated Multiple Instance Learning for Histopathology Whole Slide Image Classification

    cs.CV 2025-09 conditional novelty 5.0 of 10

    A spatial-aware MLP-Mixer with polar rotary position embedding and channel-shifting SAC blocks achieves state-of-the-art slide-level classification on CAMELYON-16, TCGA-LUNG, and TCGA-BRCA.

  10. Docopilot: Improving Multimodal Models for Document-Level Understanding

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A new academic-paper dataset and a retrieval-free fine-tuned InternVL2 model improve multi-page document QA accuracy and latency on several benchmarks.

  11. SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compression

    cs.CL 2025-07 conditional novelty 5.0 of 10

    SARA combines short natural-language snippets with vector-compressed summaries of the remaining retrieved documents, improving RAG answer quality under 512/1024-token context budgets.

  12. MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    MiniLongBench, a 237-sample compression of LongBench, is claimed to reproduce model rankings with a 0.97 Spearman correlation at 4.5% of the evaluation cost.

  13. VideoSEMA: a scalable and efficient Mamba-like attention for video understanding

    cs.CV 2026-07 conditional novelty 4.0 of 10

    VideoSEMA uses SEMA spatial attention plus softmax temporal attention to outperform larger video transformers and Mamba models on K400/SSv2 and degrade less at 1024² resolution.

Pith tools