Pith. sign in

REVIEW 5 cited by

Long-Context Language Modeling with Parallel Context Encoding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.16617 v2 pith:GS5JCQOG submitted 2024-02-26 cs.CL

Howard Yen , Tianyu Gao , Danqi Chen This is my paper · ORCID
classification cs.CL
keywords cepecontextwindowcontextsencodinglanguagemodelsapplications
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Extending large language models (LLMs) to process longer inputs is crucial for a wide range of applications. However, the substantial computational cost of transformers and limited generalization of positional encoding restrict the size of their context window. We introduce Context Expansion with Parallel Encoding (CEPE), a framework that can be applied to any existing decoder-only LLMs to extend their context window. CEPE employs a small encoder to process long inputs chunk by chunk, enabling the frozen decoder to utilize additional contexts via cross-attention. CEPE is efficient, generalizable, and versatile: trained with 8K-token documents, it extends the context window of LLAMA-2 to 128K tokens, offering 10x the throughput with only 1/6 of the memory. CEPE yields strong performance on language modeling and in-context learning. CEPE also excels in retrieval-augmented applications, while existing long-context models degenerate with retrieved contexts. We further introduce a CEPE variant that can extend the context window of instruction-tuned models using only unlabeled data, and showcase its effectiveness on LLAMA-2-CHAT, leading to a strong instruction-following model that can leverage very long contexts on downstream tasks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MetaSyn: A Benchmark for LLM Agents on Meta-Analysis Articles from Nature Portfolio

    cs.CL 2026-06 unverdicted novelty 8.0 of 10

    MetaSyn is a stage-level benchmark of 442 meta-analyses showing LLM agents retrieve up to 90.9% of eligible studies but include at most 52.7% in their final reports.

  2. ARC-Encoder: learning compressed text representations for large language models

    cs.CL 2025-10 conditional novelty 6.0 of 10

    ARC-Encoder pools queries in an encoder's last attention layer to produce compressed continuous representations that a frozen decoder consumes as token embeddings.

  3. AbsenceBench: Language Models Can't Tell What's Missing

    cs.CL 2025-06 conditional novelty 6.0 of 10

    LLMs that ace Needle-in-a-Haystack struggle to identify deliberately omitted content, a new benchmark called AbsenceBench shows.

  4. RoToR: Towards More Reliable Responses for Order-Invariant Inputs

    cs.CL 2025-02 conditional novelty 6.0 of 10

    RoToR makes a frozen LLM order-invariant by circularly rotating a single global sort of segment position IDs, and Selective Routing combines it with the original model for mixed lists.

  5. Position: Episodic Memory is the Missing Piece for Long-Term LLM Agents

    cs.AI 2025-02 conditional novelty 5.0 of 10

    The authors propose episodic memory, with five defining properties, as the unifying framework needed for LLM agents to learn and remember over long time horizons.

Pith tools