Pith. sign in

REVIEW 12 cited by

Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2409.04701 v3 pith:K44LOKHU submitted 2024-09-07 cs.CL cs.IR

Michael Günther , Isabelle Mohr , Daniel James Williams , Bo Wang , Han Xiao This is my paper · ORCID
classification cs.CLcs.IR
keywords chunkingembeddingembeddingslatemodelstextchunkcontextual
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Many use cases require retrieving smaller portions of text, and dense vector-based retrieval systems often perform better with shorter text segments, as the semantics are less likely to be over-compressed in the embeddings. Consequently, practitioners often split text documents into smaller chunks and encode them separately. However, chunk embeddings created in this way can lose contextual information from surrounding chunks, resulting in sub-optimal representations. In this paper, we introduce a novel method called late chunking, which leverages long context embedding models to first embed all tokens of the long text, with chunking applied after the transformer model and just before mean pooling - hence the term late in its naming. The resulting chunk embeddings capture the full contextual information, leading to superior results across various retrieval tasks. The method is generic enough to be applied to a wide range of long-context embedding models and works without additional training. To further increase the effectiveness of late chunking, we propose a dedicated fine-tuning approach for embedding models.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Right Reset: Chunking by Prefix Removal

    cs.CL 2026-08 accept novelty 6.0 of 10

    Right Reset measures how much a causal language model's hidden-state trajectory for a right-hand window changes when the left prefix is removed, and uses that as a text chunk boundary score.

  2. CMDR: Contextual Multimodal Document Retrieval

    cs.IR 2026-07 conditional novelty 6.0 of 10

    A contextual multimodal document retrieval benchmark (CMDR-Bench) and embedding model (CMDR-Embed) that jointly encodes multiple document pages and splits them into page-level representations, trained with a context-a...

  3. MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A two-stage training recipe that converts causal VLMs into bidirectional multimodal embedding models, achieving SOTA on MMEB.

  4. Context is Gold to find the Gold Passage: Evaluating and Training Contextual Document Embeddings

    cs.IR 2025-05 conditional novelty 6.0 of 10

    A new benchmark (ConTEB) and training method (InSeNT) show that context-aware chunk embeddings greatly improve retrieval on context-dependent queries, with minimal computational overhead.

  5. EXIT: Context-Aware Extractive Compression for Enhancing Retrieval-Augmented Generation

    cs.CL 2024-12 conditional novelty 6.0 of 10

    A parallel sentence-classification compressor can make RAG question answering more accurate and faster than abstractive compression and often faster than using the original retrieved documents.

  6. CoRet: Improved Retriever for Code Editing

    cs.LG 2025-05 conditional novelty 5.0 of 10

    CoRet fine-tunes a code encoder with a repository-level likelihood loss and call-graph context, improving retrieval recall on SWE-bench and Long Code Arena.

  7. Retrieval Augmented Decision-Making: A Requirements-Driven, Multi-Criteria Framework for Structured Decision Support

    cs.AI 2025-05 reject novelty 5.0 of 10

    RAD automatically extracts weighted, hierarchical decision criteria from documents and uses LLMs to generate structured decision reports, but its evaluation is largely self-referential.

  8. Reconstructing Context: Evaluating Advanced Chunking Strategies for Retrieval-Augmented Generation

    cs.IR 2025-04 conditional novelty 5.0 of 10

    A comparative evaluation on NFCorpus and MSMarco shows contextual retrieval beats late chunking on retrieval metrics at higher computational cost, and neither consistently beats standard early chunking.

  9. GeAR: Generation Augmented Retrieval

    cs.IR 2025-01 conditional novelty 5.0 of 10

    GeAR adds a text-generation objective and cross-attention fusion to a bi-encoder retriever, and the authors report improved sentence-level retrieval inside documents while keeping global ranking as fast as a standard ...

  10. LLM as HPC Expert: Extending RAG Architecture for HPC Data

    cs.DC 2024-12 conditional novelty 5.0 of 10

    HyCE, a RAG extension that retrieves and executes HPC commands using their natural-language descriptions, raised the authors' LLM-judged answer quality from 77.7% to 82.3%.

  11. LPCAN: Lightweight Pyramid Cross-Attention Network for Rail Surface Defect Detection Using RGB-D Data

    cs.CV 2026-01 reject novelty 4.0 of 10

    A lightweight RGB-D cross-attention network is proposed for rail defect detection, but the SOTA accuracy and generalization claims are internally inconsistent and the implementation is not public.

  12. Knowledge-Embedded and Hypernetwork-Guided Few-Shot Substation Meter Defect Image Generation Method

    cs.CV 2026-01 reject novelty 3.0 of 10

    Fine-tuning Stable Diffusion with DreamBooth-style knowledge and hypernetwork-guided crack control maps can synthesize substation meter defect images that boost a YOLOv8 defect detector's mAP when added to the training set.

Pith tools