REVIEW 4 major objections 5 minor 1 cited by
Reranking with Compressed Document Representation
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A reranker fed only eight compressed embeddings per document matches full-text rivals within one nDCG point, while running up to 16 times faster.
desk verdict A useful, honest paper that shows frozen RAG-style compressed embeddings can serve as reranker input with near-textual effectiveness, but the headline speedup is simulated, not measured. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the fixed-size compressed document representation produced by PISCO. PISCO appends eight memory tokens to a document, passes the sequence through a frozen language model, and keeps the eight final hidden states as the document's embedding; the compressor is trained so a RAG decoder can generate from these embeddings. RRK freezes those embeddings and feeds them to a Mistral-7B decoder fine-tuned by LoRA, projecting the last-layer end-of-sequence representation to a scalar via a linear head and training against a teacher score with mean-squared error. This reduces reranking to a fixed 32-token sequence (8 memory tokens plus up to 24 query tokens), so inference cost does not depend on document length. A 16-layer variant of the decoder trades about two nDCG@10 points for additional speed.
What would settle it
Run RRK and a textual reranker on the same long-document collection with documents above 1,024 tokens, where PISCO was never trained; if RRK's nDCG@10 drops by substantially more than the roughly one-point gap seen on BeIR as document length grows, while the textual reranker holds, the constant-quality claim is falsified.
Extended reading notes
Core claim
Using the frozen PISCO compressor, each document is encoded offline into eight memory-token embeddings; at rerank time the input is those eight embeddings plus up to 24 query tokens, a fixed 32 tokens. A Mistral-7B decoder, fine-tuned with LoRA, is trained with a mean-squared-error loss to match the scores of a DeBERTa-based cross-encoder teacher on MS MARCO. The resulting RRK models reproduce teacher-level nDCG@10 within a point on BeIR and within the same range on LoTTe, while the fixed input size keeps inference time flat as documents grow; with all 32 layers it is up to 16 times faster than the equivalent textual Mistral reranker, and the 16-layer variant is faster than ModernBERT-large though about two points less effective. On TREC-NEWS, RRK at 768 input tokens scores 53.1 nDCG@10, better than any textual model at any tested length, even though the PISCO compressor and the reranker were trained only on documents of 128 tokens or fewer.
Load-bearing premise
The paper relies on the frozen PISCO compressor preserving query-relevant detail after squeezing a document into eight embeddings, despite being trained only on documents of at most 128 tokens; the long-document results assume this compression generalizes well beyond its training length.
Editorial extensions
If this is right
- Reranking long documents (whole web pages, news articles, legal texts) no longer costs more than reranking short passages, because the online input is fixed at 32 tokens regardless of document length.
- The same compressed index used for RAG generation can be reused for reranking, so the storage overhead of 270 GB for MS MARCO is shared across both stages.
- Distillation from a small, accurate cross-encoder into a larger decoder works with compressed inputs, meaning a practitioner can distill any favorite reranker into the compressed format using only MS MARCO queries and a retriever's top-50 list, with no relevance judgments.
- The 16-layer RRK variant is faster than a 400M-parameter ModernBERT-large while staying within about two nDCG@10 points, making compressed reranking competitive with state-of-the-art encoder-only rerankers.
- LLM-based rerankers become viable in latency-sensitive settings where full-text 7B rerankers were previously too slow.
Reading between the lines
- If PISCO or a successor compressor is trained on longer documents (e.g., 512 or 1,024 tokens), the long-document gap the authors observe at 1K tokens may close, and compressed reranking could dominate textual reranking for long-form retrieval.
- The constant-latency advantage depends on short queries (at most 24 tokens); for query-heavy benchmarks where queries are as long as documents, RRK's speed advantage disappears, and a query-compression or late-interaction variant would be needed to preserve it.
- The 8-embedding-per-document choice is a compression-rate knob; testing 4, 16, or 32 memory tokens on the same distillation setup could reveal the trade-off curve between effectiveness and index size, which the paper does not explore.
- Because the compressed index is reused for RAG generation, a future system could rerank, generate, and cite from the same embeddings, potentially removing the need to decompress text for the final answer.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes RRK, a reranker that uses a frozen PISCO compressor to map each document to 8 memory-token embeddings, which are then concatenated with a short query (up to 24 tokens) to form a fixed 32-token input for a Mistral-7B decoder fine-tuned via distillation from a Naver-DeBERTa teacher. The authors evaluate RRK on TREC-DL, BeIR, LoTTe, and TREC-NEWS, reporting that it matches the effectiveness of textual-input rerankers within about one nDCG@10 point while maintaining constant inference time with respect to document length and achieving up to a 16x speed-up over textual 512-token inputs. They also test a lighter 16-layer variant and present long-document results for TREC-NEWS.
Significance. If the efficiency claims are established, this is a valuable direction: it shows that an off-the-shelf RAG document compressor can be repurposed for reranking, converting a 7B-parameter LLM reranker from a length-dependent to a fixed-cost model. The experimental design is a strength: compressed and textual models are trained under identical distillation conditions (same teacher, same training data, same document lengths), making the effectiveness comparison fair. The use of public benchmarks and the link to the Bergen library support reproducibility. However, the central efficiency contribution is currently supported only by a simulated latency model, not by end-to-end wall-clock measurement, and the effectiveness equivalence is asserted without error bars or significance tests. The long-document generalization is acknowledged by the authors as limited by the compressor's 128-token training cap, so the strong long-document claims rest on a single dataset.
major comments (4)
- [§5, Fig. 1] The headline efficiency claims ('up to 16 times faster', '10x speed-up') rest entirely on a simulated latency model, not on measured end-to-end time. The Figure 1 caption states that the comparison is simulated for 50 queries × 50 documents with batch size 256 on an A100 80G. The asserted '<10% of total reranking time' for PISCO index access in §4 is given without any measurement protocol. Since the paper's central contribution is solving the efficiency problem of LLM-based rerankers, this is a load-bearing gap. Please provide a direct wall-clock latency/throughput comparison on the same hardware that includes the cost of retrieving the 270GB compressed index and feeding the 8 memory-token embeddings, for the same batch sizes and input lengths used in the effectiveness evaluation.
- [Tables 1 and 2] All effectiveness results are reported as single runs without error bars, confidence intervals, or significance tests. The claim of 'almost the same effectiveness (-0.9 for BeIR)' is based on a one-point difference in a single run, which is within typical reranking variance on nDCG@10. Please report multiple seeds or bootstrap intervals, and test significance for the key comparisons (RRK vs Mistral textual, RRK vs ModernBERT) so that the equivalence claim is supported.
- [§4, Table 2, Limitations] The long-document evaluation is based on a single dataset (TREC-NEWS) and uses a PISCO compressor trained on documents of at most 128 tokens, a limitation the authors acknowledge. The conclusion that RRK 'achieves the best results with a 768 token input' is not supported without variance estimates and more long-document benchmarks. Moreover, the drop at 1024 tokens is consistent with the 128-token training cap; the paper should either train or adapt the compressor on longer documents or explicitly restrict the claim to the tested setting.
- [§2 and §3] The exact input construction for the RRK reranker is underspecified: how are the up to 24 query tokens combined with the 8 memory-token embeddings (order, separator tokens), how are the precomputed embeddings loaded from the index, and what LoRA rank/configuration is used for the decoder? Without this information, the experiments are not reproducible; please specify the input format and release the training/inference code (or include it in the Bergen library).
minor comments (5)
- [Table 1] The Climate-FEVER row for the Mistral 256 column reports '2 7.5', which appears to be a typo for 27.5.
- [§5, first paragraph] The sentence 'Let us first draw some some general observations' contains a duplicated word 'some'.
- [Abstract and §5] The claim that RRK 'challenges smaller rerankers' is not directly supported by the main table, which compares only to ModernBERT-large and Mistral-7B; consider adding a small cross-encoder or late-interaction baseline such as ColBERT for a more complete efficiency-effectiveness comparison.
- [Table 4] The NFCorpus entry for the llama 1B column reads '0.30.38', which appears to be a typo for '0.38'.
- [§5] The statement that the RRK model is 'impacted only during offline compression' would benefit from explicitly reporting the offline compression cost and clarifying that online reranking excludes index-building time.
Circularity Check
Empirical evaluation with self-cited components (PISCO, Naver-DeBERTa) but central effectiveness claims are externally benchmarked; no circular derivation.
full rationale
This paper makes no derivational claim that reduces to its own inputs. The RRK reranker is built from a frozen PISCO compressor (prior work by overlapping authors, [17]) and is distilled from Naver-DeBERTa (also a Naver Labs model, [12]); these are self-citations, but they are used as system components, not as evidence for the paper's conclusions. The central claims—that a fixed 32-token compressed input nearly matches textual rerankers in nDCG@10 and generalizes to long documents—are evaluated on external benchmarks (TREC-DL 19/20, BeIR, LoTTe pooled, TREC-NEWS) against independently trained baselines (Mistral-7B textual, ModernBERT-large) under the same training recipe. The MSE distillation loss to the teacher on MSMARCO training queries does not force scores on held-out benchmarks. The paper's Limitations explicitly acknowledge that the 128-token training cap limits long-document effectiveness and that long queries break the efficiency advantage, showing these outcomes are not definitionally guaranteed. The efficiency claim rests on the simulated latency curves in Figure 1 rather than on a measured end-to-end system comparison, and the statement that PISCO index access is '<10% of total reranking time' is asserted without a protocol; these are verification gaps, not circularity. Score 2 reflects the presence of several self-citations (PISCO, SPLADE-V3, Naver-DeBERTa, BERGEN) none of which is load-bearing in a way that forces the reported results.
Assumptions & free parameters
free parameters (6)
- Memory token count =
8
- Maximum query length =
24 tokens
- Maximum training length =
128 tokens
- Training epochs =
2
- Learning rate =
1e-4
- Number of decoder layers (RRK 16) =
16 of 32
assumptions (4)
- domain assumption PISCO embeddings preserve the relevance information needed for query-document ranking even though PISCO was trained for RAG answer generation.
- domain assumption Teacher scores from Naver-DeBERTa are a suitable distillation target for training a compressed-input reranker.
- domain assumption Reranking the top 50 SPLADE-v3 documents is a fair and representative evaluation condition.
- domain assumption Standard IR benchmarks (BeIR, TREC-DL, LoTTe pooled) are representative for ranking effectiveness.
Cite this review
Pith. "Pith review of Reranking with Compressed Document Representation." pith.science (2026). https://pith.science/paper/JQLFRTGJ
@misc{pith2026250515394,
author = {Pith},
title = {Pith review of: Reranking with Compressed Document Representation},
year = {2026},
howpublished = {\url{https://pith.science/paper/JQLFRTGJ}},
note = {Machine review of arXiv:2505.15394}
}
read the original abstract
Reranking, the process of refining the output of a first-stage retriever, is often considered computationally expensive, especially with Large Language Models. Borrowing from recent advances in document compression for RAG, we reduce the input size by compressing documents into fixed-size embedding representations. We then teach a reranker to use compressed inputs by distillation. Although based on a billion-size model, our trained reranker using this compressed input can challenge smaller rerankers in terms of both effectiveness and efficiency, especially for long documents. Given that text compressors are still in their early development stages, we view this approach as promising.
Figures
Forward citations
Cited by 1 Pith paper
-
MICE: Minimal Interaction Cross-Encoders for efficient Re-ranking
MICE is a cross-encoder-derived late-interaction ranker that retains most in-domain effectiveness and beats same-size ColBERT by 5-8 nDCG@10 points while cutting latency up to 4x with precomputed document vectors.
Reference graph
Works this paper leans on
-
[1]
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Sto- ica, Saurabh Tiwary, and Tong Wang. 2018. Ms marco: Ahumangeneratedmachinereadingcom- prehension dataset.Preprint, arXiv:1611.09268. 2
arXiv 2018
-
[2]
Sebastian Bruch, Franco Maria Nardini, Cosimo Rulli, and Rossano Venturini. 2024. Efficient inverted indexes for approximate retrieval over 4 Reranking with Compressed Document Representation learned sparse representations. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, page 1...
work page 2024
-
[3]
Cesare Campagnano, Antonio Mallia, Jack Pertschuk, and Fabrizio Silvestri. 2025. E2rank: Efficient and effective layer-wise reranking. In Advances in Information Retrieval, pages 417–426, Cham. Springer Nature Switzerland. 2
work page 2025
-
[4]
Hongliu Cao. 2024. Recent advances in text embedding: A comprehensive review of top- performing methods on the mteb benchmark. Preprint, arXiv:2406.01607. 2
arXiv 2024
-
[5]
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2021. Overview of the trec 2020 deep learning track.Preprint, arXiv:2102.07662. 2
arXiv 2021
- [6]
-
[7]
Hervé Déjean, Stéphane Clinchant, and Thibault Formal. 2024. A thorough comparison of cross- encoders and llms for reranking splade.Preprint, arXiv:2403.10407. 1, 2
arXiv 2024
-
[8]
Revanth Gangi Reddy, JaeHyeok Doo, Yifei Xu, Md Arafat Sultan, Deevya Swain, Avirup Sil, and Heng Ji. 2024. FIRST: Faster improved listwise reranking with single token decoding. InProceed- ings of the 2024 Conference on Empirical Methods inNaturalLanguageProcessing , pages8642–8652, Miami, Florida, USA. Association for Computa- tional Linguistics. 1
work page 2024
Show all 36 references
-
[9]
Luyu Gao, Zhuyun Dai, and Jamie Callan. 2020. Modularized transfomer-based ranking frame- work. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4180–4190, Online. Association for Computational Linguistics. 1
2020
-
[10]
Luyu Gao, Zhuyun Dai, and Jamie Callan
-
[11]
Tao Ge, Jing Hu, Xun Wang, Si-Qing Chen, and Furu Wei. 2023. In-context autoencoder for con- text compression in a large language model.arXiv preprint arXiv:2307.06945. 1
2023 arXiv
-
[12]
Carlos Lassance and Stéphane Clinchant. 2023. Naver labs europe (splade) @ trec deep learning
2023
-
[13]
Carlos Lassance, Hervé Déjean, Stéphane Clin- chant, and Nicola Tonellotto. 2024. Two-step splade: Simple, efficient and effective approxima- tion of splade.Preprint, arXiv:2404.13357. 7
2024 arXiv
-
[14]
Carlos Lassance, Hervé Déjean, Thibault Formal, and Stéphane Clinchant. 2024. Splade-v3: New baselines for splade.Preprint, arXiv:2403.06789. 2, 8
2024 arXiv
-
[15]
Sean Lee, Aamir Shakir, Darius Koenig, and Julius Lippand. 2023. Open source strikes bread - new fluffy embedding model. https://www. mixedbread.com/blog/mxbai-embed-large-v1. Accessed: 2025-05-05. 8
2023
-
[16]
Zheng Liu, Chaofan Li, Shitao Xiao, Chaozhuo Li, Defu Lian, and Yingxia Shao. 2025. Ma- tryoshka re-ranker: A flexible re-ranking architec- ture with configurable depth and width.Preprint, arXiv:2501.16302. 1
2025 arXiv
-
[17]
Maxime Louis, Hervé Déjean, and Stéphane Clin- chant. 2025. Pisco: Pretty simple compression for retrieval-augmented generation. Preprint, arXiv:2501.16075. 1, 8, 9
2025 arXiv
-
[18]
Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2023. Fine-Tuning LLaMA for Multi-Stage Text Retrieval. arXiv preprint. ArXiv:2310.08319 [cs]. 1
2023 arXiv
-
[19]
Sean MacAvaney, Franco Maria Nardini, Raf- faele Perego, Nicola Tonellotto, Nazli Goharian, and Ophir Frieder. 2020. Efficient document re- ranking for transformers by precomputing term representations. InProceedings of the 43rd Inter- national ACM SIGIR Conference on Research...
2020
-
[20]
Rodrigo Nogueira and Kyunghyun Cho. 2020. Passage re-ranking with bert. Preprint, arXiv:1901.04085. 1
2020 arXiv
-
[21]
Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. 2021. The expando-mono-duo design pat- tern for text ranking with pretrained sequence- to-sequence models.Preprint, arXiv:2101.05667. 1
2021 arXiv
-
[22]
David Rau, Hervé Déjean, Nadezhda Chirkova, Thibault Formal, Shuai Wang, Vassilina Nikoulina, and Stéphane Clinchant. 2024. Bergen: A bench- marking library for retrieval-augmented genera- tion. arXiv preprint arXiv:2407.01102. 7
2024 arXiv
-
[23]
David Rau, Shuai Wang, Hervé Déjean, and Stéphane Clinchant. 2024. Context embeddings for efficient answer generation in rag.Preprint, arXiv:2407.09252. 1
2024 arXiv
-
[24]
Khattab, Jon Saad-Falcon, Christopher Potts, and Matei A
Keshav Santhanam, O. Khattab, Jon Saad-Falcon, Christopher Potts, and Matei A. Zaharia. 2021. Colbertv2: Effective and efficient retrieval via 5 Reranking with Compressed Document Representation lightweight late interaction. InNorth American Chapter of the Association for Comp...
2021
-
[25]
Ian Soboroff, Shudong Huang, and Donna Har- man. 2018. Trec 2018 news track overview. In TREC, volume 409, page 410. 4
2018
-
[26]
Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Haisu Liu, Quan Shi, Zachary S Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan O Arik, Danqi Chen, and Tao Yu. 2024. Bright: A realistic and challenging benchmark for reasoning-intensive r...
2024
-
[27]
Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. 2023. Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent. arXiv preprint. ArXiv:2304.09542 [cs]. 1
2023 arXiv
-
[28]
Nandan Thakur, Nils Reimers, Andreas Ruckl’e, Abhishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models.ArXiv, abs/2104.08663. 2
2021 arXiv
-
[29]
Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. 2024. Smarter, better, faster, longer: A modern bidi- rec...
2024 arXiv
-
[30]
Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2024. C-pack: Packed resources for general chinese em- beddings. Preprint, arXiv:2309.07597. 8
2024 arXiv
-
[31]
Yutao Zhu, Huaying Yuan, Shuting Wang, Jiong- nan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, ZhengLiu, ZhichengDou, andJi-RongWen
-
[32]
Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni, Xuanhui Wang, and Michael Bendersky. 2023. Rankt5: Fine- tuning t5 for text ranking with ranking losses. In Proceedings of the 46th International ACM SI- GIR Conference on Research and Development in ...
2023
-
[33]
Shengyao Zhuang, Honglei Zhuang, Bevan Koop- man, and Guido Zuccon. 2023. A Setwise Ap- proach for Effective and Highly Efficient Zero- shot Ranking with Large Language Models.arXiv preprint. ArXiv:2310.09497 [cs]. 1 6 Reranking with Compressed Document Representation A. Selec...
2023 arXiv
-
[2021]
arXiv preprint
Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipeline. arXiv preprint. ArXiv:2101.08751 [cs]. 1
-
[2022]
Preprint, arXiv:2302.12574. 2, 8
-
[2024]
Preprint, arXiv:2308.07107
Large language models for information re- trieval: A survey. Preprint, arXiv:2308.07107. 1
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.