Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compression

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SARA claims that retrieval-augmented generation can fit tight context budgets by splitting retrieved evidence into fine-grained natural-language spans that preserve critical facts plus single-token semantic compression vectors for the…

desk verdict Good hybrid idea for RAG compression, but the headline gains are partly a fine-tuning artifact; the paper needs a LoRA-tuned RAG control before the claims hold. read the letter →

arxiv 2507.05633 v1 pith:HDMIZZTV submitted 2025-07-08 cs.CL cs.AIcs.IR

Yiqiao Jin , Kartik Sharma , Vineeth Rakesh , Yingtong Dou , Menghai Pan , Mahashweta Das , Srijan Kumar This is my paper
classification cs.CLcs.AIcs.IR
keywords retrieval-augmentedgenerationcontextcompressionsemanticvectorsevidencererankingconditionalself-informationhybridrepresentationlong-contextreasoninginstructiontuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that retrieval-augmented generation does not have to choose between fitting a small context window and keeping the facts straight. Its proposal, SARA, splits retrieved evidence into two forms: a few natural-language spans that keep exact names, dates, and numbers, and compression vectors, single-token embeddings of 256-token chunks that the LLM itself can decode, that convey the rest. An iterative evidence-selection step uses those vectors to pick the next most informative documents, minimizing redundancy while maximizing coverage. Across nine datasets, five open-source LLMs, and multiple retrievers, the hybrid context consistently beats plain RAG and pure-compression baselines under 512- and 1024-token budgets, with the largest margins on knowledge-intensive question answering. The central lesson, as the paper states it, is that compression and local precision are complementary, and that small models can close much of the gap to larger ones simply by how their context is packed.

What carries the argument

The load-bearing object is the compression vector: a chunk's sentence embedding projected by an MLP into the LLM's token space and decoded back by the LLM through a prompt like "The token <C> can be interpreted as: [CHUNK]." Training proceeds in two stages: an autoencoding alignment and context-reconstruction stage (Eqs. 1 and 2) that makes the vectors interpretable, followed by instruction tuning with LoRA that teaches the model to answer from a mixed prompt of k text spans and n−k vectors. The iterative evidence reranker (Algorithm 1) selects contexts either by minimizing the embedding-space discrepancy of the selected set from the query or by maximizing conditional self-information, the surprisal of candidate evidence given what is already selected, to enforce novelty. The working hypothesis is that a single trained vector, despite occupying one token, retains enough semantics to act as a faithful abstractive summary for reasoning.

What would settle it

Construct a question-answering set whose answers require an exact number or named entity, and place the decisive fact only in the compressed-vector portion of the context, never in the k natural-language spans. The paper's own reconstruction evidence (Table 10: '1903' becomes '1900s', '3400 years' becomes 'over 3,000 years') predicts that accuracy on such questions drops relative to the same questions with the fact in the text channel; a large measured drop would show the vector channel cannot be trusted for fact-critical content, narrowing the method's operating envelope to questions answerable from high-level semantics.

Watch

Extended reading notes

Core claim

The central claim is that a retrieved context can be represented at two complementary resolutions and reasoned over jointly: fine-grained natural-language spans that preserve critical entities and numerical values, alongside compact, interpretable compression vectors that carry the high-level semantics of the remaining chunks. The paper argues that this hybrid beats the two pure alternatives, feeding all text, which wastes budget on redundancy, and compressing everything, which loses the details factual answers depend on. The compression vectors are trained through an autoencoding reconstruction objective on a generic Wikipedia corpus, so that the decoder LLM can expand a vector back into its chunk; the paper's own reconstructions (Tables 5 and 10) show high-level semantics survive while exact numbers often blur, and it frames this as the reason to keep decisive facts in the natural-language channel. An iterative evidence reranker then selects documents by novelty, either embedding-space discrepancy from the query or conditional self-information from a small proxy model, so the budget is spent on non-redundant, query-relevant evidence. On this basis the paper reports consistent gains over standard RAG and over the LLMLingua, LongLLMLingua, ICAE, and xRAG baselines, and argues the gains come from using the context budget better rather than from a stronger generator.

Load-bearing premise

A single compression vector per 256-token chunk must preserve exactly the fine-grained facts, numbers, dates, and entity names, that correct answers hinge on, even though it is trained only to reconstruct generic Wikipedia prose and the paper's own reconstructions show exact numeric details routinely blur.

Editorial extensions

If this is right

  • Under a 512-token budget, the hybrid context beats the best compression baselines by 6 to 12 F1 points on knowledge-intensive tasks such as TriviaQA and HotpotQA, and by roughly 15 to 30 percent relative on average.
  • Because the gains come from context packaging rather than raw capability, a 7B model using SARA can match or surpass a 24B model using plain RAG.
  • The reconstruction objective is the most load-bearing training signal: removing it costs 7 to 9 F1 points, removing reranking costs a little, and removing compression hurts most on noisy long-form datasets.
  • The design is modular: improvements persist across five LLMs, sparse and dense retrievers, and four embedding models, with larger gains when the compressor and the LLM share an architecture such as the Mistral family.
  • On held-out datasets the gains appear mainly in answer relevance (up to +55.0 points), while correctness improves only modestly (from +0.3 to +2.2), indicating that evidence selection generalizes but reasoning over compressed content is partly domain-dependent.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A query-dependent adaptation of the k/n ratio, lowering the number of text spans for fact-dense multi-hop queries and raising it for narrative ones, is a natural untested extension; the paper's fixed k=5 sits below the observed optimum around k=7 to 8.
  • The documented blurring of exact numbers in the paper's own reconstructions ('1903' to '1900s', '3400 years' to 'over 3,000 years') defines a stress test the paper does not run: questions whose decisive fact appears only in the vector channel should measurably suffer, delimiting when SARA can be trusted over pure text.
  • Chunk size, fixed at 256 tokens, is a second untested lever; since reconstruction fidelity falls on longer contexts, sweeping chunk size against a fixed token budget would separate the vector channel's capacity from the chunking policy.
  • The conditional-self-information selector is effectively uncertainty-driven evidence gathering with a small proxy model; wiring the decoder's own uncertainty into the selection step is the obvious next move the authors leave implicit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SARA, a RAG framework that, under a 512/1024-token budget, keeps a small number of top retrieved passages as natural language text and encodes the remaining chunks as learned compression vectors produced by a sentence-embedding model plus an MLP projection aligned to the LLM's token space through an autoencoding objective. An iterative evidence-selection step reranks contexts using the compression vectors, either by embedding novelty or by conditional self-information. The authors report F1/ROUGE-L and LLM-based metrics across 9 datasets and 5 open LLMs, claiming consistent gains over RAG, compression, and summarization baselines, with additional ablations and held-out out-of-domain evaluation.

Significance. If the reported gains are causal, SARA would provide a practical recipe for context-efficient RAG: hybrid natural-language plus compressed-vector contexts with iterative reranking, evaluated across model families, retrievers, and embedding models. The paper's strengths include a broad evaluation suite, out-of-domain datasets, ablations that isolate the compression and reranking components, and a design that is explicitly model-agnostic. However, the principal empirical claim currently rests on a comparison in which SARA is LoRA-instruction-tuned on the evaluation datasets while the RAG and compression baselines are not; this confound must be resolved before the significance of the results can be assessed.

major comments (4)
  1. [Section 2.4 and Section 3.1] The instruction-tuning stage applies LoRA to the generator on downstream tasks, and Section 3.1 states that SQuAD-v2, NarrativeQA, QASPER, QuALITY, HotpotQA, and TriviaQA are used for both training and evaluation. The main comparisons (Tables 2-4, Figures 2-3) pit this tuned SARA against vanilla RAG, LLMLingua, LongLLMLingua, ICAE, and xRAG, none of which are LoRA-tuned on those datasets. Because the ablation SARA-C is also LoRA-tuned, it isolates the marginal effect of compression, but the paper never reports a LoRA-tuned RAG control; the headline gains (+17.71 relevance, +13.72 correctness, +15.53 similarity on Mistral-7B) could therefore be driven by task-specific adapter training rather than by the hybrid context representation. Please add a RAG baseline that receives the same LoRA instruction tuning with the same data and hyperparameters, or report SARA without instruction tuning, and present the comparison under both conditions.
  2. [Table 1, Impr.% row] The improvement row is internally inconsistent: on QASPER it reports SARA-EMB's relative gain over GraphRAG (40.55 vs. 37.05 F1 = 9.4%), on NarrativeQA and TriviaQA F1 it uses SARA-CSI, on TriviaQA R-L and QuALITY R-L it uses SARA-EMB, and the reference 'best' baseline shifts between GraphRAG and InstructRAG. Please define SARA-CSI and SARA-EMB in the main text, state exactly which variant and which baseline each improvement is computed against, and recompute the row with a single consistent rule.
  3. [Section 2.3, Appendix D, Tables 5 and 10] The reconstruction examples show that the compression channel systematically distorts exact numerics ('1903' to '1900s', '3400 years' to 'over 3,000 years'), which the paper itself acknowledges. This does not by itself contradict the empirical results, but it leaves open the mechanism by which the vector channel helps in the hybrid setup. Please add an analysis of cases in which the answer span appears only in a compressed chunk and not among the k natural-language contexts, reporting how often SARA answers correctly and whether the reconstructed semantics are the source of the answer. A useful control is replacing the compression vectors with random embeddings of the same shape while keeping the LoRA tuning intact, which would reveal whether the gains come from the learned content or merely from the additional fine-tuning.
  4. [Section 2.3 and Appendix A.1] The compressor architecture is not specified: the paper mentions a 'lightweight compressor–combining a sentence embedding model and an MLP' but does not state which sentence embedding is used in the main experiments, the MLP dimensions, the training data size and number of steps, the optimizer, or whether inference uses one vector per 256-token chunk or per sentence. The code release promised in the text is not linked. Please provide these details or a public anonymous code repository so that the central result can be reproduced.
minor comments (5)
  1. [Table 1] SARA-CSI and SARA-EMB appear in Table 1 without any definition in the main text; please define them when Table 1 is first referenced.
  2. [Equation (5)] Equation (5) defines the conditional self-information as a per-token average, but the following sentence defines I(vi|Vsel) as -log P(vi|Vsel); the notation should be aligned.
  3. [Figure 1] The figure contains labels 'Metric1 (novelty)' and 'Metric2 (relevance)' that are never mapped to Equations (3)-(5) in the caption or text; please explain these labels.
  4. [Section 3.3] The numbers +17.71, +13.72, and +15.53 are attributed to 'On Mistral-7B,' but Figure 3 reports only QASPER; please clarify whether these values are QASPER-only or averaged over datasets.
  5. [Section 3.3] The claim that gains are larger when the compressor and LLM share the same architecture is unclear, since the compressor is described as an embedding model plus MLP; please specify what 'same architecture' means and how this was verified.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SARA's reported gains are external benchmark comparisons; the compression and reranking components are not fitted to the target metrics, and the training/evaluation overlap is a standard supervised benchmark setup rather than a construction-level reduction.

full rationale

The paper's derivation chain is empirical rather than definitional. Eq. (1)-(2) train the compressor to reconstruct chunks and contexts from compression vectors, with the alignment objective Lalign and the reconstruction objective Lrecons; the target metrics (F1, ROUGE-L, relevance, correctness, similarity) never appear in these objectives, so the compression vectors are not fitted to the headline numbers. The evidence-selection rules (Eq. 3 and Eq. 4-5) rerank by embedding discrepancy or conditional self-information, again without reference to the evaluation metrics. The main experimental concern is that Section 2.4 says 'we instruct-tune the LLM generator on downstream tasks with LoRA (Hu et al., 2021) using top-n contexts retrieved via BM25', and Section 3.1 states 'We use SQuAD-v2, NarrativeQA, QASPER, QuALITY, HotpotQA, and TriviaQA for both training and evaluation', so SARA receives task-specific adapter training on the same benchmarks where it is compared against untuned RAG and prompt-compression baselines. This is a genuine comparison and confounding issue, but it is not circularity in the sense used here: the evaluation is on held-out test instances with standard metrics, and the ablations SARA-C, SARA-P, and SARA-R are all LoRA-tuned, so they isolate the marginal contribution of compression, reconstruction, and reranking. The paper further reports out-of-domain results on MultifieldQA-en, 2WikiMultihopQA, and QMSum that were not used for fine-tuning. Self-citations (e.g., Jin et al. 2024b, Liu et al. 2025, Sharma et al. 2024) are contextual and not load-bearing; no uniqueness theorem or ansatz is imported from the authors' prior work. Missing a LoRA-tuned RAG control weakens the strength of the central empirical comparison, but no prediction in the paper reduces to its input by construction, so the circularity score is 0.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The method rests on three load-bearing assumptions: (1) single-vector chunk representations retain fine-grained facts after auto-encoding; (2) embedding distance is a good proxy for information gain during selection; and (3) supervised fine-tuning on six QA datasets transfers to the held-out benchmarks. Of these, (1) is only partially supported by the reconstruction samples, and (3) is only weakly supported for answer correctness. The principal free parameters are chunk size, n, k, and LoRA settings, all chosen by hand.

free parameters (5)
  • chunk size = 256 tokens
    Chosen for all datasets; no sensitivity analysis on chunk size is provided, though compression fidelity is known to degrade with longer contexts.
  • number of retrieved chunks n = 10
    Maximum number of chunks processed; this limits how much of the corpus is considered.
  • number of natural-language contexts k = 5
    Sensitivity analysis (Figure 5) shows performance peaks around 7-8, so the fixed value of 5 may not be optimal across datasets.
  • LoRA hyperparameters = rank=16, alpha=32, dropout=0.1
    Ad hoc choices; no grid search reported.
  • aggregation function in Eq. 3 = unspecified (mean assumed)
    The paper never defines Aggregate(...) for selected context embeddings, which affects ordering.
assumptions (3)
  • domain assumption A single projected sentence-embedding vector per 256-token chunk retains enough fine-grained information for the LLM to answer questions after auto-encoding fine-tuning.
    The entire compressed channel depends on this; the paper's own reconstructions (Tables 5, 10) show systematic loss of numeric detail, so the assumption is only partially validated.
  • domain assumption Embedding-space distance is a reliable proxy for incremental evidence value in Eq. 3.
    Algorithm 1 selects evidence by minimizing the distance between the query vector aggregate and the selected set; no theoretical or independent empirical justification is given.
  • domain assumption Fine-tuning on the training splits of QASPER, NarrativeQA, TriviaQA, QuALITY, HotpotQA, and SQuAD-v2 transfers to held-out datasets.
    The paper evaluates OOD datasets but correctness gains there are much smaller (+0.3 to +2.2) than relevance gains, indicating limited transfer of reasoning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compression." pith.science (2026). https://pith.science/paper/HDMIZZTV

@misc{pith2026250705633,
  author       = {Pith},
  title        = {Pith review of: SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HDMIZZTV}},
  note         = {Machine review of arXiv:2507.05633}
}
read the original abstract

Retrieval-augmented Generation (RAG) extends large language models (LLMs) with external knowledge but faces key challenges: restricted effective context length and redundancy in retrieved documents. Pure compression-based approaches reduce input size but often discard fine-grained details essential for factual accuracy. We propose SARA, a unified RAG framework that balances local precision and global knowledge coverage under tight context budgets. SARA combines natural-language text snippets with semantic compression vectors to jointly enhance context efficiency and answer correctness. It represents contexts at two complementary levels: 1) fine-grained natural-language spans that preserve critical entities and numerical values, and 2) compact, interpretable vectors that summarize high-level semantics. An iterative evidence-selection module employs the compression vectors for dynamic reranking of contexts. Across 9 datasets and 5 open-source LLMs spanning 3 model families (Mistral, Llama, and Gemma), SARA consistently improves answer relevance (+17.71), answer correctness (+13.72), and semantic similarity (+15.53), demonstrating the importance of integrating textual and compressed representations for robust, context-efficient RAG.

Figures

Figures reproduced from arXiv: 2507.05633 by the authors.

Figure 1
Figure 1. SARA reasons over a mixture of compressed evidence and natural language contexts [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Performance of RAG and SARA across different LLMs in terms of LLM-based metrics on [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 2
Figure 2. Generalizability across models. We report lexical metrics (F1 score and ROUGE￾L) on QASPER (Dasigi et al., 2021) before and after applying SARA. In general, performance gains are more significant when the compressor and LLM share the same ar￾chitecture (e.g. Mistral). Among the Mistral family, we observe an average boost in Answer Relevance of 20.12 and Answer Correctness of 7.07. MistralNemo and MistralSmall achiev… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Performance of SARA’s vari￾ants. Context Reconstruction is Critical. Removing the re￾construction objective (SARA-P) results in the most sub￾stantial performance drop ( [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Sensitivity analysis with total contexts fixed at [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: During Compression Learning, SARA learns to reconstruct text from compression vectors. For embedding alignment (Section 2.3), we adopt a curriculum learning strategy, starting with shorter sentences and gradually transition into complex examples. Specifically, we use s…
Figure 7
Figure 7. Figure 7: Distribution of number of tokens per chunk in each dataset. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Results on different compressors. B Additional Experiments B.1 Intrinsic Analysis of Compression Vectors B.2 Generalization on Additional Embedding Models Aside from Salesforce/SFR-Embedding-Mistral (SFR), we experimented with additional embeddings, including Linq-AI-R…
Figure 9
Figure 9. Figure 9: Number of words generated from compression vectors when we vary from 1 to 3 sentences. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Context Size in terms of number of tokens according to Mistral-7B’s tokenizer. All [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SlideAgent: Hierarchical Agentic Framework for Multi-Page Visual Document Understanding

    cs.CL 2025-10 unverdicted novelty 6.0 of 10

    SARA combines natural-language snippets with semantic compression vectors in RAG to improve answer relevance, correctness, and similarity on 9 datasets across 5 LLMs.

Reference graph

Works this paper leans on

73 extracted references · 46 canonical work pages · cited by 1 Pith paper

  1. [1]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  2. [2]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  3. [3]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    work page

  4. [4]

    Self-rag: Learning to retrieve, generate, and critique through self-reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In ICLR, 2023

    work page 2023

  5. [5]

    Longbench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. In ACL, pages 3119--3137, 2024

    work page 2024

  6. [6]

    Curriculum learning

    Yoshua Bengio, J \'e r \^o me Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML, pages 41--48, 2009

    2009

  7. [7]

    xrag: Extreme context compression for retrieval-augmented generation with one token

    Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Si-Qing Chen, Furu Wei, Huishuai Zhang, and Dongyan Zhao. xrag: Extreme context compression for retrieval-augmented generation with one token. arXiv:2405.13792, 2024

    arXiv 2024

  8. [8]

    Adapting language models to compress contexts

    Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. In EMNLP, pages 3829--3846, 2023

    work page 2023

Show all 73 references
  1. [9]

    Provence: efficient and robust context pruning for retrieval-augmented generation

    Nadezhda Chirkova, Thibault Formal, Vassilina Nikoulina, and St \'e phane Clinchant. Provence: efficient and robust context pruning for retrieval-augmented generation. arXiv:2501.16214, 2025

    2025 arXiv

  2. [10]

    A dataset of information-seeking questions and answers anchored in research papers

    Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of information-seeking questions and answers anchored in research papers. In NAACL, pages 4599--4610, 2021

    2021

  3. [11]

    Longnet: Scaling transformers to 1,000,000,000 tokens

    Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng, and Furu Wei. Longnet: Scaling transformers to 1,000,000,000 tokens. arXiv:2307.02486, 2023

    2023 arXiv

  4. [12]

    From local to global: A graph rag approach to query-focused summarization

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. From local to global: A graph rag approach to query-focused summarization. arXiv:2404.16130, 2024

    2024 arXiv

  5. [13]

    Ragas: Automated evaluation of retrieval augmented generation

    Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. Ragas: Automated evaluation of retrieval augmented generation. In EACL, pages 150--158, 2024

    2024

  6. [14]

    Enabling large language models to generate text with citations

    Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations. In EMNLP, pages 6465--6488. ACL, 2023

    2023

  7. [15]

    In-context autoencoder for context compression in a large language model

    Tao Ge, Hu Jing, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model. In ICLR, 2024

    2024

  8. [16]

    Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In COLING, pages 6609--6625, 2020

    2020

  9. [17]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2021

    2021

  10. [18]

    Exit: Context-aware extractive compression for enhancing retrieval-augmented generation

    Taeho Hwang, Sukmin Cho, Soyeong Jeong, Hoyun Song, SeungYoon Han, and Jong C Park. Exit: Context-aware extractive compression for enhancing retrieval-augmented generation. arXiv:2412.12559, 2024

    2024 arXiv

  11. [19]

    Atlas: Few-shot learning with retrieval augmented language models

    Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented language models. JMLR, 24 0 (251): 0 1--43, 2023

    2023

  12. [20]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023 a

    arXiv

  13. [21]

    Llmlingua: Compressing prompts for accelerated inference of large language models

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Llmlingua: Compressing prompts for accelerated inference of large language models. In EMNLP, pages 13358--13376, 2023 b

    2023

  14. [22]

    Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression

    Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression. In ACL, 2024

    2024

  15. [23]

    Llm maybe longlm: Selfextend llm context window without tuning

    Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. Llm maybe longlm: Selfextend llm context window without tuning. In ICML, pages 22099--22114, 2024 a

    2024

  16. [24]

    Better to ask in english: Cross-lingual evaluation of large language models for healthcare queries

    Yiqiao Jin, Mohit Chandra, Gaurav Verma, Yibo Hu, Munmun De Choudhury, and Srijan Kumar. Better to ask in english: Cross-lingual evaluation of large language models for healthcare queries. In Web Conference, pages 2627--2638, 2024 b

    2024

  17. [25]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In ACL, pages 1601--1611, 2017

    2017

  18. [26]

    Compressed context memory for online language model interaction

    Jang-Hyun Kim, Junyoung Yeom, Sangdoo Yun, and Hyun Oh Song. Compressed context memory for online language model interaction. In ICLR, 2024 a

    2024

  19. [27]

    Linq-embed-mistral:elevating text retrieval with improved gpt data through task-specific control and quality refinement

    Junseong Kim, Seolhwa Lee, Jihoon Kwon, Sangmo Gu, Yejin Kim, Minkyung Cho, Jy-yong Sohn, and Chanyeol Choi. Linq-embed-mistral:elevating text retrieval with improved gpt data through task-specific control and quality refinement. Linq AI Research Blog, 2024 b . URL https://get...

    2024

  20. [28]

    The narrativeqa reading comprehension challenge

    Tom \'a s Ko c isk \`y , Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, G \'a bor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6: 0 317--328, 2018

    2018

  21. [29]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. NeurIPS, 33: 0 9459--9474, 2020

    2020

  22. [30]

    Making large language models a better foundation for dense retrieval

    Chaofan Li, Zheng Liu, Shitao Xiao, and Yingxia Shao. Making large language models a better foundation for dense retrieval. arXiv:2312.15503, 2023 a

    2023

  23. [31]

    Graphreader: Building graph-based agent to enhance long-context abilities of large language models

    Shilong Li, Yancheng He, Hangyu Guo, Xingyuan Bu, Ge Bai, Jie Liu, Jiaheng Liu, Xingwei Qu, Yangguang Li, Wanli Ouyang, et al. Graphreader: Building graph-based agent to enhance long-context abilities of large language models. In EMNLP, pages 12758--12786, 2024 a

    2024

  24. [32]

    Understanding retrieval robustness for retrieval-augmented image captioning

    Wenyan Li, Jiaang Li, Rita Ramos, Raphael Tang, and Desmond Elliott. Understanding retrieval robustness for retrieval-augmented image captioning. In ACL, pages 9285--9299, 2024 b

    2024

  25. [33]

    Towards general text embeddings with multi-stage contrastive learning

    Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv:2308.03281, 2023 b

    2023 arXiv

  26. [34]

    Grappi: A retrieve-divide-solve graphrag framework for large-scale protein-protein interaction exploration

    Ziwen Li, Xiang'Anthony' Chen, and Youngseung Jeon. Grappi: A retrieve-divide-solve graphrag framework for large-scale protein-protein interaction exploration. In NAACL, 2025

    2025

  27. [35]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81, 2004

    2004

  28. [36]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36: 0 34892--34916, 2023

    2023

  29. [37]

    LlamaIndex

    Jerry Liu. LlamaIndex . https://github.com/jerryjliu/llama_index, 11 2022. DOI: 10.5281/zenodo.1234 https://doi.org/10.5281/zenodo.1234

    2022 doi

  30. [38]

    Culturevlm: Characterizing and improving cultural understanding of vision-language models for over 100 countries

    Shudong Liu, Yiqiao Jin, Cheng Li, Derek F Wong, Qingsong Wen, Lichao Sun, Haipeng Chen, Xing Xie, and Jindong Wang. Culturevlm: Characterizing and improving cultural understanding of vision-language models for over 100 countries. arXiv:2501.01282, 2025

    2025 arXiv

  31. [39]

    Pisco: Pretty simple compression for retrieval-augmented generation

    Maxime Louis, Herv \'e D \'e jean, and St \'e phane Clinchant. Pisco: Pretty simple compression for retrieval-augmented generation. arXiv:2501.16075, 2025 a

    2025 arXiv

  32. [40]

    Oscar: Online soft compression and reranking

    Maxime Louis, Thibault Formal, Herv \'e Dejean, and St \'e phane Clinchant. Oscar: Online soft compression and reranking. arXiv:2504.07109, 2025 b

    2025

  33. [41]

    Semi-supervised fine-tuning for large language models

    Junyu Luo, Xiao Luo, Xiusi Chen, Zhiping Xiao, Wei Ju, and Ming Zhang. Semi-supervised fine-tuning for large language models. In NAACL, pages 2795--2808, 2025

    2025

  34. [42]

    Sfr-embedding-mistral:enhance text retrieval with transfer learning

    Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfr-embedding-mistral:enhance text retrieval with transfer learning. Salesforce AI Research Blog, 2024. URL https://www.salesforce.com/blog/sfr-embedding/

    2024

  35. [43]

    Learning to compress prompts with gist tokens

    Jesse Mu, Xiang Li, and Noah Goodman. Learning to compress prompts with gist tokens. NeurIPS, 36: 0 19327--19352, 2023

    2023

  36. [44]

    Mteb: Massive text embedding benchmark

    Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. In ACL, pages 2014--2037, 2023

    2014

  37. [45]

    Leave no context behind: Efficient infinite context transformers with infini-attention

    Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave no context behind: Efficient infinite context transformers with infini-attention. arXiv:2404.07143, 2024

    2024 arXiv

  38. [46]

    Gpt-4o, 2025

    OpenAI. Gpt-4o, 2025. URL https://chat.openai.com/

    2025

  39. [47]

    Llmlingua-2: Data distillation for efficient and faithful task-agnostic prompt compression

    Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor R \"u hle, Yuqing Yang, Chin-Yew Lin, et al. Llmlingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. In ACL, pages 963--981, 2024

    2024

  40. [48]

    Quality: Question answering with long input texts, yes! In NAACL, pages 5336--5358, 2022

    Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, et al. Quality: Question answering with long input texts, yes! In NAACL, pages 5336--5358, 2022

    2022

  41. [49]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, volume 32, 2019

    2019

  42. [50]

    Know what you don’t know: Unanswerable questions for squad

    Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for squad. In ACL, pages 784--789, 2018

    2018

  43. [51]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In EMNLP. Association for Computational Linguistics, 11 2019. URL https://arxiv.org/abs/1908.10084

    2019 arXiv

  44. [52]

    Semantic answer similarity for evaluating question answering models

    Julian Risch, Timo M \"o ller, Julian Gutsch, and Malte Pietsch. Semantic answer similarity for evaluating question answering models. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 149--157, 2021

    2021

  45. [53]

    Simple bm25 extension to multiple weighted fields

    Stephen Robertson, Hugo Zaragoza, and Michael Taylor. Simple bm25 extension to multiple weighted fields. In CIKM, pages 42--49, 2004

    2004

  46. [54]

    Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D. Manning. Raptor: Recursive abstractive processing for tree-organized retrieval. In ICLR, 2024

    2024

  47. [55]

    A mathematical theory of communication

    Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27 0 (3): 0 379--423, 1948

    1948

  48. [56]

    Og-rag: Ontology-grounded retrieval-augmented generation for large language models

    Kartik Sharma, Peeyush Kumar, and Yunqing Li. Og-rag: Ontology-grounded retrieval-augmented generation for large language models. arXiv:2412.15235, 2024

    2024 arXiv

  49. [57]

    Mpnet: Masked and permuted pre-training for language understanding

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. NeurIPS, 33: 0 16857--16867, 2020

    2020

  50. [58]

    Searching for best practices in retrieval-augmented generation

    Xiaohua Wang, Zhenghua Wang, Xuan Gao, Feiran Zhang, Yixin Wu, Zhibo Xu, Tianyuan Shi, Zhengyuan Wang, Shizheng Li, Qi Qian, et al. Searching for best practices in retrieval-augmented generation. In EMNLP, 2024

    2024

  51. [59]

    A survey on curriculum learning

    Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on curriculum learning. IEEE transactions on pattern analysis and machine intelligence, 44 0 (9): 0 4555--4576, 2021

    2021

  52. [60]

    Instructrag: Instructing retrieval-augmented generation via self-synthesized rationales

    Zhepei Wei, Wei-Lin Chen, and Yu Meng. Instructrag: Instructing retrieval-augmented generation via self-synthesized rationales. In ICLR, 2025

    2025

  53. [61]

    Transformers: State-of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing. In EMNLP, pages 38--45, 2020

    2020

  54. [62]

    Large language models can be good privacy protection learners

    Yijia Xiao, Yiqiao Jin, Yushi Bai, Yue Wu, Xianjun Yang, Xiao Luo, Wenchao Yu, Xujiang Zhao, Yanchi Liu, Haifeng Chen, et al. Large language models can be good privacy protection learners. In EMNLP, 2024 a

    2024

  55. [63]

    Proteingpt: Multimodal llm for protein property prediction and structure understanding

    Yijia Xiao, Edward Sun, Yiqiao Jin, Qifan Wang, and Wei Wang. Proteingpt: Multimodal llm for protein property prediction and structure understanding. arXiv preprint arXiv:2408.11363, 2024 b

    2024 arXiv

  56. [64]

    Recomp: Improving retrieval-augmented lms with compression and selective augmentation

    Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: Improving retrieval-augmented lms with compression and selective augmentation. In ICLR, 2024

    2024

  57. [65]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In EMNLP, pages 2369--2380, 2018

    2018

  58. [66]

    Compact: Compressing retrieved documents actively for question answering

    Chanwoong Yoon, Taewhoo Lee, Hyeon Hwang, Minbyul Jeong, and Jaewoo Kang. Compact: Compressing retrieved documents actively for question answering. In EMNLP, 2024

    2024

  59. [67]

    Rankrag: Unifying context ranking with retrieval-augmented generation in llms

    Yue Yu, Wei Ping, Zihan Liu, Boxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi, and Bryan Catanzaro. Rankrag: Unifying context ranking with retrieval-augmented generation in llms. In NeurIPS, 2024

    2024

  60. [68]

    Jasper and stella: distillation of sota embedding models

    Dun Zhang, Jiacheng Li, Ziyang Zeng, and Fulong Wang. Jasper and stella: distillation of sota embedding models. arXiv:2412.19048, 2024 a

    2024 arXiv

  61. [69]

    Adacomp: Extractive context compression with adaptive predictor for retrieval-augmented large language models

    Qianchi Zhang, Hainan Zhang, Liang Pang, Hongwei Zheng, and Zhiming Zheng. Adacomp: Extractive context compression with adaptive predictor for retrieval-augmented large language models. arXiv:2409.01579, 2024 b

    2024 arXiv

  62. [70]

    Found in the middle: How language models use long contexts better via plug-and-play positional encoding

    Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, and Zhangyang Wang. Found in the middle: How language models use long contexts better via plug-and-play positional encoding. In NeurIPS, 2024 c

    2024

  63. [71]

    Competeai: Understanding the competition behaviors in large language model-based agents

    Qinlin Zhao, Jindong Wang, Yixuan Zhang, Yiqiao Jin, Kaijie Zhu, Hao Chen, and Xing Xie. Competeai: Understanding the competition behaviors in large language model-based agents. In ICML, 2024

    2024

  64. [72]

    Qmsum: A new benchmark for query-based multi-domain meeting summarization

    Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. Qmsum: A new benchmark for query-based multi-domain meeting summarization. In NAACL, pages 5905--5921, 2021

    2021

  65. [73]

    Generalizing from short to long: Effective data synthesis for long-context instruction tuning

    Wenhao Zhu, Pinzhen Chen, Hanxu Hu, Shujian Huang, Fei Yuan, Jiajun Chen, and Alexandra Birch. Generalizing from short to long: Effective data synthesis for long-context instruction tuning. arXiv:2502.15592, 2025

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.