Pith. sign in

REVIEW 12 cited by

Compressing Context to Enhance Inference Efficiency of Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.06201 v1 pith:ZQUZLY7V submitted 2023-10-09 cs.CL

Yucheng Li , Bo Dong , Chenghua Lin , Frank Guerin This is my paper · ORCID
classification cs.CL
keywords contextinferenceefficiencyinputlongmemoryperformancereduction
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models (LLMs) achieved remarkable performance across various tasks. However, they face challenges in managing long documents and extended conversations, due to significantly increased computational requirements, both in memory and inference time, and potential context truncation when the input exceeds the LLM's fixed context length. This paper proposes a method called Selective Context that enhances the inference efficiency of LLMs by identifying and pruning redundancy in the input context to make the input more compact. We test our approach using common data sources requiring long context processing: arXiv papers, news articles, and long conversations, on tasks of summarisation, question answering, and response generation. Experimental results show that Selective Context significantly reduces memory cost and decreases generation latency while maintaining comparable performance compared to that achieved when full context is used. Specifically, we achieve a 50\% reduction in context cost, resulting in a 36\% reduction in inference memory usage and a 32\% reduction in inference time, while observing only a minor drop of .023 in BERTscore and .038 in faithfulness on four downstream applications, indicating that our method strikes a good balance between efficiency and performance.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Shapley Context Pruning: A Cooperative Game Perspective for Context Reranking and Pruning

    cs.AI 2026-05 conditional novelty 6.0 of 10

    SCP ranks and prunes context sentences using Shapley values from a learned Deep Sets value function, matching or beating baselines on several multi-hop QA datasets at 50% compression.

  2. When Compression Becomes an Attack Surface: Black-Box Attacks on Prompt-Compressed LLM Agents

    cs.CR 2025-10 reject novelty 6.0 of 10

    The paper claims prompt compression is a new attack surface, but the abstract's COMA attack never appears in the body and the body's SoftCom requires white-box access.

  3. DCMI: A Differential Calibration Membership Inference Attack Against Retrieval-Augmented Generation

    cs.CR 2025-09 conditional novelty 6.0 of 10

    DCMI infers RAG database membership by subtracting the system's yes-probability on a perturbed query from the original query, cancelling the interference of non-member retrieved documents.

  4. MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent

    cs.CL 2025-07 unverdicted novelty 6.0 of 10

    MemAgent uses multi-conversation RL to train a memory agent that reads text in segments and overwrites memory, extrapolating from 8K training to 3.5M token QA with under 5% loss and 95%+ on 512K RULER.

  5. SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling

    cs.LG 2025-05 conditional novelty 6.0 of 10

    SALE is a training-free sparse attention method that uses 4-bit quantized query-key estimates and a relative attention score to skip unimportant blocks, achieving over 3.36x prefill speedup on 64K+ token contexts with...

  6. Not All Tokens Are What You Need In Thinking

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A method that scores each chain-of-thought token by answer-conditioned perplexity and trains models on the compressed traces preserves or improves reasoning accuracy with significantly fewer tokens.

  7. Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning

    cs.AI 2025-05 conditional novelty 6.0 of 10

    Enhancing a model's long-context ability before supervised fine-tuning improves math reasoning accuracy, including on short-input benchmarks.

  8. SearchOS-V1: Towards Robust Open-Domain Information-Seeking Agent Collaboration

    cs.AI 2026-07 conditional novelty 5.0 of 10

    A multi-agent web-search framework that stores progress in shared evidence, coverage, and failure state reports the best F1 scores among compared baselines on WideSearch (80.3 item F1) and GISA (76.5 set F1).

  9. FlexRAG: A Flexible and Comprehensive Framework for Retrieval-Augmented Generation

    cs.CL 2025-06 conditional novelty 5.0 of 10

    FlexRAG is a modular, open-source RAG framework with text, multimodal, and web retrieval, plus evaluation tools and efficient memory-mapped indexing.

  10. 100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?

    cs.CL 2025-05 conditional novelty 5.0 of 10

    100-LongBench and LongScore evaluate LLMs at controlled context lengths and rank models by relative performance drop from a short-context baseline, not by raw accuracy.

  11. SELF: Self-Extend the Context Length With Logistic Growth Function

    cs.CL 2025-05 conditional novelty 5.0 of 10

    SELF replaces Self-Extend's fixed token grouping with a logistic-growth grouping schedule, giving mixed but sometimes large gains on long-context benchmarks.

  12. Hierarchical Reranking for Scalable Financial RAG System

    cs.IR 2026-07 reject novelty 4.0 of 10

    A finance-specific RAG pipeline combining table-to-JSON conversion, two-stage reranking, and long-context split-fusion reports NDCG@20=0.7918 and second place in the ICAIF '24 FinanceRAG challenge.

Pith tools