REVIEW 12 cited by
Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recently, large language models (LLMs) have shown remarkable capabilities including understanding context, engaging in logical reasoning, and generating responses. However, this is achieved at the expense of stringent computational and memory requirements, hindering their ability to effectively support long input sequences. This survey provides an inclusive review of the recent techniques and methods devised to extend the sequence length in LLMs, thereby enhancing their capacity for long-context understanding. In particular, we review and categorize a wide range of techniques including architectural modifications, such as modified positional encoding and altered attention mechanisms, which are designed to enhance the processing of longer sequences while avoiding a proportional increase in computational requirements. The diverse methodologies investigated in this study can be leveraged across different phases of LLMs, i.e., training, fine-tuning and inference. This enables LLMs to efficiently process extended sequences. The limitations of the current methodologies is discussed in the last section along with the suggestions for future research directions, underscoring the importance of sequence length in the continued advancement of LLMs.
Forward citations
Cited by 12 Pith papers
-
Large Language Models are Good Relational Learners
Rel-LLM combines a GNN encoder with a frozen LLM via soft prompts and masked attribute pretraining, reporting improved average performance on RelBench relational database tasks.
-
Evaluating the Sensitivity of LLMs to Prior Context
Prior conversational context, especially from a different knowledge domain, can sharply reduce LLM multiple-choice accuracy, and repeating the task near the query mitigates the drop.
-
EVADE-Bench: Multimodal Benchmark for Evaluating and Enhancing Evasive Content Detection
A new benchmark of 2,833 evasive text samples and 13,961 images shows current LLMs and VLMs frequently miss veiled policy violations in Chinese e-commerce ads.
-
LLMSynthor: Macro-Aligned Micro-Records Synthesis with Large Language Models
LLMSynthor iteratively prompts an LLM to propose corrective batches of micro-records, aligning synthetic data with target macro-statistics while preserving realistic joint dependencies.
-
Top-Theta Attention: Sparsifying Transformers by Compensated Thresholding
Static per-head thresholds calibrated on a small dataset can replace top-k selection in transformer attention, achieving 3 to 10x sparsity with negligible accuracy loss.
-
Lag-Relative Sparse Attention In Long Context Training
Training a language model with LagKV's static sparse attention mask improves its robustness to KV-cache compression at 2x and 4x rates, while claims of overall average gains are not consistently supported.
-
MeVe: A Modular System for Memory Verification and Effective Context Control in Language Models
MeVe, a five-stage modular RAG pipeline, reduces average context tokens by 57-75% versus plain top-k retrieval in a simulated proof-of-concept, without improving answer-quality metrics.
-
Deep Research Agents: A Systematic Examination And Roadmap
A survey that organizes LLM-powered deep research agents into static versus dynamic workflows and single versus multi agent architectures, and reviews their benchmarks and open challenges.
-
Dynamic Chunking and Selection for Reading Comprehension of Ultra-Long Context in Large Language Models
Dynamic chunking plus question-aware chunk selection improves long-context QA, but the headline numbers are partly inflated by choosing hyperparameters on the test benchmarks.
-
Exploring the Landscape of Text-to-SQL with Large Language Models: Progresses, Challenges and Opportunities
A systematic review organizing LLM-based text-to-SQL methods into pre-processing, in-context learning, fine-tuning, and post-processing paradigms, with a catalog of datasets, metrics, challenges, and future directions.
-
Modular Techniques for Synthetic Long-Context Data Generation in Language Model Training and Evaluation
A synthetic long-context data generation framework is described, but with no empirical evaluation or comparison to existing methods.
-
Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architectures
Memory-augmented Transformer research is organized into a three-axis taxonomy bridging neuroscience memory concepts to network designs, but no new result is produced.
Discussion (0). Continue with ORCID to comment.