REVIEW 25 cited by
Long-context LLMs Struggle with Long In-context Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs) have made significant strides in handling long sequences. Some models like Gemini could even to be capable of dealing with millions of tokens. However, their performance evaluation has largely been confined to metrics like perplexity and synthetic tasks, which may not fully capture their true abilities in more challenging, real-world scenarios. We introduce a benchmark (LongICLBench) for long in-context learning in extreme-label classification using six datasets with 28 to 174 classes and input lengths from 2K to 50K tokens. Our benchmark requires LLMs to comprehend the entire input to recognize the massive label spaces to make correct predictions. We evaluate on 15 long-context LLMs and find that they perform well on less challenging classification tasks with smaller label space and shorter demonstrations. However, they struggle with more challenging task like Discovery with 174 labels, suggesting a gap in their ability to process long, context-rich sequences. Further analysis reveals a bias towards labels presented later in the sequence and a need for improved reasoning over multiple pieces of information. Our study reveals that long context understanding and reasoning is still a challenging task for the existing LLMs. We believe LongICLBench could serve as a more realistic evaluation for the future long-context LLMs.
Forward citations
Cited by 25 Pith papers
-
Can Large Language Models Match the Conclusions of Systematic Reviews?
On 284 medical questions derived from Cochrane systematic reviews, the best of 24 LLMs, DeepSeek V3, matches expert conclusions 62.40% of the time, and all tested models struggle with uncertain or low-quality evidence.
-
RH-RAG: Trustworthy Long-Form Generation for Privacy-Constrained Settings
A multi-agent RAG framework that adds planning, bounded memory, and NLI-based revision to local 7-8B models, reported to improve faithfulness and coherence in long-form generation.
-
UNIBROWSE: A Data-to-Agent Framework for Multimodal BrowseComp
A unified KG-plus-live-web data pipeline covering all three multimodal BrowseComp information-flow patterns, plus an exploration-degree filter, yields a 35B agent at 54.4 avg accuracy.
-
ThumbnailTruth: A Multi-Modal LLM Approach for Detecting Misleading YouTube Thumbnails Across Diverse Cultural Settings
Claude 3.5 Sonnet, prompted with thumbnails, subtitles, and video summaries, detects misleading YouTube thumbnails with up to 93.8% accuracy on a new cross-country dataset.
-
LLM4MEA: Data-free Model Extraction Attacks on Sequential Recommenders via Large Language Models
An LLM-driven agent generates synthetic interaction sequences that, when queried against a target sequential recommender, produce surrogate models with higher agreement to the target than random or autoregressive data...
-
SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments
SetupBench, a 93-instance environment-bootstrap benchmark, finds coding agents succeed on only 34.4-62.4% of setup tasks, with database configuration and repo setup being the hardest.
-
From General to Targeted Rewards: Surpassing GPT-4 in Open-Ended Long-Context Generation
ProxyReward trains long-form generation models by rewarding how well an AI judge can answer generated yes/no questions about the response, improving open-source models on ProxyQA.
-
LIFELONG SOTOPIA: Evaluating Social Intelligence of Language Agents Over Lifelong Social Interactions
Language agents' believability and goal achievement decline over multi-episode social interactions, and curated memory summaries only partially close the gap with humans.
-
Long-Short Alignment for Effective Long-Context Modeling in LLMs
A long-short misalignment metric quantifies output distribution drift across context lengths, correlates with long-context performance, and a regularizer based on it improves fine-tuned LLMs.
-
e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMs
e3 couples asymmetric skill chaining, negative-gradient RL, and a difficulty/budget curriculum so a 1.7B model extrapolates test-time compute to 2x its training budget and sets reported <2B state-of-the-art on AIME/HMMT 2025.
-
Respecting Temporal-Causal Consistency: Entity-Event Knowledge Graphs for Retrieval-Augmented Generation
A mention-level entity-event knowledge graph for RAG modestly improves temporal-causal question answering on a new narrative benchmark, with gains mostly coming from adding HyDE-style hypothetical answers.
-
ECoRAG: Evidentiality-guided Compression for Long Context RAG
Evidentiality-guided, adaptive context compression improves open-domain QA accuracy and token efficiency over standard RAG and prior compressors.
-
Adaptive Task Vectors for Large Language Models
Adaptive Task Vectors use a small model to generate query-specific steering vectors for frozen LLMs, reporting strong accuracy and generalization, though the theoretical equivalences to LoRA and Prefix-Tuning are not ...
-
Evaluating the Sensitivity of LLMs to Prior Context
Prior conversational context, especially from a different knowledge domain, can sharply reduce LLM multiple-choice accuracy, and repeating the task near the query mitigates the drop.
-
Uncertainty Unveiled: Can Exposure to More In-context Examples Mitigate Uncertainty for Large Language Models?
Giving LLMs more in-context examples reduces predictive uncertainty, mainly by lowering epistemic uncertainty, and often improves accuracy.
-
LIFEBench: Evaluating Length Instruction Following in Large Language Models
LIFEBench's evaluation of 26 LLMs shows most follow short length instructions but degrade sharply beyond a few hundred words, and none reliably hit vendor-claimed maximum output lengths.
-
X$^3$-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment
X3-OPD improves audio-grounded reasoning by training the audio student on its own rollouts with token-level teacher feedback, using a three-tier paired text-audio corpus.
-
Lost in Context: Addressing Context Anxiety in Large Language Models
Context anxiety — abandoning solvable tasks over perceived token limits — is measurable and reducible by fine-tuning on anxiety-free reasoning traces, but the paper's causal mechanism is not actually tested.
-
Vital Trace: Protocol-Constrained Patient-State Reasoning for Longitudinal Clinical Trajectories
TRACE, a multi-agent LLM framework with structured clinical protocols, reports large next-action prediction gains on MIMIC-IV over long-context and RAG baselines, though the evaluation is under-specified.
-
QueryBandits for Hallucination Mitigation: Exploiting Semantic Features for No-Regret Rewriting
A contextual bandit that chooses among five query-rewrite strategies, conditioned on 17 linguistic features, reduces LLM hallucination on QA benchmarks and beats static prompting and no-rewrite baselines.
-
MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning
MAPLE uses graph-influence scores to select and pseudo-label the most useful unlabeled examples, then adaptively chooses demonstrations per query, improving many-shot in-context learning with few human labels.
-
State Compression in Two-Agent LLM Relays: A Closed-World Study of Constraint Preservation
In a two-agent travel-booking relay, structured JSON hand-offs preserved constraint feasibility (0.96) while 250-word narrative summaries collapsed it (0.48), showing hand-off format, not brevity, drives reliability.
-
LPCAN: Lightweight Pyramid Cross-Attention Network for Rail Surface Defect Detection Using RGB-D Data
A lightweight RGB-D cross-attention network is proposed for rail defect detection, but the SOTA accuracy and generalization claims are internally inconsistent and the implementation is not public.
-
LoCoBench: A Benchmark for Long-Context Large Language Models in Complex Software Engineering
LoCoBench introduces a large synthetic long-context software engineering benchmark, but its evaluation tables contradict its own scoring formula and its headline degradation result is inverted by the data it reports.
-
Knowledge-Embedded and Hypernetwork-Guided Few-Shot Substation Meter Defect Image Generation Method
Fine-tuning Stable Diffusion with DreamBooth-style knowledge and hypernetwork-guided crack control maps can synthesize substation meter defect images that boost a YOLOv8 defect detector's mAP when added to the training set.
Discussion (0). Continue with ORCID to comment.