REVIEW 4 cited by
Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs) exhibit positional bias, struggling to utilize information from the middle or end of long contexts. Our study explores LLMs' long-context reasoning by probing their hidden representations. We find that while LLMs encode the position of target information, they often fail to leverage this in generating accurate responses. This reveals a disconnect between information retrieval and utilization, a "know but don't tell" phenomenon. We further analyze the relationship between extraction time and final accuracy, offering insights into the underlying mechanics of transformer models.
Forward citations
Cited by 4 Pith papers
-
Controlling the Risk of Corrupted Contexts for Language Models via Early-Exiting
An early-exit rule with a zero-shot fallback, calibrated by Learn-then-Test risk control, keeps the average loss from corrupted in-context demonstrations under a preset bound.
-
Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding
Positional Contrastive Decoding, a training-free method that contrasts standard and over-rotated RoPE logits, improves long-context retrieval and QA by a few points.
-
Enhancing Visual Reliance in Text Generation: A Bayesian Perspective on Mitigating Hallucination in Large Vision-Language Models
EVRB is a three-part inference-time method that prunes ambiguous visual tokens, divides the model's output distribution by a text-only prior, and triggers early stopping to reduce hallucination in LVLMs.
-
Extracting Visual Facts from Intermediate Layers for Mitigating Hallucinations in Multimodal Large Language Models
Selecting the intermediate layer where image-conditioned and text-only predictions diverge most, and adding that layer's contrastive visual signal back to the final logits, reduces object hallucinations in four large ...
Discussion (0). Sign in to comment.