REVIEW 5 cited by
PRELUDE: A Benchmark Designed to Require Global Comprehension and Reasoning over Long Contexts
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
PRELUDE: A Benchmark Designed to Require Global Comprehension and Reasoning over Long Contexts
read the original abstract
We introduce PRELUDE, a benchmark for evaluating long-context understanding through the task of determining whether a character's prequel story is consistent with the canonical narrative of the original book. Our task poses a stronger demand for global comprehension and deep reasoning than existing benchmarks -- as the prequels are not part of the original story, assessing their plausibility typically requires searching and integrating information that is only indirectly related. Empirically, 88% of instances require evidence from multiple parts of the narrative. Experimental results highlight the challenge of our task: in-context learning, RAG and in-domain training with state-of-the-art LLMs, and commercial DeepResearch services, lag behind humans by >15%. A further human study reveals that models often produce correct answers with flawed reasoning, leading to an over 30% gap in reasoning accuracy compared to humans. These findings underscore the substantial room for improvement in long-context understanding and reasoning.
Forward citations
Cited by 5 Pith papers
-
Many-Shot CoT-ICL: Making In-Context Learning Truly Learn
Many-shot CoT-ICL functions as test-time learning when demonstrations are ordered for smooth conceptual progression rather than similarity, enabling a new selection method that improves reasoning performance.
-
A New Role for Relevance: Guiding Corpus Interaction in Agentic Search
Ordering ripgrep by embedding relevance, seeding entry paragraphs, and reranking matches yields higher accuracy with fewer tool calls than RISE, DCI, and retrieval agents.
-
Many-Shot CoT-ICL: Making In-Context Learning Truly Learn
Many-shot CoT-ICL improves when demonstrations are ordered for smooth conceptual progression, with CDS delivering up to 5.42 percentage-point gains on math tasks using 64 examples.
-
HGMEM: Hypergraph-based Working Memory to Improve Multi-step RAG for Long-Context Complex Relational Modeling
A working memory represented as a hypergraph, whose hyperedges are updated, inserted, and progressively merged by the LLM, improves multi-step RAG on long-context sense-making benchmarks.
-
Towards High-Level Semantic Intelligence
A survey proposing that AI's next stage should be understood as High-Level Semantic Intelligence: mastering humor, sarcasm, metaphor, empathy, persuasion, and narrative across modalities.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.