Pith. sign in

REVIEW 5 cited by

DetectiveQA: Evaluating Long-Context Reasoning on Detective Novels

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2409.02465 v2 pith:FTPH3EWG submitted 2024-09-04 cs.CL

Zhe Xu , Jiasheng Ye , Xiaoran Liu , Xiangyang Liu , Tianxiang Sun , Zhigeng Liu , Qipeng Guo , Linlin Li
show 3 more authors
Qun Liu Xuanjing Huang Xipeng Qiu
This is my paper · ORCID
classification cs.CL
keywords reasoninglong-contextllmschallengesdatasetdetectivedetectiveqanovels
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Recently, significant efforts have been devoted to enhancing the long-context capabilities of Large Language Models (LLMs), particularly in long-context reasoning. To facilitate this research, we propose \textbf{DetectiveQA}, a dataset specifically designed for narrative reasoning within long contexts. We leverage detective novels, averaging over 100k tokens, to create a dataset containing 1200 human-annotated questions in both Chinese and English, each paired with corresponding reference reasoning steps. Furthermore, we introduce a step-wise reasoning metric, which enhances the evaluation of LLMs' reasoning processes. We validate our approach and evaluate the mainstream LLMs, including GPT-4, Claude, and LLaMA, revealing persistent long-context reasoning challenges and demonstrating their evidence-retrieval challenges. Our findings offer valuable insights into the study of long-context reasoning and lay the base for more rigorous evaluations.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions

    cs.CL 2025-07 unverdicted novelty 7.0 of 10

    MemoryAgentBench is a new multi-turn benchmark assessing four memory competencies in LLM agents—accurate retrieval, test-time learning, long-range understanding, and selective forgetting—showing that existing methods ...

  2. When Your Agent Opens the Chat App: Agent-Controlled Search over Raw Chat Logs Rivals Structured Memory

    cs.CL 2026-08 conditional novelty 6.0 of 10

    ReFind, an agent-controlled keyword search over raw chat logs, outperforms structured memory systems such as HippoRAG 2 on conversational-memory benchmarks without building any semantic index.

  3. HGMEM: Hypergraph-based Working Memory to Improve Multi-step RAG for Long-Context Complex Relational Modeling

    cs.CL 2025-12 conditional novelty 6.0 of 10

    A working memory represented as a hypergraph, whose hyperedges are updated, inserted, and progressively merged by the LLM, improves multi-step RAG on long-context sense-making benchmarks.

  4. Evaluation Hallucination in Multi-Round Incomplete Information Lateral-Driven Reasoning Tasks

    cs.CL 2025-05 conditional novelty 5.0 of 10

    LLM-as-judge scoring of multi-round lateral thinking tasks can be fooled by answer leakage and question substitution, so response-based metrics may overstate reasoning ability.

  5. Towards High-Level Semantic Intelligence

    cs.AI 2026-07 conditional novelty 4.0 of 10

    A survey proposing that AI's next stage should be understood as High-Level Semantic Intelligence: mastering humor, sarcasm, metaphor, empathy, persuasion, and narrative across modalities.

Pith tools