Pith. sign in

REVIEW 2 cited by

DocPuzzle: A Process-Aware Benchmark for Evaluating Realistic Long-Context Reasoning Capabilities

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2502.17807 v1 pith:74W3EDCG submitted 2025-02-25 cs.AI

Tianyi Zhuang , Chuqiao Kuang , Xiaoguang Li , Yihua Teng , Jihao Wu , Yasheng Wang , Lifeng Shang This is my paper · ORCID
classification cs.AI
keywords reasoningmodelsbenchmarkcapabilitiesdocpuzzlelikeevaluatingevaluation
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present DocPuzzle, a rigorously constructed benchmark for evaluating long-context reasoning capabilities in large language models (LLMs). This benchmark comprises 100 expert-level QA problems requiring multi-step reasoning over long real-world documents. To ensure the task quality and complexity, we implement a human-AI collaborative annotation-validation pipeline. DocPuzzle introduces an innovative evaluation framework that mitigates guessing bias through checklist-guided process analysis, establishing new standards for assessing reasoning capacities in LLMs. Our evaluation results show that: 1)Advanced slow-thinking reasoning models like o1-preview(69.7%) and DeepSeek-R1(66.3%) significantly outperform best general instruct models like Claude 3.5 Sonnet(57.7%); 2)Distilled reasoning models like DeepSeek-R1-Distill-Qwen-32B(41.3%) falls far behind the teacher model, suggesting challenges to maintain the generalization of reasoning capabilities relying solely on distillation.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. On the Sensitivity of Instruction-tuned LLMs to Harmful Sentences in Long Inputs

    cs.CL 2025-10 conditional novelty 6.0 of 10

    LLMs detect harmful sentences in long inputs best at around 25% prevalence and when placed early, and worse when sparse, late, or implicit.

  2. LongReasonArena: A Long Reasoning Benchmark for Large Language Models

    cs.CL 2025-08 conditional novelty 6.0 of 10

    A benchmark where AI models must run long algorithmic computations reveals a linear drop in accuracy as the log of reasoning steps increases, with top models scoring 7-16% on the hardest level.

Pith tools