REVIEW 6 cited by
ReCEval: Evaluating Reasoning Chains via Correctness and Informativeness
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Multi-step reasoning ability is fundamental to many natural language tasks, yet it is unclear what constitutes a good reasoning chain and how to evaluate them. Most existing methods focus solely on whether the reasoning chain leads to the correct conclusion, but this answer-oriented view may confound reasoning quality with other spurious shortcuts to predict the answer. To bridge this gap, we evaluate reasoning chains by viewing them as informal proofs that derive the final answer. Specifically, we propose ReCEval (Reasoning Chain Evaluation), a framework that evaluates reasoning chains via two key properties: (1) correctness, i.e., each step makes a valid inference based on information contained within the step, preceding steps, and input context, and (2) informativeness, i.e., each step provides new information that is helpful towards deriving the generated answer. We evaluate these properties by developing metrics using natural language inference models and V-Information. On multiple datasets, we show that ReCEval effectively identifies various error types and yields notable improvements compared to prior methods. We analyze the impact of step boundaries, and previous steps on evaluating correctness and demonstrate that our informativeness metric captures the expected flow of information in high-quality reasoning chains. Finally, we show that scoring reasoning chains based on ReCEval improves downstream task performance. Our code is publicly available at: https://github.com/archiki/ReCEval
Forward citations
Cited by 6 Pith papers
-
Rethinking Human Preference Evaluation of LLM Rationales
A fine-grained attribute-based evaluation of LLM rationales can explain human preferences and reveal model trade-offs that binary comparisons obscure.
-
Knowledge or Reasoning? A Close Look at How LLMs Think Across Domains
LLM reasoning can be scored separately for knowledge and step-by-step information gain, and doing so shows SFT and RL affect these two capacities differently across medicine and math.
-
MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLM
MIRAGE is a benchmark that separates reasoning hallucinations from perception errors in multimodal LLMs, and Logos is a curriculum reinforcement fine-tuning method that reduces logical hallucinations.
-
A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data
A metric-oriented survey that classifies intrinsic quality and trustworthiness metrics for LLM-generated data across six modalities and documents systematic evaluation gaps in the current literature.
-
Retrieval Augmented Decision-Making: A Requirements-Driven, Multi-Criteria Framework for Structured Decision Support
RAD automatically extracts weighted, hierarchical decision criteria from documents and uses LLMs to generate structured decision reports, but its evaluation is largely self-referential.
-
Chain-of-Code Collapse: Reasoning Failures in LLMs via Adversarial Prompting in Code Generation
Prompt rewrites of LeetCode problems cause large accuracy swings in nine LLMs, but invalid negation test cases and inconsistent tables make the headline numbers unreliable.
Discussion (0). Continue with ORCID to comment.