REVIEW 7 cited by
LLMs can Find Mathematical Reasoning Mistakes by Pedagogical Chain-of-Thought
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Self-correction is emerging as a promising approach to mitigate the issue of hallucination in Large Language Models (LLMs). To facilitate effective self-correction, recent research has proposed mistake detection as its initial step. However, current literature suggests that LLMs often struggle with reliably identifying reasoning mistakes when using simplistic prompting strategies. To address this challenge, we introduce a unique prompting strategy, termed the Pedagogical Chain-of-Thought (PedCoT), which is specifically designed to guide the identification of reasoning mistakes, particularly mathematical reasoning mistakes. PedCoT consists of pedagogical principles for prompts (PPP) design, two-stage interaction process (TIP) and grounded PedCoT prompts, all inspired by the educational theory of the Bloom Cognitive Model (BCM). We evaluate our approach on two public datasets featuring math problems of varying difficulty levels. The experiments demonstrate that our zero-shot prompting strategy significantly outperforms strong baselines. The proposed method can achieve the goal of reliable mathematical mistake identification and provide a foundation for automatic math answer grading. The results underscore the significance of educational theory, serving as domain knowledge, in guiding prompting strategy design for addressing challenging tasks with LLMs effectively.
Forward citations
Cited by 7 Pith papers
-
P-CoT: A Pedagogically-motivated Participatory Chain-of-Thought Prompting for Phonological Reasoning in LLMs
P-CoT prompting improves many LLM results on PhonologyBench tasks, but it does not consistently beat baselines across all models and tasks as the paper claims.
-
Ask-Before-Detection: Identifying and Mitigating Conformity Bias in LLM-Powered Error Detector for Math Word Problem Solutions
LLM error detectors favor conventional solution formats; generating an adaptive reference solution before grading mitigates this conformity bias on a 200-example GSM8K subset.
-
CoDAE: Adapting Large Language Models for Education via Chain-of-Thought Data Augmentation
Fine-tuning four open-source LLMs on LLM-generated Socratic guidance data changes tutor behavior, reducing answer disclosure on some models, but gains are inconsistent and are measured by an LLM judge rather than huma...
-
The potential -- and the pitfalls -- of using pre-trained language models as cognitive science theories
Pretrained language models can serve as credible cognitive science theories only if researchers validate linking hypotheses and avoid pitfalls of commission and omission.
-
Enhancing LLM Reasoning with Multi-Path Collaborative Reactive and Reflection agents
A multi-path, reactive-plus-reflection agent framework improves gpt-3.5-turbo accuracy on MMLU physics, math, and moral reasoning subsets compared with CoT, self-consistency, and self-refine baselines.
-
Systematic Optimization of Open Source Large Language Models for Mathematical Reasoning
A hyperparameter search for LLM math reasoning that reports simulated, not measured, performance gains.
-
Evaluation of LLMs for mathematical problem solving
A three-model, three-dataset LLM math evaluation using a multi-dimensional reasoning rubric, undermined by contradictory accuracy tables.
Discussion (0). Continue with ORCID to comment.