REVIEW 7 cited by
Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Training large language models (LLMs) to spend more time thinking and reflection before responding is crucial for effectively solving complex reasoning tasks in fields such as science, coding, and mathematics. However, the effectiveness of mechanisms like self-reflection and self-correction depends on the model's capacity to accurately assess its own performance, which can be limited by factors such as initial accuracy, question difficulty, and the lack of external feedback. In this paper, we delve into a two-player paradigm that separates the roles of reasoning and critique models, where the critique model provides step-level feedback to supervise the reasoning (actor) model during both test-time and train-time. We first propose AutoMathCritique, an automated and scalable framework for collecting critique data, resulting in a dataset of $76,321$ responses paired with step-level feedback. Fine-tuning language models with this dataset enables them to generate natural language feedback for mathematical reasoning. We demonstrate that the critique models consistently improve the actor's performance on difficult queries at test-time, especially when scaling up inference-time computation. Motivated by these findings, we introduce the critique-based supervision to the actor's self-training process, and propose a critique-in-the-loop self-improvement method. Experiments show that the method improves the actor's exploration efficiency and solution diversity, especially on challenging queries, leading to a stronger reasoning model. Lastly, we take the preliminary step to explore training self-talk reasoning models via critique supervision and showcase its potential. Our code and datasets are at \href{https://mathcritique.github.io/}{https://mathcritique.github.io/}.
Forward citations
Cited by 7 Pith papers
-
Stepwise Reasoning Checkpoint Analysis: A Test Time Scaling Method to Enhance LLMs' Reasoning
A checkpoint-based search and candidate augmentation method improves small LLM mathematical reasoning accuracy over existing test-time scaling baselines.
-
TrimR: Verifier-based Training-Free Thinking Compression for Efficient Test-Time Scaling
TrimR uses a lightweight verifier to detect redundant thoughts in reasoning models and truncates them, cutting runtime by up to 70% without retraining.
-
XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitation
XRPO extends GRPO with adaptive rollout allocation, in-context example seeding for unsolved prompts, and novelty-weighted advantages, reporting roughly 1-4% higher accuracy and faster convergence.
-
R4ec: A Reasoning, Reflection, and Refinement Framework for Recommendation Systems
R4ec trains a small reflection model to critique and refine LLM-generated user and item knowledge, which then improves downstream recommendation accuracy.
-
Making VLMs More Robot-Friendly: Self-Critical Distillation of Low-Level Procedural Reasoning
A self-critique, revision, and verification loop makes small vision-language models produce more detailed and more executable robot plans, beating their own baselines and, on the paper's judge-based evaluation, plans ...
-
Step-Wise Formal Verification for LLM-Based Mathematical Problem Solving
MATH-VF formalizes LLM math solutions into SimpleMath and uses a tool-augmented critic to verify each reasoning step and offer corrective feedback.
-
Large Language Models for Planning: A Comprehensive and Systematic Survey
A structured survey of LLM planning methods, benchmarks, and interpretability work, organized around a three-way taxonomy.
Discussion (0). Continue with ORCID to comment.