pith. sign in

arxiv: 2305.20050 · v1 · submitted 2023-05-31 · 💻 cs.LG · cs.AI· cs.CL

Let's Verify Step by Step

Hunter Lightman , Vineet Kosaraju , Yura Burda , Harri Edwards , Bowen Baker , Teddy Lee , Jan Leike , John Schulman
show 2 more authors
Ilya Sutskever Karl Cobbe
This is my paper

Pith reviewed 2026-05-10 15:30 UTC · model grok-4.3

classification 💻 cs.LG cs.AIcs.CL
keywords process supervisionoutcome supervisionMATH datasetprocess reward modellarge language modelsreasoningactive learningPRM800K
0
0 comments X

The pith

Process supervision outperforms outcome supervision for training models to solve MATH problems.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper compares outcome supervision, which rewards only a correct final answer, with process supervision, which rewards each correct intermediate reasoning step. The authors train large language models on the challenging MATH dataset and find that process supervision produces substantially more accurate solutions. Their best process-supervised model reaches 78 percent accuracy on a representative subset of the MATH test set. They further show that active learning makes the collection of step-level labels more effective and release the full set of 800,000 human step labels used in their experiments.

Core claim

Process supervision significantly outperforms outcome supervision for training models to solve problems from the challenging MATH dataset. Our process-supervised model solves 78% of problems from a representative subset of the MATH test set. Additionally, we show that active learning significantly improves the efficacy of process supervision.

What carries the argument

A process reward model trained on human-provided step-level correctness labels that scores each intermediate reasoning step rather than only the final answer.

Load-bearing premise

The collected human step-level labels are consistent and unbiased enough to produce a reward model that generalizes to unseen problems.

What would settle it

A head-to-head test on the full MATH test set in which the process-supervised model solves no more problems than a comparably trained outcome-supervised model.

read the original abstract

In recent years, large language models have greatly improved in their ability to perform complex multi-step reasoning. However, even state-of-the-art models still regularly produce logical mistakes. To train more reliable models, we can turn either to outcome supervision, which provides feedback for a final result, or process supervision, which provides feedback for each intermediate reasoning step. Given the importance of training reliable models, and given the high cost of human feedback, it is important to carefully compare the both methods. Recent work has already begun this comparison, but many questions still remain. We conduct our own investigation, finding that process supervision significantly outperforms outcome supervision for training models to solve problems from the challenging MATH dataset. Our process-supervised model solves 78% of problems from a representative subset of the MATH test set. Additionally, we show that active learning significantly improves the efficacy of process supervision. To support related research, we also release PRM800K, the complete dataset of 800,000 step-level human feedback labels used to train our best reward model.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript investigates process versus outcome supervision for training large language models on multi-step mathematical reasoning tasks from the MATH dataset. The central finding is that process supervision significantly outperforms outcome supervision, with the best process-supervised model solving 78% of problems on a representative subset of the MATH test set. The authors additionally demonstrate that active learning improves the efficacy of process supervision and release the PRM800K dataset of 800,000 step-level human feedback labels.

Significance. If the reported outperformance generalizes, the work provides valuable empirical evidence favoring process supervision for building more reliable LLM reasoners on challenging benchmarks. The 78% solve rate is a notable quantitative result, and the public release of PRM800K is a clear strength that will support reproducible follow-on research. The grounding in independent human labels on held-out problems avoids circularity and strengthens the evaluation.

major comments (2)
  1. [Abstract] Abstract: the 78% solve rate and the claim of significant outperformance over outcome supervision are both measured on a 'representative subset' of the MATH test set. No statistical confirmation is provided (e.g., Kolmogorov-Smirnov test or chi-squared comparison of difficulty levels 1-5 and category distributions) that the subset matches the full test distribution. This is load-bearing for the headline conclusion that process supervision is superior for the MATH dataset, as any post-hoc selection or skew could inflate both the absolute figure and the gap versus outcome supervision.
  2. [Experimental results] Experimental results section: the comparison between process and outcome supervision should explicitly document controls for model size and total training compute to rule out the possibility that observed differences arise from unequal resource allocation rather than the supervision method itself.
minor comments (1)
  1. [Abstract] The paper should clarify the exact selection procedure for the representative subset and include a table or figure comparing its statistics to the full MATH test set.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their positive evaluation, recommendation for minor revision, and constructive comments. We address each major comment below and will revise the manuscript accordingly to strengthen the presentation of our results.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the 78% solve rate and the claim of significant outperformance over outcome supervision are both measured on a 'representative subset' of the MATH test set. No statistical confirmation is provided (e.g., Kolmogorov-Smirnov test or chi-squared comparison of difficulty levels 1-5 and category distributions) that the subset matches the full test distribution. This is load-bearing for the headline conclusion that process supervision is superior for the MATH dataset, as any post-hoc selection or skew could inflate both the absolute figure and the gap versus outcome supervision.

    Authors: We appreciate the referee's emphasis on rigorous validation of the subset. The subset was constructed by selecting problems to match the overall distribution of difficulty levels (1-5) and categories from the full MATH test set, based on the dataset's provided metadata. While we did not include formal statistical tests in the original submission, we agree this would bolster the claim. In the revised manuscript, we will add a dedicated paragraph and table in the Experimental Results section (or a new appendix) that compares the distributions using chi-squared tests for categories and difficulty levels, along with summary statistics. This will confirm the subset's representativeness and support the headline findings without altering the reported numbers. revision: yes

  2. Referee: [Experimental results] Experimental results section: the comparison between process and outcome supervision should explicitly document controls for model size and total training compute to rule out the possibility that observed differences arise from unequal resource allocation rather than the supervision method itself.

    Authors: We agree that explicit documentation of these controls is essential for a fair comparison. All models in the main experiments used identical base architectures and sizes (the 7B LLaMA model), the same training hyperparameters, batch sizes, and number of training steps, resulting in equivalent total compute for process-supervised and outcome-supervised variants. The only difference was the form of the supervision signal and associated reward model training. These details appear in the 'Models and Training' and 'Experimental Setup' sections, but we will revise the Experimental Results section to include a concise, dedicated statement (and possibly a small table) explicitly confirming equal model size and compute allocation to eliminate any ambiguity. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical results grounded in independent human labels

full rationale

The paper reports an empirical comparison of process versus outcome supervision on the MATH dataset, with the central 78% solve-rate claim obtained by direct evaluation of a trained model on a held-out representative subset using newly collected human step-level labels (PRM800K). No mathematical derivation chain exists that reduces any result to its inputs by construction. There are no self-definitional equations, no fitted parameters renamed as predictions, no load-bearing self-citations, and no uniqueness theorems imported from prior author work. The evaluation relies on external benchmarks (MATH) and independent human annotations rather than tautological reuse of training signals.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim depends on the reliability of human step-level annotations and the assumption that the selected MATH subset reflects the full test distribution; no new entities or free parameters are introduced beyond standard training hyperparameters.

axioms (1)
  • domain assumption Human step-level feedback is accurate and unbiased
    The process reward model is trained directly on these labels; any systematic bias would propagate to the reported performance gap.

pith-pipeline@v0.9.0 · 5504 in / 1160 out tokens · 30812 ms · 2026-05-10T15:30:13.237539+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. DualKV: Shared-Prompt Flash Attention for Efficient RL Training with Large Rollouts and Long Contexts

    cs.LG 2026-05 conditional novelty 8.0

    DualKV is a new FlashAttention variant that shares prompt KV across multiple rollouts in RL training, delivering 1.63-3.82x speedups on 8B-30B models while remaining mathematically identical to standard attention.

  2. The Coupling Tax: How Shared Token Budgets Undermine Visible Chain-of-Thought Under Fixed Output Limits

    cs.LG 2026-05 unverdicted novelty 8.0

    Shared token budgets between visible chain-of-thought and answers create a coupling tax that makes non-thinking competitive on math benchmarks, with a truncation decomposition predicting the crossover and split budget...

  3. MedPRMBench: A Fine-grained Benchmark for Process Reward Models in Medical Reasoning

    cs.CL 2026-04 unverdicted novelty 8.0

    MedPRMBench is the first fine-grained benchmark for process reward models in medical reasoning, featuring 6500 questions, 13000 chains, 113910 step labels, and a baseline that improves downstream QA accuracy by 3.2-6....

  4. Co-Designing Quantum Codes with Transversal Diagonal Gates via Multi-Agent Systems

    quant-ph 2025-10 accept novelty 8.0 full

    A Lean-verified multi-agent system produces a catalogue of 14,116 quantum codes with transversal diagonal gates for small parameters, extracts infinite families, and resolves specific distance-3 cases with constructio...

  5. ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation

    cs.CR 2025-07 unverdicted novelty 8.0

    ExCyTIn-Bench is the first benchmark of 7542 questions from Microsoft Sentinel threat investigation graphs, where the best LLM agent achieves a reward of 0.606.

  6. EDGE-OPD: Internalizing Privileged Context with Evidence Guided On-Policy Distillation

    cs.AI 2026-05 unverdicted novelty 7.0

    EDGE-OPD adds guided rollouts and evidence masking to on-policy self-distillation, enabling successful learning of target identities where standard OPSD and RLSD fail.

  7. GS-QA: A Benchmark for Geospatial Question Answering

    cs.DB 2026-05 unverdicted novelty 7.0

    GS-QA is a new benchmark of 2,800 QA pairs on 28 templates using OSM and Wikipedia data to evaluate LLMs on spatial predicates, multi-source reasoning, and diverse answer types including distances and counts.

  8. Diagnosis Is Not Prescription: Linguistic Co-Adaptation Explains Patching Hazards in LLM Pipelines

    cs.CL 2026-05 unverdicted novelty 7.0

    Causal diagnosis identifies the routing module as bottleneck in LLM agents but prompt patching there degrades results due to linguistic co-adaptation, while upstream patching improves them.

  9. The Readout Shortcut: Positional Number Copying Dominates Arithmetic CoT Readout in Small Language Models

    cs.LG 2026-05 unverdicted novelty 7.0

    In 1-3B instruction-tuned LMs on GSM8K, arithmetic CoT readout is dominated by positional copying of the trailing number before the answer delimiter, accounting for 54-92 percentage points of accuracy.

  10. LinAlg-Bench: A Forensic Benchmark Revealing Structural Failure Modes in LLM Mathematical Reasoning

    cs.AI 2026-05 unverdicted novelty 7.0

    LinAlg-Bench shows LLMs switch from execution errors to computational abandonment and structured fabrication at 4x4 matrix scale, indicating a working memory limit rather than knowledge gaps.

  11. Dynamic Chunking for Diffusion Language Models

    cs.CL 2026-05 unverdicted novelty 7.0

    DCDM replaces positional blocks with learnable semantic chunks via differentiable Chunking Attention, yielding consistent gains over block and unstructured diffusion baselines up to 1.5B parameters.

  12. Learning from Language Feedback via Variational Policy Distillation

    cs.LG 2026-05 unverdicted novelty 7.0

    VPD frames language feedback learning as variational EM so the teacher policy refines itself via trust-region updates on outcomes while the student learns dense token distributions on its own rollouts, outperforming f...

  13. Distributionally Robust Multi-Task Reinforcement Learning via Adaptive Task Sampling

    cs.LG 2026-05 unverdicted novelty 7.0

    DRATS derives a minimax objective from a feasibility formulation of MTRL to adaptively sample tasks with the largest return gaps, leading to better worst-task performance on MetaWorld benchmarks.

  14. Where Does Reasoning Break? Step-Level Hallucination Detection via Hidden-State Transport Geometry

    cs.CL 2026-05 unverdicted novelty 7.0

    Hallucination is detected as a transport-cost excursion in hidden-state trajectories, localized via contrastive PCA in a teacher model and distilled to a BiLSTM student.

  15. Multi-Objective and Mixed-Reward Reinforcement Learning via Reward-Decorrelated Policy Optimization

    cs.LG 2026-05 unverdicted novelty 7.0

    RDPO applies magnitude-aware quantile normalization and Mahalanobis whitening to decorrelate heterogeneous rewards in multi-objective RL, improving instruction following and writing quality on LongCat-Flash post-train...

  16. Test-Time Hinting for Black-Box Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 7.0

    Test-Time Hinting trains a hint generator to prepend contextual guidance to VLM prompts, improving accuracy on natural-image VQA benchmarks with generalization to unseen tasks and models.

  17. Reward-Weighted On-Policy Distillation with an Open Property-Equivalence Verifier for NL-to-SVA Generation

    cs.AR 2026-05 unverdicted novelty 7.0

    Reward-Weighted On-Policy Distillation with an open property-equivalence verifier produces a 7B model that surpasses prior SOTA on NL-to-SVA generation across pass@1/5/10 metrics.

  18. Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why

    cs.LG 2026-05 unverdicted novelty 7.0

    Distillation signals align better with ideal updates on incorrect student rollouts than correct ones, with optimal teacher context depending on student capacity and task.

  19. Likelihood scoring for continuations of mathematical text: a self-supervised benchmark with tests for shortcut vulnerabilities

    cs.LG 2026-05 unverdicted novelty 7.0

    Presents a likelihood-based benchmark for equation-suffix prediction in technical papers with controls to detect shortcut vulnerabilities in model forecasts.

  20. Equilibrium Residuals Expose Three Regimes of Matrix-Game Strategic Reasoning in Language Models

    cs.LG 2026-05 unverdicted novelty 7.0

    LLMs rely on semantic cues for matrix-game equilibria but can acquire approximate computation via residual training on small instances, with a Lipschitz proof enabling transfer to larger anonymous games.

  21. Beyond Accuracy: Evaluating Strategy Diversity in LLM Mathematical Reasoning

    cs.AI 2026-05 unverdicted novelty 7.0

    Frontier LLMs achieve 95-100% accuracy on AMC/AIME problems but recover far fewer distinct valid strategies than human references, while collectively generating 50 novel strategies.

  22. AgentPSO: Evolving Agent Reasoning Skill via Multi-agent Particle Swarm Optimization

    cs.AI 2026-05 unverdicted novelty 7.0

    AgentPSO evolves reusable multi-agent reasoning skills via PSO-inspired natural-language updates, outperforming static agents and test-time multi-agent baselines on math and general reasoning tasks with cross-benchmar...

  23. A Single Layer to Explain Them All:Understanding Massive Activations in Large Language Models

    cs.CL 2026-05 conditional novelty 7.0

    Massive activations first appear in a single ME Layer due to RMSNorm and FFN, remain invariant thereafter, and a simple softening method raises LLM performance while reducing attention sinks.

  24. A Single Layer to Explain Them All:Understanding Massive Activations in Large Language Models

    cs.CL 2026-05 unverdicted novelty 7.0

    Massive activations originate in a specific ME Layer across LLM families; reducing their token rigidity via a targeted method boosts performance and mitigates attention sinks.

  25. KL for a KL: On-Policy Distillation with Control Variate Baseline

    cs.LG 2026-05 unverdicted novelty 7.0

    vOPD stabilizes on-policy distillation gradients by subtracting a closed-form per-token negative reverse KL baseline as a detached control variate, preserving unbiasedness while lowering variance and matching expensiv...

  26. Mathematical Reasoning via Intervention-Based Time-Series Causal Discovery Using LLMs as Concept Mastery Simulators

    cs.LG 2026-05 unverdicted novelty 7.0

    CIKA uses LLM-based interventions to probe causal effects of concepts on math reasoning success, achieving competitive results on benchmarks like Omni-MATH and GSM8K with a frozen 7B model.

  27. LaTER: Efficient Test-Time Reasoning via Latent Exploration and Explicit Verification

    cs.CL 2026-05 unverdicted novelty 7.0

    LaTER reduces LLM token usage 16-33% on reasoning benchmarks by exploring in latent space then switching to explicit CoT verification, with gains like 70% to 73.3% on AIME 2025 in the training-free version.

  28. Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Control

    cs.LG 2026-05 unverdicted novelty 7.0

    Star Elastic trains N nested submodels in a single post-training job on a parent reasoning LLM, supporting elastic budget control that matches or exceeds independent baselines while cutting training compute by up to 360x.

  29. Maximizing Rollout Informativeness under a Fixed Budget: A Submodular View of Tree Search for Tool-Use Agentic Reinforcement Learning

    stat.ML 2026-05 unverdicted novelty 7.0

    InfoTree casts intermediate state selection in tree search as monotone submodular maximization under fixed rollout budgets, yielding closed-form UUCB terms and lifting mixed-outcome ratios while outperforming flat GRP...

  30. Correct Is Not Enough: Training Reasoning Planners with Executor-Grounded Rewards

    cs.AI 2026-05 unverdicted novelty 7.0

    TraceLift trains reasoning planners with executor-grounded rewards that multiply a rubric-based reasoning quality score by measured uplift on a frozen executor, outperforming execution-only training on math and code b...

  31. Decoding-Time Debiasing via Process Reward Models: From Controlled Fill-in to Open-Ended Generation

    cs.CL 2026-05 unverdicted novelty 7.0

    Decoding-time use of process reward models for bias mitigation raises fairness scores by up to 0.40 on a bilingual benchmark while preserving fluency across four LLMs and extends to open-ended generation with low overhead.

  32. NeuroState-Bench: A Human-Calibrated Benchmark for Commitment Integrity in LLM Agent Profiles

    cs.AI 2026-05 accept novelty 7.0

    NeuroState-Bench is a human-calibrated benchmark with 144 tasks and 306 side-query probes showing that commitment integrity in LLM agent profiles diverges from task success, with 31 of 32 profiles changing rank under ...

  33. RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences

    cs.CL 2026-05 unverdicted novelty 7.0

    RMGAP benchmark shows state-of-the-art reward models reach at most 49.27% Best-of-N accuracy when forced to select responses matching diverse preferences.

  34. Selector-Guided Autonomous Curriculum for One-Shot Reinforcement Learning from Verifiable Rewards

    cs.LG 2026-05 unverdicted novelty 7.0

    SGAC replaces reward-variance heuristics with a multi-feature learnable selector emphasizing output entropy, yielding 68% accuracy on Hendrycks MATH with Qwen2.5-Math-1.5B versus 64-66% baselines.

  35. BoostLoRA: Growing Effective Rank by Boosting Adapters

    cs.LG 2026-04 unverdicted novelty 7.0

    BoostLoRA grows effective adapter rank linearly via iterative boosting on hard examples with orthogonal low-rank updates, outperforming both single-shot ultra-low-rank adapters and full fine-tuning on math and code ta...

  36. Training Computer Use Agents to Assess the Usability of Graphical User Interfaces

    cs.CL 2026-04 unverdicted novelty 7.0

    uxCUA is a trained computer use agent that assesses GUI usability more accurately than larger models by learning to prioritize and execute important user interactions on labeled interface datasets.

  37. Co-Evolving LLM Decision and Skill Bank Agents for Long-Horizon Tasks

    cs.AI 2026-04 unverdicted novelty 7.0

    COSPLAY co-evolves an LLM decision agent with a skill bank agent to improve long-horizon game performance, reporting over 25.1% average reward gains versus frontier LLM baselines on single-player benchmarks.

  38. R2IF: Aligning Reasoning with Decisions via Composite Rewards for Interpretable LLM Function Calling

    cs.LG 2026-04 unverdicted novelty 7.0

    R2IF improves LLM function-calling accuracy by up to 34.62% on BFCL using a composite reward system with CER and SMV components optimized via GRPO, while increasing interpretability through positive CoT effectiveness.

  39. Fine-Tuning Small Reasoning Models for Quantum Field Theory

    cs.LG 2026-04 unverdicted novelty 7.0

    Small 7B reasoning models were fine-tuned on synthetic and curated QFT problems using RL and SFT, yielding performance gains, error analysis, and public release of data and traces.

  40. Navigating the Conceptual Multiverse

    cs.HC 2026-04 unverdicted novelty 7.0

    The conceptual multiverse system with a verification framework for decision structures helps users in philosophy, AI alignment, and poetry build clearer working maps of open-ended problems by making implicit LLM choic...

  41. Self-Consistency from Only Two Samples: CoT-PoT Ensembling for Efficient LLM Reasoning

    cs.CL 2026-04 unverdicted novelty 7.0

    CoT-PoT ensembling achieves self-consistency accuracy in LLMs with only two samples for 78.6% of tasks, reducing computation by 9.3x compared to standard methods.

  42. IG-Search: Step-Level Information Gain Rewards for Search-Augmented Reasoning

    cs.AI 2026-04 unverdicted novelty 7.0

    IG-Search computes step-level information gain rewards from policy probabilities to improve credit assignment in RL training for search-augmented QA, yielding 1.6-point gains over trajectory-level baselines on multi-h...

  43. AI Achieves a Perfect LSAT Score

    cs.AI 2026-04 unverdicted novelty 7.0

    Language models achieve a perfect LSAT score, with experiments showing that internal thinking phases and a fine-tuned process reward model are key to high performance on logical reasoning questions.

  44. Backdoors in RLVR: Jailbreak Backdoors in LLMs From Verifiable Reward

    cs.CR 2026-04 accept novelty 7.0

    RLVR can be backdoored with under 2% poisoned data using an asymmetric reward trigger, implanting jailbreaks that cut safety performance by 73% on average without harming benign tasks.

  45. Structural Evaluation Metrics for SVG Generation via Leave-One-Out Analysis

    cs.LG 2026-04 unverdicted novelty 7.0

    Element-level leave-one-out analysis yields per-element quality scores and four structural metrics (purity, coverage, compactness, locality) that quantify SVG modularity and enable artifact detection.

  46. Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models

    cs.CL 2026-04 unverdicted novelty 7.0

    OPD for LLMs suffers length inflation and repetition collapse; StableOPD uses reference divergence and rollout mixing to prevent it and improve math reasoning performance by 7.2% on average.

  47. QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatch

    cs.LG 2026-04 unverdicted novelty 7.0

    QaRL aligns quantized rollouts with training in LLM RL and uses TBPO with dual clipping to stabilize optimization, delivering +5.5 improvement over standard quantized-rollout baselines on Qwen3-30B math problems while...

  48. Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning

    cs.LG 2026-04 unverdicted novelty 7.0

    This survey introduces the Generate-Filter-Control-Replay (GFCR) taxonomy to structure rollout pipelines for RL-based post-training of reasoning LLMs.

  49. Sampling for Quality: Training-Free Reward-Guided LLM Decoding via Sequential Monte Carlo

    cs.LG 2026-04 unverdicted novelty 7.0

    Sequential Monte Carlo sampling from a reward-augmented sequence distribution improves LLM performance on HumanEval by up to 54.9% and MATH500 by up to 8.8%, outperforming standard sampling and GRPO.

  50. WMF-AM: Probing LLM Working Memory via Depth-Parameterized Cumulative State Tracking

    cs.AI 2026-03 unverdicted novelty 7.0

    WMF-AM is a depth-parameterized benchmark that measures LLMs' cumulative state tracking ability without scratchpads, validated on 28 models across arithmetic and non-arithmetic tasks with ablations confirming the construct.

  51. Drift and selection in LLM text ecosystems

    cs.CL 2026-03 unverdicted novelty 7.0

    Recursive LLM text generation drives public corpora toward shallow equilibria via drift unless normative selection for quality sustains deeper structure with a bounded divergence.

  52. Pramana: Fine-Tuning Large Language Models for Epistemic Reasoning through Navya-Nyaya

    cs.AI 2026-02 conditional novelty 7.0

    Fine-tuning LLMs on Navya-Nyaya's six-phase reasoning structure yields 100% semantic correctness on held-out logical problems despite only 40% strict format adherence.

  53. ModeX: Evaluator-Free Best-of-N Selection for Open-Ended Generation

    cs.CL 2026-01 unverdicted novelty 7.0

    ModeX selects the modal semantic output from multiple LLM generations via a similarity graph and recursive spectral clustering without needing reward models or evaluators.

  54. Schoenfeld's Anatomy of Mathematical Reasoning by Language Models

    cs.CL 2025-12 unverdicted novelty 7.0

    ThinkARM abstracts LLM reasoning traces into Schoenfeld episodes and shows that exploration steps correlate with correctness while efficiency methods selectively suppress evaluative feedback.

  55. d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Models

    cs.CL 2025-12 unverdicted novelty 7.0

    d-TreeRPO uses tree rollouts for fine-grained verifiable rewards and time-scheduled self-distillation to reduce probability estimation gaps in diffusion LLMs, delivering substantial gains on Sudoku, Countdown, GSM8K, ...

  56. Reinforcement Learning for Reasoning in Large Language Models with One Training Example

    cs.LG 2025-04 accept novelty 7.0

    One training example via RLVR boosts LLM math reasoning from 17.6% to 35.7% average across six benchmarks.

  57. PRIMETIME : Limits of LLMs in Temporal Primitives

    cs.NE 2025-04 unverdicted novelty 7.0

    PRIMETIME generator reveals that LLM datetime parsing and arithmetic primitives are individually unreliable but fully learnable via fine-tuning, enabling frontier-level accuracy on event planning with small LoRA models.

  58. MIST: A Co-Design Framework for Heterogeneous, Multi-Stage LLM Inference

    cs.AR 2025-04 unverdicted novelty 7.0

    MIST is a new simulator for heterogeneous multi-stage LLM inference that combines hardware traces with analytical models to explore configuration trade-offs in hybrid CPU-accelerator systems.

  59. Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation

    cs.AI 2025-03 conditional novelty 7.0

    Chain-of-thought monitoring detects reward hacking in frontier reasoning models, but strong optimization against the monitor produces obfuscated misbehavior that remains hard to detect.

  60. OPPO: Bayesian Value Recursion for Token-Level Credit Assignment in LLM Reasoning

    cs.LG 2026-05 unverdicted novelty 6.0

    OPPO computes token-level advantages via Bayesian recursion on oracle signals, recovering distillation methods as a special case and improving over GRPO on math and code benchmarks.

Reference graph

Works this paper leans on

22 extracted references · 22 canonical work pages · cited by 178 Pith papers · 15 internal anchors

  1. [1]

    A General Language Assistant as a Laboratory for Alignment

    A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861 ,

    work page internal anchor Pith review arXiv

  2. [2]

    Sparks of Artificial General Intelligence: Early experiments with GPT-4

    S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 ,

    work page internal anchor Pith review arXiv

  3. [3]

    Training Verifiers to Solve Math Word Problems

    K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 ,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    16 Solving math word problems with process- and outcome-based feedback A

    A. Creswell, M. Shanahan, and I. Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09712,

    work page arXiv

  5. [5]

    Reinforcement Learning with a Corrupted Reward Channel

    T. Everitt, V. Krakovna, L. Orseau, M. Hutter, and S. Legg. Reinforcement learning with a corrupted reward channel. arXiv preprint arXiv:1705.08417 ,

    work page Pith review arXiv

  6. [6]

    L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overopti- mization. arXiv preprint arXiv:2210.10760 ,

    work page internal anchor Pith review arXiv

  7. [7]

    Measuring Mathematical Problem Solving With the MATH Dataset

    D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 ,

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    Large Language Models are Zero-Shot Reasoners

    T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916 ,

    work page internal anchor Pith review arXiv

  9. [9]

    Solving Quantitative Reasoning Problems with Language Models

    A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ra- masesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14858,

    work page internal anchor Pith review arXiv

  10. [10]

    Y. Li, Z. Lin, S. Zhang, Q. Fu, B. Chen, J.-G. Lou, and W. Chen. On the advance of making language models better reasoners. arXiv preprint arXiv:2206.02336,

    work page arXiv

  11. [11]

    On faithfulness and factuality in abstractive summarization

    J. Maynez, S. Narayan, B. Bohnet, and R. McDonald. On faithfulness and factuality in abstractive summarization. arXiv preprint arXiv:2005.00661 ,

    work page arXiv 2005

  12. [12]

    WebGPT: Browser-assisted question-answering with human feedback

    R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, et al. Webgpt: Browser-assisted question- answering with human feedback. arXiv preprint arXiv:2112.09332 ,

    work page internal anchor Pith review arXiv

  13. [13]

    14 M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, D. Dohan, A. Lewkowycz, M. Bosma, D. Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114,

    work page internal anchor Pith review arXiv

  14. [14]

    GPT-4 Technical Report

    OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    Training language models to follow instructions with human feedback

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 ,

    work page internal anchor Pith review arXiv

  16. [16]

    J. Shen, Y. Yin, L. Li, L. Shang, X. Jiang, M. Zhang, and Q. Liu. Generate & rank: A multi-task framework for math word problems. arXiv preprint arXiv:2109.03034,

    work page arXiv

  17. [17]

    Stuhlm¨ uller and J

    A. Stuhlm¨ uller and J. Byun. Supervise process, not outcomes.https://ought. org/updates/2022-04-06-process ,

    work page 2022

  18. [18]

    Solving math word problems with process- and outcome-based feedback

    J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275 ,

    work page internal anchor Pith review Pith/arXiv arXiv

  19. [19]

    X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171,

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [20]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903,

    work page internal anchor Pith review Pith/arXiv arXiv

  21. [21]

    D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human pref- erences. arXiv preprint arXiv:1909.08593 ,

    work page internal anchor Pith review arXiv 1909

  22. [22]

    15 A MathMix Similar to Lewkowycz et al. (2022) we construct a large-scale dataset of high- quality math-relevant tokens for use in a lightweight pretraining stage, before finetuning on comparably smaller datasets like MATH and PRM800K. This dataset, which we call MathMix, has two main differences compared to the one used to train Minerva. First, it is sm...

    work page 2022