Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Rollout Roulette: A Probabilistic Inference Approach to Inference-Time Scaling of LLMs using Particle-Based Monte Carlo Methods

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that replacing deterministic top-k search with softmax-sampling particle filtering gives inference-time scaling a 4–16x better scaling rate, letting small open models match much larger closed models on math reasoning.

desk verdict Particle-resampled beam search is a genuinely promising scaling trick, but the paper's SMC theory does not match its own Algorithm 1; worth reviewing as an empirical method. read the letter →

arxiv 2502.01618 v5 pith:Z6ELNVS6 submitted 2025-02-03 cs.LG cs.AI

Isha Puri , Shivchander Sudalairaj , Guangxuan Xu , Kai Xu , Akash Srivastava This is my paper · ORCID
classification cs.LGcs.AI
keywords inference-timescalingparticlefilteringsequentialMonteCarloprocessrewardmodelslargelanguagemathematicalreasoningbeamsearchstate-space
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that inference-time scaling of large language models should be treated as probabilistic inference rather than search, and that carrying this out with particle filtering makes test-time compute far more sample-efficient. Where beam search and dynamic variable-time search greedily prune low-scoring partial solutions and can never recover them, the proposed method keeps a weighted population of candidate trajectories and resamples them stochastically, so promising paths are favored but not allowed to dominate. On math and broader reasoning benchmarks the authors report a 4–16x better scaling rate than deterministic search counterparts, and show that Qwen2.5-Math-1.5B-Instruct surpasses GPT-4o accuracy with only 4 rollouts while Qwen2.5-Math-7B-Instruct reaches o1-level accuracy with 32. If this holds, small open models can close much of the gap with proprietary frontier models purely by spending inference compute more intelligently.

What carries the argument

The load-bearing machinery is particle filtering (sequential Monte Carlo) applied to a state-space model whose transition kernel is the LLM $p_M(x_t \mid c, x_{<t})$ and whose observation kernel is a Bernoulli emission $B(o_t; \hat{r}(c, x_{\le t}))$ scored by a process reward model. At each step the algorithm extends every particle with a sampled next step, assigns weight $w^{(i)}_t \propto w^{(i)}_{t-1} \hat{r}(c, x^{(i)}_{\le t})$, and resamples the population by drawing indices from a softmax over these weights, which is what keeps exploration alive: low-scoring but potentially correct trajectories survive with nonzero probability. This weighted population is meant to approximate the posterior $\hat{p}_M(x_{1:T} \mid c, o_{1:T}=1)$ over accepted trajectories, and the paper uses that approximation to justify unbiased estimates of expected accuracy and to add MCMC-style extensions (Particle Gibbs, parallel tempering) that allocate compute across iterations and parallel chains.

What would settle it

Run Algorithm 1 on a small toy state-space model with enumerable states and compare the weighted-particle estimate of any expectation, such as expected accuracy, against the exact value computed from $\hat{p}_M(x_{1:T} \mid c, o_{1:T}=1)$; if the estimate stays biased as the number of particles grows, the unbiasedness theorem fails and the method is a stochastic search heuristic. A cheaper check is to inspect whether the implementation multiplies incremental likelihoods $w^{(i)}_t \propto w^{(i)}_{t-1} \hat{r}(c, x^{(i)}_{\le t})$ or instead replaces the weight with the raw cumulative PRM score, since the pseudo-code in Algorithm 1 does the latter and that breaks the standard SMC recursion.

Watch

Extended reading notes

Core claim

The paper's central claim is that re-framing inference-time scaling as posterior inference in a state-space model, with the LLM as the transition model and a process reward model as the approximate emission model, turns a brittle optimization problem into a well-posed sampling problem. The proposed particle filtering algorithm maintains N candidate trajectories, extends each by sampling from the LLM, scores partial trajectories with the PRM, and resamples through a softmax over those scores; the resulting population approximates the posterior over fully accepted sequences $\hat{p}_M(x_{1:T} \mid c, o_{1:T}=1)$, and sampling from this typical set instead of chasing the mode avoids the early-pruning failure of beam search. On MATH500 and AIME 2024 the authors find particle filtering consistently outperforms self-consistency, best-of-N, weighted best-of-N, beam search, and DVTS at equal compute budgets, and that its accuracy grows 4–16x faster with budget than the best search baseline. The headline results are that Qwen2.5-Math-1.5B-Instruct surpasses GPT-4o at a budget of 4 generations, and Qwen2.5-Math-7B-Instruct reaches 87.7% on MATH500 and 10/30 on AIME 2024 at a budget of 32, matching or exceeding o1-preview.

Load-bearing premise

The entire probabilistic-interpretation claim rests on the softmax resampling step in Algorithm 1 being a valid sequential Monte Carlo weight update, so the particle population really converges to the posterior over accepted trajectories; the paper's Appendix C proof is a two-sentence assertion that never derives this from the algorithm's actual weight update.

Editorial extensions

If this is right

  • On Llama-3.2-1B-Instruct, a budget of 8 particles matches what DVTS achieves with 32 generations, and on Llama-3.1-8B-Instruct the gap is 8 versus 128, so the same accuracy can be bought with 4–16x less test-time compute.
  • Small open math models become competitive with proprietary frontier models: Qwen2.5-Math-1.5B-Instruct surpasses GPT-4o on MATH500 with 4 rollouts, and Qwen2.5-Math-7B-Instruct reaches 87.7% on MATH500 and 10/30 on AIME 2024 with 32 rollouts, matching or exceeding o1-preview.
  • The probabilistic framing makes inference-time scaling a proper inference problem, so standard SMC theory becomes applicable, including unbiased estimators and MCMC kernels; the paper uses this to build Particle Gibbs and parallel-tempering variants that allocate compute across iterations and chains.
  • The method transfers beyond math: using a math-trained PRM as the reward model, particle filtering improves accuracy over all baselines on FinanceBench (70.33) and NumGLUE Task 2 Chemistry (84.22).
  • Because high-scoring candidates are favored but never allowed to dominate, the method avoids irreversible early pruning, meaning compute is not wasted on trajectories that the reward model initially underrates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves the softmax temperature in the resampling step fixed; treating it as a tunable exploration-exploitation knob, for example annealing it over the course of generation, is a natural extension that could improve scaling further on multi-modal tasks.
  • A corrected weight update that accumulates incremental likelihood products, $w_t \propto w_{t-1} \cdot \hat{r}(x_{1:t})$, instead of raw PRM scores would align the algorithm with standard SMC and may make the unbiasedness proof go through cleanly; this is a testable modification of Algorithm 1.
  • If the early-pruning diagnosis is right, the advantage of particle filtering over beam search should grow as the reward model becomes noisier or as tasks have more genuinely distinct solution paths; comparing the two methods under controlled PRM noise levels would test that prediction.
  • The framework invites more advanced SMC tooling, such as twisted SMC with a learned value function or conditional particle filters, to reduce estimator variance.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes to treat inference-time scaling of LLMs as probabilistic inference in a state-space model, using particle filtering (PF) to sample from a posterior over reasoning trajectories defined by an LLM transition model and a PRM-based emission model. The authors present Algorithm 1 (PF), extensions to Particle Gibbs and parallel tempering, and an extensive empirical evaluation on MATH500, AIME 2024, FinanceBench, and NumGLUE. They claim a 4-16x better scaling rate than deterministic search baselines, and that small open models such as Qwen2.5-Math-1.5B-Instruct can surpass GPT-4o with only 4 rollouts, while Qwen2.5-Math-7B-Instruct reaches o1-level accuracy with 32 rollouts. A theorem is stated asserting unbiased estimation of expected accuracy from the particle weights, with a proof in Appendix C.

Significance. If the empirical results hold, the paper would make a valuable practical contribution: inference-time scaling with stochastic resampling instead of deterministic beam-style pruning is an appealing and simple idea, and the breadth of experiments across model families and non-math benchmarks is a strength. The claimed 4-16x compute savings would be practically important, and the connection to sequential Monte Carlo could open a useful design space. However, the manuscript's theoretical backbone is central to its framing, and that backbone is not currently sound: the stated algorithm does not implement the weight recursion that the theorem relies on, and the proof is only a citation-level assertion. The empirical claims also lack error bars and are partly based on test-set hyperparameter selection. The paper deserves a major revision to either fix the theoretical mismatch, or to reframe the contribution as a stochastic search heuristic with strong empirical results and remove the unsubstantiated unbiasedness guarantees.

major comments (4)
  1. [§3.1 / Appendix E, Algorithm 1] Algorithm 1 updates the weight vector as w = [rhat(x^{(i)}_{1:t})] and resamples with probabilities proportional to exp(w_i) at every step, whereas §3.1 specifies the multiplicative weight recursion w^{(i)}_t ∝ w^{(i)}_{t-1} · rhat(c, x^{(i)}_{<t}). These updates are inconsistent. If rhat(x_{1:t}) denotes the cumulative product of per-step rewards, resampling with that cumulative product at every step double-counts earlier rewards, yielding an effective target proportional to p_M(x_{1:T}|c) ∏_{s=1}^T r_s^{T-s+1} rather than Eq. (2); if rhat(x_{1:t}) denotes only the current step's reward, the cumulative likelihood in Eq. (2) is never formed. The proof of Theorem 1/Theorem 2 in Appendix C is a two-sentence appeal to 'the unbiasedness property of particle filtering' and does not establish that this property holds for the algorithm actually implemented. Because the unbiasedness theorem is the stated basis for calling the method probabilistic inference rather than stochastic beam search, this issue is load-bearing. The authors should either modify Algorithm 1 to implement a valid SMC weight recursion (e.g., update via incremental likelihood factors and resample only when the effective sample size is low), or replace the theorem and the 'principled probabilistic inference' claims with an honest description of the method as a stochastic search heuristic.
  2. [§4.5, §4.2] The main results on MATH500 (Table 1, Figure 4) are obtained after selecting the process reward model, temperature, reward aggregation strategy, and particle budget using 100-question subsets of MATH500 (Section 4.5). If those 100 questions are part of the 500 problems used for the reported accuracy, the headline numbers reflect test-set selection. The paper should state explicitly whether the validation questions were excluded from the final evaluation and, ideally, choose hyperparameters on a separate held-out split or report sensitivity across a range of hyperparameter choices.
  3. [§4.4, Table 1] The central scaling claim ('4-16x better scaling rate') and the model-comparison claims (e.g., Qwen2.5-Math-7B PF 87.7% vs o1-preview 87.0% on MATH500; 6/30 vs 4/30 on AIME 2024) are reported without error bars, confidence intervals, or any statistical significance assessment, as acknowledged in the NeurIPS checklist. On AIME 2024 with 30 problems, the 2-problem difference between PF and beam search is within binomial noise; on MATH500 a 0.7-point difference is likewise not established as significant. The authors should provide per-seed or bootstrap variability for the main comparisons, or explicitly temper the claims (e.g., 'matches' instead of 'surpasses' where differences are within noise).
  4. [§3.1 / Figure 4] The compute budget is not defined consistently. Algorithm 1 generates one new step per particle at each time step, so over a full rollout the number of policy-model calls is N × T, not N, yet Figure 4 and the text treat the particle count N as the 'budget (# of model generations)' and compare against beam search/DVTS under the same label. If the baselines count one generation as one complete solution, while PF counts one particle as one complete solution, the total number of decoding calls is larger for PF at the same nominal budget. The paper should define 'generation' precisely and ensure the budget axis measures the same resource for all methods, including the cost of the PRM and ORM.
minor comments (5)
  1. [§3.1 / Appendix C] The theorem is labeled 'Theorem 1' in the main text but 'Theorem 2' in Appendix C, and the paragraph before it refers to 'Theorem 2 (proof in Appendix C)'. The numbering and cross-references should be made consistent.
  2. [§2, Eq. (1)] Equation (1) writes the emission as p(ot | c, x_{<t}) but defines it as B(ot; r(c, xt)); the observation should depend on xt, not x_{<t}. Also, the notation x_{<t−1} in Section 2 and Eq. (1) is confusing; it should be x_{<t} or x_{1:t-1}.
  3. [Appendix E, Algorithms 2 and 3] In Algorithms 2 and 3, the transition step is written as 'x^{(i)}_{t+1} ~ pM(· | c, x^{(i)}_{t+1})' which conditions on the variable being generated; this should presumably be pM(· | c, x^{(i)}_{1:t}).
  4. [§5, Related Work] The sentence 'While formulating it as a MAB problem allows it to use a scheduling on the softmax temperature and to derive regret bounds, we no longer have the same unbiasedness from the particle filtering / SMC formulation' is confusing and appears to compare the wrong pair of methods; it should be clarified or rephrased.
  5. [§4.3 / Table 2] The caption for Table 2 is missing; the table title currently reads 'Method FinanceBench NumGLUE Task 2 (Chemistry)' as a heading. The caption and formatting should be fixed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity. The paper's headline numbers are empirical comparisons against external baselines (beam search, DVTS, BoN/WBoN) under a fixed compute budget using an off-the-shelf PRM/ORM, and no fitted parameter is later renamed as a prediction or forced by construction.

full rationale

The derivation chain is: define an SSM posterior over trajectories with a PRM likelihood (Eq. 2), approximate it with Algorithm 1's particle population, then evaluate accuracy on external benchmarks. None of these steps reduces to its own output: the PRM (Qwen2.5-Math-PRM-7B) and ORM are external models, the baselines are standard published methods, and the 4-16x scaling claim is read directly off accuracy-vs-budget curves rather than predicted from a fitted parameter. The theoretical guarantee (Theorem 1/2) is not established - Appendix C is a two-sentence appeal to 'the unbiasedness property of particle filtering' that never shows Algorithm 1's softmax-over-cumulative-reward resampling implements the recursive weight update w_t proportional to w_{t-1} times rhat(x_{<t}) - but an unsupported or incorrect proof is a correctness and rigor problem, not circularity, because the theorem is not assumed as an input to the empirical evaluation. The closest concern is that the PRM, reward aggregation, and temperature were chosen using ablations on a 100-question subset of MATH500 and then the full MATH500 is used for headline results; this is a mild evaluation-selection leak, but it does not make any reported number equivalent by construction to an input, and the pattern is not a self-citation chain or a definitional identity. Therefore no circularity is found.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no invented physical entities, but it relies on several unproven modeling choices. The most important is that softmax PRM resampling is a valid SMC update, which is asserted rather than derived. The hyperparameter selection on a subset of the evaluation distribution is an additional burden on the central claims.

free parameters (4)
  • generation temperature = 0.8
    Selected via ablation on a 100-question subset of MATH500 (Section 4.5) and used for all main experiments, following Beeching et al.
  • process reward model = Qwen2.5-Math-PRM-7B
    Chosen because it outperformed other PRMs in an ablation on a 100-question subset of MATH500 (Section 4.5).
  • reward aggregation strategy = ambiguous: product stated as default, model aggregation recommended by ablation
    Section 3.1 defaults to product of step rewards, but Section 4.5 reports model aggregation scales best with budget; the final results do not clearly state which aggregation is used.
  • particle budget N per dataset = 32 for main math results, 8 for FinanceBench, 64 for Appendix F
    Budgets are problem-dependent and the paper notes that hyperparameters like budget require tuning across domains (Section 6).
assumptions (4)
  • domain assumption PRM scores can be interpreted as a Bernoulli emission likelihood p(ot | c, x<t).
    Section 3 defines the emission model as Bernoulli with reward r(c,x_t); PRM scores are uncalibrated and used as approximate probabilities.
  • ad hoc to paper Softmax resampling in Algorithm 1 yields a valid particle filter approximation of the posterior.
    The unbiasedness theorem (Theorem 1, Appendix C) is asserted without derivation; the weight update in the pseudo-code uses rhat(x_{1:t}) rather than incremental importance weights.
  • domain assumption The LLM's token proposal pM(x_t | c, x<t-1) covers the posterior support adequately at temperature 0.8.
    Section 4.1 sets temperature 0.8; no analysis of proposal mismatch or weight degeneracy is provided.
  • domain assumption The 100-question MATH500 subset used for ablations is representative enough to select hyperparameters that generalize to the full test set.
    Sections 4.1 and 4.5 select PRM, temperature, and aggregation on this subset, then report final results on full MATH500 without error bars.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rollout Roulette: A Probabilistic Inference Approach to Inference-Time Scaling of LLMs using Particle-Based Monte Carlo Methods." pith.science (2026). https://pith.science/paper/Z6ELNVS6

@misc{pith2026250201618,
  author       = {Pith},
  title        = {Pith review of: Rollout Roulette: A Probabilistic Inference Approach to Inference-Time Scaling of LLMs using Particle-Based Monte Carlo Methods},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z6ELNVS6}},
  note         = {Machine review of arXiv:2502.01618}
}
read the original abstract

Large language models (LLMs) have achieved significant performance gains via scaling up model sizes and/or data. However, recent evidence suggests diminishing returns from such approaches, motivating scaling the computation spent at inference time. Existing inference-time scaling methods, usually with reward models, cast the task as a search problem, which tends to be vulnerable to reward hacking as a consequence of approximation errors in reward models. In this paper, we instead cast inference-time scaling as a probabilistic inference task and leverage sampling-based techniques to explore the typical set of the state distribution of a state-space model with an approximate likelihood, rather than optimize for its mode directly. We propose a novel inference-time scaling approach by adapting particle-based Monte Carlo methods to this task. Our empirical evaluation demonstrates that our methods have a 4-16x better scaling rate over our deterministic search counterparts on various challenging mathematical reasoning tasks. Using our approach, we show that Qwen2.5-Math-1.5B-Instruct can surpass GPT-4o accuracy in only 4 rollouts, while Qwen2.5-Math-7B-Instruct scales to o1 level accuracy in only 32 rollouts. Our work not only presents an effective method to inference-time scaling, but also connects the rich literature in probabilistic inference with inference-time scaling of LLMs to develop more robust algorithms in future work. Code, videos, and further information available at https://probabilistic-inference-scaling.github.io.

Figures

Figures reproduced from arXiv: 2502.01618 by the authors.

Figure 1
Figure 1. A true example of PRM assigning a lower score to the first step of a solution that turns out [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Inference-time scaling with particle filtering: initialize [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. State-space model for inference-time scaling. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Accuracy vs. Generation Budget across models using different inference-time strategies. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 6
Figure 6. Figure 6: Comparison of PF and Particle Gibbs with different numbers of iterations, evaluated on a [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Comparison of PF and PT with different particle group sizes, evaluated on a 100-question [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: A side-by-side comparison between particle filtering and its closest search-based counterpart, [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Soft Best-of-n Sampling for Model Alignment

    cs.IT 2025-05 conditional novelty 6.0 of 10

    Soft Best-of-n sampling provably approaches the optimal tilted reward distribution at O(1/n) KL divergence and relative reward error, with sample complexity that grows exponentially in sequence length for blockwise sampling.

  2. Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlo

    cs.CL 2025-04 conditional novelty 6.0 of 10

    Sequential Monte Carlo control of LLM generation, which checks partial outputs and resamples promising candidates, lets small open models beat much larger closed models on constrained code, SQL, goal, and molecule generation.

Reference graph

Works this paper leans on

50 extracted references · 37 canonical work pages · cited by 2 Pith papers

  1. [1]

    Aimo validation aime dataset

    AI-MO. Aimo validation aime dataset. https://huggingface.co/datasets/AI-MO/ aimo-validation-aime, 2023. Accessed: 2025-01-24

    work page 2023

  2. [2]

    Particle Markov Chain Monte Carlo Methods

    Christophe Andrieu, Arnaud Doucet, and Roman Holenstein. Particle Markov Chain Monte Carlo Methods. Journal of the Royal Statistical Society Series B: Statistical Methodology , 72(3):269–342, June 2010

    work page 2010

  3. [3]

    Scaling test-time compute with open models, 2024

    Edward Beeching, Lewis Tunstall, and Sasha Rush. Scaling test-time compute with open models, 2024

    2024

  4. [4]

    Le, Christopher Ré, and Azalia Mirhoseini

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V . Le, Christopher Ré, and Azalia Mirhoseini. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling, July 2024

    work page 2024

  5. [5]

    Boltzmann exploration done right

    Nicolò Cesa-Bianchi, Claudio Gentile, Gábor Lugosi, and Gergely Neu. Boltzmann exploration done right. Advances in neural information processing systems , 30, 2017

    work page 2017

  6. [6]

    Process reinforcement through implicit rewards, 2025

    Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. Process reinforcement through implicit rewards, 2025

    work page 2025

  7. [7]

    Sequential Monte Carlo Methods for Dynamic Sys- tems: Journal of the American Statistical Association: V ol 93, No 443

    Pierre Del Moral. Sequential Monte Carlo Methods for Dynamic Sys- tems: Journal of the American Statistical Association: V ol 93, No 443. https://www.tandfonline.com/doi/abs/10.1080/01621459.1998.10473765, 1997

    work page arXiv 1998

  8. [8]

    Step-by-step reasoning for math problems via twisted sequential monte carlo, 2024

    Shengyu Feng, Xiang Kong, Shuang Ma, Aonan Zhang, Dong Yin, Chong Wang, Ruoming Pang, and Yiming Yang. Step-by-step reasoning for math problems via twisted sequential monte carlo, 2024

    work page 2024

Show all 50 references
  1. [9]

    Tenenbaum, Vikash K

    Gabriel Grand, Joshua B. Tenenbaum, Vikash K. Mansinghka, Alexander K. Lew, and Jacob Andreas. Self-steering language models, 2025

    2025

  2. [10]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ah- mad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, A...

    2024

  3. [11]

    rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking, January 2025

    Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking, January 2025

    2025

  4. [12]

    Financebench: A new benchmark for financial question answering, 2023

    Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. Financebench: A new benchmark for financial question answering, 2023

    2023

  5. [13]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020

    2020

  6. [14]

    Algorithms for multi-armed bandit problems.arXiv preprint arXiv:1402.6028, 2014

    V olodymyr Kuleshov and Doina Precup. Algorithms for multi-armed bandit problems.arXiv preprint arXiv:1402.6028, 2014

    2014 arXiv

  7. [15]

    Lew, Tan Zhi-Xuan, Gabriel Grand, and Vikash K

    Alexander K. Lew, Tan Zhi-Xuan, Gabriel Grand, and Vikash K. Mansinghka. Sequential monte carlo steering of large language models using probabilistic programs, 2023

    2023

  8. [16]

    Let’s verify step by step, 2023

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step, 2023

    2023

  9. [17]

    Let’s Verify Step by Step, May 2023

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s Verify Step by Step, May 2023

    2023

  10. [18]

    Lew, Tim Vieira, and Timothy J

    João Loula, Benjamin LeBrun, Li Du, Ben Lipkin, Clemente Pasti, Gabriel Grand, Tianyu Liu, Yahya Emara, Marjorie Freedman, Jason Eisner, Ryan Cotterell, Vikash Mansinghka, Alexander K. Lew, Tim Vieira, and Timothy J. O’Donnell. Syntactic and semantic control of large language ...

    2025

  11. [19]

    Cambridge university press, 2003

    David JC MacKay.Information theory, inference and learning algorithms. Cambridge university press, 2003

    2003

  12. [20]

    Numglue: A suite of fundamental yet challenging mathematical reasoning tasks

    Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, and Ashwin Kalyan. Numglue: A suite of fundamental yet challenging mathematical reasoning tasks. ACL, 2022

    2022

  13. [21]

    OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennet...

    2024

  14. [22]

    Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters, August 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters, August 2024

    2024

  15. [23]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024

    2024

  16. [24]

    Swendsen and Jian-Sheng Wang

    Robert H. Swendsen and Jian-Sheng Wang. Nonlinear filtering: Interacting particle resolution - ScienceDirect. https://www.sciencedirect.com/science/article/abs/pii/S0764444297847787, 1986

    1986

  17. [25]

    Bayesian Filtering and Smoothing

    Simo Särkkä. Bayesian Filtering and Smoothing. Institute of Mathematical Statistics Textbooks. Cambridge University Press, 2013

    2013

  18. [26]

    Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y . Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations, 2024

    2024

  19. [27]

    Self-consistency improves chain of thought reasoning in language models, 2023

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2023

    2023

  20. [28]

    An implementation of generative prm

    Wei Xiong, Hanning Zhang, Nan Jiang, and Tong Zhang. An implementation of generative prm. https://github.com/RLHFlow/RLHF-Reward-Modeling , 2024

    2024

  21. [29]

    Qwen2.5-math technical report: Toward mathematical expert model via self-improvement

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via se...

    2024 arXiv

  22. [30]

    Advancing llm reasoning generalists with preference trees, 2024

    Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. Advancing llm reasoning generalists with preference trees, 2024. 13

    2024

  23. [31]

    The lessons of developing process reward models in mathematical reasoning

    Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301, 2025

    2025 arXiv

  24. [32]

    The Lessons of Developing Process Reward Models in Mathematical Reasoning, January 2025

    Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The Lessons of Developing Process Reward Models in Mathematical Reasoning, January 2025

    2025

  25. [33]

    Probabilistic inference in language models via twisted sequential monte carlo, 2024

    Stephen Zhao, Rob Brekelmans, Alireza Makhzani, and Roger Grosse. Probabilistic inference in language models via twisted sequential monte carlo, 2024

    2024

  26. [34]

    Language agent tree search unifies reasoning acting and planning in language models, 2024

    Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models, 2024. 14 A Aggregation Strategies for PRM Scores To compute particle weights during generation, we aggregate per...

    2024

  27. [35]

    Please see the Evaluation section for our empirical results

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: We show experiments that directly address and back up every claim we make in the abstract and introduction. Please see...

  28. [36]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss several limitations of our work in the conclusion section. For clarity, we copy them here: "However, inference-time scaling comes with com...

  29. [37]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: We have 1 theorem in our paper, and its proof is provided in Appendix C. Guidelines: • The a...

  30. [38]

    We also provide key details into the hyperparameter selection and ablation process, which significantly helps reproducability

    Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  31. [39]

    How- ever, we will completely open source our code upon acceptance of the paper to encourage as many people as possible to use our work

    Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: We do not currently include ...

  32. [40]

    We list exactly which models and versions we used as generator models

    Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: We provide full details on al...

  33. [41]

    Experiment statistical significance 23 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: Due to computational limitations on an academic ...

  34. [42]

    Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [No] Justification: Although we include info...

  35. [43]

    Our research does not have negative societal consequences, nor does it involve human subjects

    Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] 24 Justification: We can confirm that, in every respect, we do not violate the NeurIPS Code of...

  36. [44]

    We discuss the positive impacts of inference scaling, as it opens up higher level language model performance to those who are only able to access smaller models

    Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Our work does not have any negative societal consequences. We discuss the positive impacts of inference s...

  37. [45]

    Instead, we only use off-the-shelf open source models, and therefore there are no possibilities of misuse

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [No] Justification: We do not relea...

  38. [46]

    Therefore, all creators of the original models are credited in this work

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Our work only us...

  39. [47]

    Justification: We do not release any new assets in this paper - instead, we discuss how to enhance the performance of already existing open-sourced models

    New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] . Justification: We do not release any new assets in this paper - instead, we discuss how to enhance the performance of already exis...

  40. [48]

    Guidelines: 26 • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  41. [49]

    Justification: Our work does not include any human subjects, and we did not need IRB approvals

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

  42. [50]

    Answer: [NA] Justification: LLM usage did not impact the core methodology, scientific rigorousness, or originality of the research

    Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the ...

    2025

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.