Pith. sign in

REVIEW 4 major objections 5 minor 24 references

SELT: Self-Evaluation Tree Search for LLMs with Task Decomposition

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SELT claims that a training-free Monte Carlo Tree Search variant, in which the LLM scores its own reasoning steps and clusters similar answers, improves accuracy on knowledge QA and tool learning.

desk verdict The paper's idea is plausible but its headline accuracy claim is not supported by the reported metrics, which mix single-answer and union scoring without any uncertainty. read the letter →

arxiv 2506.07557 v1 pith:22QSC6KG submitted 2025-06-09 cs.CL cs.AI

Mengsong Wu , Di Zhang , Yuqiang Li , Dongzhan Zhou , Wenliang Chen This is my paper
classification cs.CLcs.AI
keywords MonteCarloTreeSearchLLMself-evaluationtest-timescalingtaskdecompositionsemanticclusteringreasoningtoollearningknowledge-basedQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that LLMs can run their own tree search if the search score is rewritten around the model's self-evaluation instead of around a separately trained reward model. SELT decomposes a question into small atomic subtasks, grows a binary reasoning tree, and groups similar simulated answers at each node using semantic clustering so the LLM can judge a new answer against representative answers. The central claim is that this combination balances exploration and exploitation well enough to beat one-shot prompting, chain-of-thought prompting, and standard MCTS on the knowledge QA benchmark MMLU and the tool-learning benchmark Seal-Tools without fine-tuning. If true, test-time accuracy gains become a search-strategy problem rather than a problem of collecting reward labels or training another model.

What carries the argument

The load-bearing mechanism is a re-scored UCT selection rule. SELT replaces the raw exploitation average with the Bayesian estimate $\frac{\mu_{\mathrm{Tree}} C_\beta + Q(v')}{C_\beta + N(v')}$, where $\mu_{\mathrm{Tree}}$ is the tree-wide mean of the LLM's self-scores and $C_\beta=2$ is the prior strength, and replaces the exploration term with $C_p\sqrt{2\ln N(v')/N(v')}$, biasing the binary search tree toward deeper paths. The second mechanism is semantic-aware clustering: TF-IDF vectors, a cosine similarity matrix, the graph Laplacian's eigenvalue gap, and spectral clustering group the answers at each node so that cluster representatives anchor the LLM's next self-evaluation. Task decomposition supplies the atomic sentence-level subtasks that populate the tree, and together these pieces convert the LLM's own judgement into the search signal with no reward-model fine-tuning.

What would settle it

A concrete check is to run SELT on held-out questions from the same splits, collect every self-score that enters the paper's Equation 2, and compute their rank correlation with gold-label correctness; if that correlation is zero or negative while SELT still beats chain-of-thought, then the reported gains cannot be explained by the self-evaluation scoring and must come from the clustering or tree structure instead. A second check would randomize the reward scores while keeping clustering and tree structure fixed: accuracy should fall back toward the baseline if the self-scores are doing the claimed work.

Watch

Extended reading notes

Core claim

The central claim is that Monte Carlo Tree Search can be driven by the LLM's own judgement rather than an external reward model, provided the UCT formula is rebuilt for self-scoring. In SELT the exploitation term is a Bayesian average, $\frac{\mu_{\mathrm{Tree}} C_\beta + Q(v')}{C_\beta + N(v')}$, and the exploration term is $C_p\sqrt{2\ln N(v')/N(v')}$, with a binary tree and a 50% chance of descending directly to the current best child. At each node, simulated answers are embedded with TF-IDF, clustered by spectral clustering on the cosine-similarity Laplacian, and the best answer of each cluster becomes a reference for scoring the next answer. With an 8-billion-parameter instruct model and 100 search steps, the paper reports that sentence-level SELT outperforms one-shot, chain-of-thought, and raw MCTS baselines on the selected MMLU subsets and on Seal-Tools.

Load-bearing premise

The whole search trusts that the LLM's own scores for its intermediate answers track whether those answers are actually correct, and the paper does not measure that correlation; the limitations section concedes that self-evaluation errors can propagate through the tree search.

Editorial extensions

If this is right

  • If the results hold, test-time compute can buy accuracy improvements over prompting without any task-specific training, so gains can be allocated at inference time.
  • Sentence-level decomposition being stronger than response-level refinement suggests that a fine-grained search over next-sentence continuations is more productive than repeatedly revising full answers on these tasks.
  • The clustering step gives a second route to the answer by selecting among cluster representatives, and the reported Clusters hit rates indicate that the correct answer is usually inside the generated set even when the final picked answer is wrong.
  • The Seal-Tools results indicate the same self-evaluating search transfers from knowledge QA to procedural tool-calling tasks, improving format and parameter prediction in the single-tool split.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested extension is that the ceiling on SELT may be final selection rather than search: because the correct answer appears to be present among cluster representatives at high rates, a better pairwise or consistency-based choice among representatives could lift accuracy more than additional search steps.
  • Because the self-scores are never calibrated against gold correctness, the Bayesian prior $C_\beta$ and the exploration constant may need re-tuning per domain; a reader could test whether the same constants transfer to mathematics or code generation.
  • A natural transfer target is long-horizon agent planning, where step-level gold rewards are even scarcer than in QA and where the same trick of using cluster representatives as scoring anchors could apply.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SELT, a training-free MCTS-based inference framework for LLMs. It modifies the UCT exploitation term to use Bayesian averaging over self-evaluation scores, changes the exploration term to encourage deeper search, decomposes tasks into atomic subtasks with inference modes, and applies spectral clustering to simulated answers at each tree node. Experiments use Llama-3.1-8B-Instruct on partially selected MMLU splits and on Seal-Tools, comparing SELT variants against 1-shot prompting, 1-shot CoT, and raw MCTS under sentence-level and response-level granularities. The central claim is that SELT achieves significant improvements in answer accuracy and reasoning robustness without task-specific fine-tuning.

Significance. If the central claim were established, SELT would be a useful training-free test-time search method for improving LLM reasoning, and the paper's motivation avoiding external reward models is reasonable. The manuscript has some strengths: it releases code, evaluates on externally labeled gold data (so there is no circularity in the final accuracy measure), ablates the UCT modifications across Sraw/Salpha/Sbeta/Salpha+beta, and tests both sentence-level and response-level reasoning. However, the reported evidence does not currently support the headline claim: the directly comparable single-answer metric is mixed on MMLU, no uncertainty or significance testing is reported, and two of the reported metrics ('Both' and 'Clusters') measure union accuracy over multiple generated candidates rather than single-answer accuracy. The core self-evaluation signal is also unvalidated. These are load-bearing gaps, but they are addressable with additional experiments and re-analysis, so the paper warrants major revision rather than acceptance.

major comments (4)
  1. [Section 4.1, Table 2] The central claim of 'significant improvements' is not supported by the reported results. The only single-answer metric directly comparable to the 1-shot and 1-shot CoT baselines is 'Picked'. Under the raw UCT setting (Sraw), SELT is worse than 1-shot CoT on Physics (67.65 vs 71.57 at sentence level) and on Chemistry (55 vs 61), and only marginally better on Math (51 vs 49). The rows 'Both' and 'Clusters' count an answer as correct if the gold label appears among two candidates or among up to five cluster representatives, so they inflate apparent accuracy relative to single-answer baselines and cannot be used to support the abstract's claim. No confidence intervals, standard errors, or significance tests are reported anywhere, and Section 4.1 states only part of each benchmark was used without giving the exact sample sizes. The strongest configuration, Salpha+beta Picked, does improve over CoT on all three MMLU domains (50 vs 49, 80.39 vs 71.57, 64 vs 61), but the paper must designate this as the primary configuration and quantify its uncertainty before claiming a significant effect.
  2. [Section 3.4, Eq. (2)] The entire search is driven by the LLM's self-assigned reward scores, used in the Bayesian averaging term Q(v') of the UCT exploitation component, yet the paper never validates that these self-scores correlate with gold-label correctness. Section 3.4 merely states that with representative references 'the LLM can assess the new simulated answer better', and the Limitations section concedes that 'errors or biases in self-evaluation could propagate through the tree search'. This is a correctness-risk concern: if the self-reward is noisy or biased, the search is maximizing an unvalidated proxy and the reported accuracy gains could vanish or invert. The authors should add a calibration or correlation analysis, e.g., comparing self-scores against gold correctness on a held-out subset, or comparing SELT against a version using oracle reward scores.
  3. [Section 3.1 and Section 4] Task decomposition and semantic clustering are presented as key innovations (contribution 2), but no ablation isolates their contribution. The experiments compare SELT against 1-shot CoT and raw MCTS, but SELT includes decomposition, clustering, and modified UCT simultaneously, so the reported gains confound multiple changes. Table 3 only varies the UCT scoring components (Sraw, Salpha, Sbeta, Salpha+beta); it does not remove or disable the task decomposition or the clustering modules. To support the claim that both innovations are effective, the authors need an ablation that turns off decomposition and/or clustering while holding the rest of the search fixed.
  4. [Algorithm 2] The selection and expansion pseudocode is underspecified and appears internally inconsistent. After `expand(v)`, the function returns the expanded node only when its visit count is zero and otherwise recursively calls `tree_policy(vinitial)`, which restarts from the initial root rather than continuing from the expanded node. In the final `else` branch, `best_child(v, Cp)` is called without assigning or returning its value. These details make it difficult to reproduce the claimed depth-prioritizing exploration behavior and should be corrected or clarified with a precise control-flow description.
minor comments (5)
  1. [References] The reference to 'V on Luxburg' should be 'von Luxburg', and 'bayes averaging' in Section 3.4 should be 'Bayesian averaging'.
  2. [Table 1] There is a typo in Table 1: 'specifc' should be 'specific'.
  3. [Tables 2 and 3] The labels 'Maths' and 'Mathematics' are used inconsistently for the same MMLU split, and the multi-column P/R/F1 headers in Tables 3 and 4 are extremely hard to parse; please reformat them.
  4. [Section 4.1] The exact number of test examples per MMLU domain and per Seal-Tools split is not reported, despite the statement that only part of each benchmark was used; please provide these numbers for reproducibility.
  5. [Section 3.4] The free parameters C_beta=2, the cluster cap of 5, and the expansion probability of 0.5 are fixed without sensitivity analysis; even a brief stability check would strengthen the claim that the results do not depend on these choices.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the method's outputs are evaluated against external benchmark labels, and the internal self-evaluation scoring is not derived from or fitted to those labels.

full rationale

The paper's derivation chain is self-contained. The modified UCT scores in Equations 2 and 3 are constructed from the LLM's own self-evaluation rewards, visit counts, and a tree-average term; no gold answer or final accuracy is fed back into the search objective. The final headline accuracy is measured against the held-out answer labels of MMLU and Seal-Tools, so the central claim is an empirical comparison rather than a tautology. The 'Both' and 'Clusters' rows are coverage metrics over candidate sets produced by the search, but these metrics are not used as inputs to the algorithm; they are evaluation choices that may inflate apparent accuracy, which is a reporting concern, not circularity. The benchmark Seal-Tools is cited from the authors' prior work, but it is an externally fixed dataset with its own labels and is therefore independent evidence under the review rules. The reliance on unvalidated LLM self-scores is a genuine correctness risk, and the paper's Limitations section concedes that biased self-evaluation could propagate through the search, but that does not make the derivation circular because the evaluation is external. No step was found where a prediction reduces to a fitted parameter or to a self-citation chain by construction.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical entities, but it relies on several unvalidated modeling assumptions. The core free parameters (C_beta, cluster cap, 0.5 probability) are chosen by hand and not swept; the self-evaluation reward is the largest unvalidated input.

free parameters (3)
  • C_beta = 2
    Expected evaluation count in Bayesian averaging (Equation 2); set by hand in Section 3.4.
  • cluster_cap = 5
    Upper bound on spectral-cluster count (Section 3.3).
  • expansion_probability = 0.5
    Probability of selecting best_child vs. expanding in tree_policy (Algorithm 2).
assumptions (5)
  • domain assumption LLM self-evaluation scores are an informative reward signal for reasoning quality
    The entire UCT reward (Equation 2) is built on LLM-generated scores; never validated against gold labels; flagged in Limitations.
  • domain assumption TF-IDF cosine similarity captures semantic equivalence of free-form LLM answers for clustering
    Used to build the similarity matrix for spectral clustering (Section 3.3); no evidence or ablation that this representation is adequate.
  • domain assumption The eigengap of the Laplacian of the answer-similarity graph indicates the correct number of answer clusters
    Used to set the cluster count (Section 3.3); not validated.
  • ad hoc to paper A binary action space is sufficient for representing LLM reasoning steps
    "After preliminary evaluation, we adopt the binary tree as the search tree" (Section 3.4); no evidence given.
  • ad hoc to paper The modified UCT exploration term maintains a useful exploration-exploitation tradeoff
    Derivation only shows monotonicity for a<=b, not that the "more gradual" ratio is beneficial (Section 3.4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of SELT: Self-Evaluation Tree Search for LLMs with Task Decomposition." pith.science (2026). https://pith.science/paper/22QSC6KG

@misc{pith2026250607557,
  author       = {Pith},
  title        = {Pith review of: SELT: Self-Evaluation Tree Search for LLMs with Task Decomposition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/22QSC6KG}},
  note         = {Machine review of arXiv:2506.07557}
}
read the original abstract

While Large Language Models (LLMs) have achieved remarkable success in a wide range of applications, their performance often degrades in complex reasoning tasks. In this work, we introduce SELT (Self-Evaluation LLM Tree Search), a novel framework that leverages a modified Monte Carlo Tree Search (MCTS) to enhance LLM reasoning without relying on external reward models. By redefining the Upper Confidence Bound scoring to align with intrinsic self-evaluation capabilities of LLMs and decomposing the inference process into atomic subtasks augmented with semantic clustering at each node, SELT effectively balances exploration and exploitation, reduces redundant reasoning paths, and mitigates hallucination. We validate our approach on challenging benchmarks, including the knowledge-based MMLU and the Tool Learning dataset Seal-Tools, where SELT achieves significant improvements in answer accuracy and reasoning robustness compared to baseline methods. Notably, our framework operates without task-specific fine-tuning, demonstrating strong generalizability across diverse reasoning tasks. Relevant results and code are available at https://github.com/fairyshine/SELT .

Figures

Figures reproduced from arXiv: 2506.07557 by the authors.

Figure 1
Figure 1. Tree Search with the Foundation Language [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Four Task Type for LLMs. (T/F, Choice, FITB, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Four Inference Mode of LLMs. (Learn, Think, [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Node Composition. Structure info is directly [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: The Whole Procedure of SELT. tree. The number of nodes grows exponentially with increasing degree, providing a sufficiently large reasoning space in this setting. Algorithm 2: Selection & Expansion Function: tree_policy(v) Input: original node v Output: selected node v…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 8 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  4. [4]

    Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. 2012. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games, 4(1):1--43

    2012

  5. [5]

    Gr \'e goire Del \'e tang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, et al. 2023. Language modeling is compression. arXiv preprint arXiv:2309.10668

    arXiv 2023

  6. [6]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  7. [7]

    Sylvain Gelly and Yizao Wang. 2006. Exploration exploitation in go: Uct for monte-carlo go. In NIPS: Neural Information Processing Systems Conference On-line trading of Exploration and Exploitation Workshop

    work page 2006

  8. [8]

    Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. 2023. Reasoning with language model is planning with world model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8154--8173

    2023

Show all 24 references
  1. [9]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  2. [10]

    Ariana Martino, Michael Iannelli, and Coleen Truong. 2023. Knowledge injection to counter large language model (llm) hallucination. In European Semantic Web Conference, pages 182--185. Springer

    2023

  3. [11]

    Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. 2024. Logicbench: Towards systematic evaluation of logical reasoning ability of large language models. In Proceedings of the 62nd Annual Meeting of the As...

    2024

  4. [12]

    Juan Ramos et al. 2003. Using tf-idf to determine word relevance in document queries. In Proceedings of the first instructional conference on machine learning, volume 242, pages 29--48. Citeseer

    2003

  5. [13]

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314

    2024 arXiv

  6. [14]

    Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Haitao Mi, and Dong Yu. 2024. Toward self-improvement of llms via imagination, searching, and criticizing. arXiv preprint arXiv:2404.12253

    2024 arXiv

  7. [15]

    Ulrike Von Luxburg. 2007. A tutorial on spectral clustering. Statistics and computing, 17:395--416

    2007

  8. [16]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  9. [17]

    Mengsong Wu, Tong Zhu, Han Han, Chuanyuan Tan, Xiang Zhang, and Wenliang Chen. 2024. Seal-tools: Self-instruct tool learning dataset for agent tuning and detailed benchmark. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 372--384. Springer

    2024

  10. [18]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36

    2024

  11. [19]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. https://openreview.net/forum?id=WE_vluYUL-X React: Synergizing reasoning and acting in language models . In The Eleventh International Conference on Learning Representations

    2023

  12. [20]

    Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. 2025. Rest-mcts*: Llm self-training via process reward guided tree search. Advances in Neural Information Processing Systems, 37:64735--64772

    2025

  13. [21]

    Di Zhang, Xiaoshui Huang, Dongzhan Zhou, Yuqiang Li, and Wanli Ouyang. 2024 a . Accessing gpt-4 level mathematical olympiad solutions via monte carlo tree self-refine with llama-3 8b. arXiv preprint arXiv:2406.07394

    2024 arXiv

  14. [22]

    Di Zhang, Jianbo Wu, Jingdi Lei, Tong Che, Jiatong Li, Tong Xie, Xiaoshui Huang, Shufei Zhang, Marco Pavone, Yuqiang Li, et al. 2024 b . Llama-berry: Pairwise optimization for o1-like olympiad-level mathematical reasoning. arXiv preprint arXiv:2410.02884

    2024 arXiv

  15. [23]

    Zirui Zhao, Wee Sun Lee, and David Hsu. 2024. Large language models as commonsense knowledge for large-scale task planning. Advances in Neural Information Processing Systems, 36

    2024

  16. [24]

    Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. 2024. https://proceedings.mlr.press/v235/zhou24r.html Language agent tree search unifies reasoning, acting, and planning in language models . In Proceedings of the 41st International Conference on ...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.