Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Are LLM Belief Updates Consistent with Bayes' Theorem?

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Larger language models update their beliefs about propositions more consistently with Bayes' theorem, with the Bayesian Coherence Coefficient rising roughly log-linearly with model scale.

desk verdict A useful, honest new measurement of LLM Bayesian coherence with a real scaling trend, but the headline metric ignores the systematic under-updating that the paper itself reports, so the 'more Bayesian' framing needs qualification. read the letter →

arxiv 2507.17951 v1 pith:74AS3QFG submitted 2025-07-23 cs.CL cs.AI

Sohaib Imran , Ihor Kendiukhov , Matthew Broerman , Aditya Thomas , Riccardo Campanella , Rob Lamb , Peter M. Atkinson This is my paper · ORCID
classification cs.CLcs.AI
keywords BayesianCoherenceCoefficientlargelanguagemodelsbeliefupdatingmodelscalingin-contextlearningcredenceconsistencymetricsAIsafety
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether language models update their beliefs about propositions in a way consistent with Bayes' theorem when evidence appears in the conversation. It introduces a Bayesian Coherence Coefficient (BCC), defined as the correlation between the expected log likelihood ratio of evidence under two candidate classes and the model's observed log odds update from prior to posterior, with all probabilities estimated by cumulative token probabilities. Across five pretrained model families, BCC increases approximately log-linearly with parameter count ($r = 0.906$, $p < 10^{-6}$) and correlates significantly with performance on four of six common benchmarks. The authors take this as early evidence that larger, more capable models learn more Bayes-consistent belief updates, which they argue matters for interpretability, steerability, and the safety risks of systems that begin to resemble expected-utility maximizers.

What carries the argument

The load-bearing object is the Bayesian Coherence Coefficient (BCC), a correlation between expected and observed log-odds updates. For a pair of candidate classes $(c_1,c_2)$, evidence $x$, conversation history $h$, and category $k$, the expected update is the log likelihood ratio $\log [P_\theta(x|c_1,h,k)/P_\theta(x|c_2,h,k)]$, and the observed update is the log posterior ratio minus the log prior ratio, $\log [P_\theta(c_1|x,h,k)/P_\theta(c_2|x,h,k)] - \log [P_\theta(c_1|h,k)/P_\theta(c_2|h,k)]$. All four probabilities are read from the model as cumulative conditional token probabilities of the class or evidence strings, with separate model instances used for priors, likelihoods, and posteriors. Because BCC is a correlation rather than an error metric, it does not reward flat, high-entropy output distributions the way the authors show an error-based alternative does; the metric therefore isolates whether updates move in the right direction and by roughly the right log-odds amount.

What would settle it

Ask a model to place bets or set prices on the candidate classes before and after each piece of evidence, and compute the same coherence coefficient from those decision-based odds updates. If the decision-based measure diverges from the token-probability BCC, or if its scaling trend with model size disappears, the paper's conclusion would be about text-generation probabilities rather than beliefs.

Watch

Extended reading notes

Core claim

The paper's central claim is that pre-trained language models become more coherent Bayesian updaters as they get larger: across five model families, the Bayesian Coherence Coefficient (BCC)—the correlation, over 6,460 (class pair, evidence, history, category) tuples, between the expected update $\log [P_\theta(x|c_1,h,k)/P_\theta(x|c_2,h,k)]$ and the observed update $\log [P_\theta(c_1|x,h,k)/P_\theta(c_2|x,h,k)] - \log [P_\theta(c_1|h,k)/P_\theta(c_2|h,k)]$—increases approximately log-linearly with parameter count ($r = 0.906$, $p < 10^{-6}$). Every tested model scores above zero, meaning even small models update in the right direction more often than a random policy, and the fitted gradient of observed versus expected updates is below 1 for all models, so models systematically under-update their credences rather than over-update them. Four of six benchmark correlations (BIG-Bench Hard, GPQA, MMLU-PRO, Math Lvl 5) are significantly positive; IFEval and MUSR are positive but not significant. The authors present this as evidence that scale and capability make in-context belief updates more Bayes-consistent, and they note the result must be reconciled with an earlier study that found no such improvement.

Load-bearing premise

Everything rests on treating the probability a model assigns to the words of a claim as the strength of its belief in that claim; the paper itself states that it is unclear whether these token probabilities reflect an action-relevant belief state.

Editorial extensions

If this is right

  • If the trend continues beyond the tested scale, larger pretrained models should be increasingly predictable and steerable by agents that share their world model, because their belief revisions move closer to the Bayesian ideal.
  • More coherent Bayesian updating makes it harder to conceal information from a model and harder to evaluate it without the model conditioning on the fact that it is being evaluated, since subtle cues in the evidence become inferable.
  • If coherent beliefs are paired with coherent preferences, future systems move closer to expected-utility maximizers, which brings clearer goal specification but also the risk of misaligned optimization, incorrigibility, and power seeking.
  • The systematic under-updating gradient (below 1) means all tested models move their credences less than full Bayesian updating demands; the authors connect this to the evidence strings being much less likely than the class strings, so datasets with more probable evidence would likely change the observed gradient.
  • Because BCC is a correlation, it is not confounded by the entropy of a model's output distribution, so it can compare coherence across models that differ widely in confidence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested but natural next experiment is to compute BCC from decision-based credences, such as prices or bets the model is prompted to place on each class before and after evidence, and compare this with the token-probability BCC; divergence would mean the paper's metric tracks text probabilities rather than beliefs.
  • The dataset and metric could be applied to instruction-tuned and reinforcement-learning-tuned models at fixed parameter counts, which the paper explicitly leaves to future work; it is open whether alignment training raises, lowers, or leaves BCC unchanged.
  • The positive correlations with general benchmarks raise the possibility that BCC is capturing general reasoning ability rather than specifically Bayesian updating; synthetic evidence with researcher-specified likelihood ratios would isolate the mechanism.
  • If BCC is accepted as a belief-coherence proxy, it could serve as a cheap monitoring signal during training or scaling: a drop in BCC even as benchmarks improve would flag a model whose in-context beliefs are becoming less coherent.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper proposes the Bayesian Coherence Coefficient (BCC), defined as the Pearson correlation between expected log-likelihood-ratio updates and observed log-odds updates over a set of (class pair, evidence, history, category) tuples, and uses it to ask whether larger pretrained language models update propositional credences more consistently with Bayes' theorem. The authors generate a dataset of ten categories with classes, evidence snippets, and conversation histories, compute the relevant cumulative token probabilities with the tested models, and report BCC for models from five families (Falcon, Llama, Qwen, GPT-2, Pythia), a training-step analysis for Pythia, and correlations with six benchmarks. They find that BCC increases with log parameter count (r = 0.906, p < 10^-6) and with four of six benchmarks, and they report that all tested models under-update, with a regression gradient of observed on expected updates less than 1. The paper concludes that larger and more capable LLMs update their credences more consistently with Bayes' rule.

Significance. The question is timely and the paper is transparent: code and data are public, the evaluation spans multiple model families, and the authors explicitly acknowledge the token-probability proxy as unvalidated. The self-consistency design, in which expected and observed updates are computed from the same autoregressive model, is appropriate and is not circular. However, the central metric is insensitive to exactly the systematic deviation from Bayes' theorem that the paper itself documents: because BCC is a correlation, a model that outputs Δ_observed = 0.1 Δ_expected plus small noise receives BCC near 1, indistinguishable from a perfectly Bayesian model. The scaling claim therefore conflates linear association with Bayesian consistency. The paper is a useful empirical starting point, but the headline conclusion needs to be re-derived with a metric that penalizes deviations from the identity line, or explicitly reframed as a claim about linear association rather than closeness to Bayes.

major comments (3)
  1. [Section 2, Eqs. (3)–(5); Table 1] The BCC is a Pearson correlation, which is invariant to separate positive affine transformations. Bayes' theorem requires Δ_observed = Δ_expected for every tuple, i.e., slope 1 and intercept 0 in the regression of Δ_observed on Δ_expected. The paper's own Table 1 reports that every model has an update gradient less than 1, so the systematic under-updating is invisible to BCC. Indeed, a model with Δ_observed = 0.1 Δ_expected + ε and arbitrarily small noise has BCC → 1, indistinguishable from a perfectly Bayesian model. Thus the main scaling result (r = 0.906, p < 10^-6) supports the statement that larger models' updates are more linearly predictable from the likelihood ratios, not that they are closer to the Bayes-optimal update. This is load-bearing for the abstract and Section 5 claims. I recommend reporting a metric that directly measures closeness to the identity line, such as a normalized mean squared error after matching variances, or the regression slope and intercept with their uncertainties, and presenting scaling results for those quantities.
  2. [Section 6; Eqs. (4)–(5)] The paper explicitly states in Section 6 that 'we use cumulative token probabilities as a proxy for credence in the proposition they compose' and that 'It is unclear whether this is an accurate proxy for action-relevant belief states.' Because every BCC value and every update gradient is computed from these text-generation probabilities, the empirical findings may characterize coherence of next-token probabilities rather than agent beliefs. This is not a circularity, since the test is a self-consistency test, but it is an unvalidated and load-bearing assumption. The paper should validate the proxy on a subset of items, for example by comparing with probabilities elicited through classification or betting tasks, or it should state clearly that the conclusions concern token-probability coherence only.
  3. [Section 4; Fig. 3; Fig. 5] The reported p-values for the scaling correlations are not reliable. The BCC for each model is computed over 6,460 tuples that are strongly non-independent because multiple class pairs, evidence items, and histories share the same category and the same model draws; p-values that treat each tuple as an independent observation are therefore far too small. Conversely, the correlation with log parameter count has only one point per model (roughly 10–16 models total), and p < 10^-6 is extremely sensitive to the exact number of models and to family membership; a mixed-effects or permutation analysis should be used. The six benchmark correlations are also not corrected for multiple comparisons. The manuscript should report confidence intervals, the exact number of models, and block-bootstrap or permutation tests that account for the dependence structure.
minor comments (5)
  1. [Section 2, Eq. (2)] The phrase 'where c1,c 2∈ C are pairs of classes' should read 'where c1 and c2 are distinct classes in C.'
  2. [Figure 2 caption] The statement that 'p values were too small to be properly rendered' should be replaced by an explicit bound, such as p < 10^-16, so that readers know the magnitude.
  3. [Section 3] The dataset is generated by GPT-4o through the ChatGPT interface, and the paper does not discuss whether the generation model's idiosyncrasies affect the resulting BCC values or provide a sensitivity analysis to the generation procedure.
  4. [Appendix C.1 and C.3] The desiderata require at least five classes per category, while the JSON schema sets minItems to 2; this inconsistency should be resolved.
  5. [Section 5, Table 1] The claim that larger models show an update gradient closer to 1 is reported descriptively in Table 1 but is not accompanied by a significance test or confidence interval; these quantities should be reported along with the BCC results.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: BCC is computed directly from model probabilities and the scaling claims are empirical, not fitted or definitionally forced.

full rationale

The paper's derivation chain defines the Bayesian Coherence Coefficient directly from model-assigned cumulative token probabilities (equations 3-5) and then correlates this coefficient with model size, training steps, and benchmark scores. No parameter is fitted to the target scaling result, and the expected and observed updates are distinct quantities constructed from the same autoregressive model, which is the point of a self-consistency test rather than a circular construction. The central assumption that cumulative token probabilities proxy for credence is explicitly acknowledged as a limitation in Section 6 ('we use cumulative token probabilities as a proxy for credence in the proposition they compose'), so it is an unvalidated premise, not a hidden reuse of the conclusion. The report that all tested models have update gradient below 1 and the use of Pearson correlation are metric-validity concerns, not circularity: they do not make the BCC equal to its inputs by definition, nor do they force the observed log-linear scaling with parameters. There are no load-bearing self-citations, no imported uniqueness theorems, and no renaming of a known result as a new derivation. Accordingly, the paper is not circular; at most it is vulnerable to criticisms about the validity of the credence proxy and the sensitivity of the chosen metric, which the authors themselves flag.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The BCC computation itself has no free parameters. The core load-bearing assumptions are that token probabilities are credences and that the generative text setup instantiates the Bayes terms. Both are flagged in the paper, with the token-probability assumption explicitly acknowledged as unvalidated.

assumptions (3)
  • domain assumption Cumulative token probabilities of class and evidence token sequences are a valid proxy for the model's credence in the corresponding proposition.
    Stated in Section 6 (Limitations): 'we use cumulative token probabilities as a proxy for credence in the proposition they compose. It is unclear whether this is an accurate proxy for action-relevant belief states.' This is load-bearing because BCC is computed entirely from these probabilities.
  • standard math Bayes' theorem, expressed as log odds updates for class pairs, is the correct normative standard for coherence.
    Background probability theory, used in Section 2 to define expected updates.
  • domain assumption The data generation desiderata (e.g., evidence favoring different classes) produce a dataset that exercises the relevant probability space.
    Dataset generation in Section 3 and Appendix C relies on GPT-4o to follow a JSON schema; the resulting distribution over classes and evidences is taken as a representative test bed, but there is no independent validation that the categories and evidences are unbiased or diverse enough.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Are LLM Belief Updates Consistent with Bayes' Theorem?." pith.science (2026). https://pith.science/paper/74AS3QFG

@misc{pith2026250717951,
  author       = {Pith},
  title        = {Pith review of: Are LLM Belief Updates Consistent with Bayes' Theorem?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/74AS3QFG}},
  note         = {Machine review of arXiv:2507.17951}
}
read the original abstract

Do larger and more capable language models learn to update their "beliefs" about propositions more consistently with Bayes' theorem when presented with evidence in-context? To test this, we formulate a Bayesian Coherence Coefficient (BCC) metric and generate a dataset with which to measure the BCC. We measure BCC for multiple pre-trained-only language models across five model families, comparing against the number of model parameters, the amount of training data, and model scores on common benchmarks. Our results provide evidence for our hypothesis that larger and more capable pre-trained language models assign credences that are more coherent with Bayes' theorem. These results have important implications for our understanding and governance of LLMs.

Figures

Figures reproduced from arXiv: 2507.17951 by the authors.

Figure 1
Figure 1. We compute priors, likelihoods and posteriors for all class (red), evidence (orange), history (grey) and category combi￾nations, as the cumulative log probabilities assigned to the under￾lined tokens, conditional upon the preceding text. Elicitation texts (blue), fixed for each category, are used to encourage the class and evidence tokens. The Bayesian Coherence Coefficient is computed as the correlation between the… view at source ↗
Figure 2
Figure 2. Scatterplots showing the observed updates (log odds up￾dates) against the expected updates (log likelihood ratios) for the Llama 3.2 1B and 3.1 8B models. Each point represents a (class pair, evidence, history, category) tuple from the dataset. The BCC of the model is the correlation (r value) between the expected and observed updates. p values were too small to be properly rendered and were therefore skipped from t… view at source ↗
Figure 3
Figure 3. Bayesian Coherence Coefficient as a function of number of model parameters. Each point represents a pre-trained model evaluated on the full dataset. The x-axis represents the number of parameters (in Billions) on a logarithmic scale [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Evolution of the BCC during training for four models with different numbers of parameters (see legend) from the Pythia model family. Each point represents a pre-trained model evaluated on the full dataset. The x-axis represents the number of training steps (in thousand…
Figure 5
Figure 5. Figure 5: BCC against (normalized) scores obtained by the models on a set of benchmarks commonly used to evaluate model performance. Each point represents a pre-trained model evaluated on the full dataset. The x-axis represents normalized benchmarks scores, with 0 as the random …
Figure 6
Figure 6. Figure 6: BCC for selected models across the categories in our dataset. to under-update, that is, why the gradient of observed vs. expected updates is less than 1 for all models. The inverse correlation between this update gradient and the negative evidence log likelihood averag…
Figure 7
Figure 7. Figure 7: Temperature invariance analysis showing how BCE and BCC metrics behave under different temperature settings. Temperature scaling modifies the entropy of model outputs without changing underlying model parameters. BCC shows more robust behavior compared to BCE across di…
Figure 8
Figure 8. Figure 8: Scatterplots showing the observed updates (log odds updates) against the expected updates (log likelihood ratios). Each point represents a (class pair, evidence, history, category) tuple from the dataset. The BCC of the model is the correlation (r value) between the ex…
Figure 9
Figure 9. Figure 9: Scatterplots showing BCC (blue) and the gradient between the observed and expected updates (red) against the evidence log likelihood averaged over the class pair. The dataset was sorted based on the average evidence log likelihood and binned into 10 equal subsets. Each…
Figure 10
Figure 10. Figure 10: Scatterplots showing the BCC (blue) and the gradient between the observed and expected updates (red) against the average of the prior and posterior log probabilities averaged over the class pair. The dataset was sorted based on the average evidence log likelihood and …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Disambiguating electrical detection of magnetization dynamics in magnetic insulators

    cond-mat.mes-hall 2026-04 unverdicted novelty 5.0 of 10

    Spin pumping and ST-FMR contributions to electrical signals in Pt/magnetic-insulator devices can be separated by geometry and field direction, showing that voltage sign is not a unique indicator of magnon chirality.

Reference graph

Works this paper leans on

25 extracted references · 8 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    A is B

    Berglund, L., Tong, M., Kaufmann, M., Balesni, M., Stickland, A. C., Korbak, T., and Evans, O. The Reversal Curse : LLMs trained on " A is B " fail to learn " B is A ", May 2024. URL http://arxiv.org/abs/2309.12288. arXiv:2309.12288 [cs]

    arXiv 2024

  3. [3]

    A., Purohit, S., Prashanth, U

    Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., and Wal, O. v. d. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling , May 2023. URL http://arxiv.org/abs/2304.01373. arXiv:2304.01373 [cs]

    arXiv 2023

  4. [4]

    and Hutter, M

    Everitt, T. and Hutter, M. The Alignment Problem for Bayesian History - Based Reinforcement Learners . DeepMind Technical Report, 2018

    work page 2018

  5. [5]

    Evaluation Faking : Unveiling Observer Effects in Safety Evaluation of Frontier AI Systems , May 2025

    Fan, Y., Zhang, W., Pan, X., and Yang, M. Evaluation Faking : Unveiling Observer Effects in Safety Evaluation of Frontier AI Systems , May 2025. URL http://arxiv.org/abs/2505.17815. arXiv:2505.17815 [cs]

    arXiv 2025

  6. [6]

    Evaluating Superhuman Models with Consistency Checks , October 2023

    Fluri, L., Paleka, D., and Tramèr, F. Evaluating Superhuman Models with Consistency Checks , October 2023. URL http://arxiv.org/abs/2306.09983. arXiv:2306.09983 [cs]

    arXiv 2023

  7. [7]

    Open LLM Leaderboard 2, 2024

    Fourrier, C., Habib, N., Lozovskaya, A., Szafer, K., and Wolf, T. Open LLM Leaderboard 2, 2024. URL https://huggingface.co/collections/open-llm-leaderboard/open-llm-leaderboard-2-660cdb7601eba6852431fffc

    work page 2024

  8. [8]

    Hammond, P. J. Consequentialist foundations for expected utility. Theory and Decision, 25 0 (1): 0 25--78, July 1988. ISSN 1573-7187. doi:10.1007/BF00129168. URL https://doi.org/10.1007/BF00129168

    work page doi:10.1007/bf00129168 1988

Show all 25 references
  1. [9]

    Measuring Mathematical Problem Solving With the MATH Dataset

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring Mathematical Problem Solving With the MATH Dataset . CoRR, abs/2103.03874, 2021. URL https://arxiv.org/abs/2103.03874. arXiv: 2103.03874

    2021 arXiv

  2. [10]

    v., Mikulik, V., Skalse, J., and Garrabrant, S

    Hubinger, E., Merwijk, C. v., Mikulik, V., Skalse, J., and Garrabrant, S. Risks from Learned Optimization in Advanced Machine Learning Systems , December 2021. URL http://arxiv.org/abs/1906.01820. arXiv:1906.01820 [cs]

    2021 arXiv

  3. [11]

    Bayesian Epistemology

    Lin, H. Bayesian Epistemology . In Zalta, E. N. and Nodelman, U. (eds.), The Stanford Encyclopedia of Philosophy . Metaphysics Research Lab, Stanford University, summer 2024 edition, 2024. URL https://plato.stanford.edu/archives/sum2024/entries/epistemology-bayesian/

    2024

  4. [12]

    W., Ren, R., Phan, L., Mu, N., Khoja, A., Zhang, O., and Hendrycks, D

    Mazeika, M., Yin, X., Tamirisa, R., Lim, J., Lee, B. W., Ren, R., Phan, L., Mu, N., Khoja, A., Zhang, O., and Hendrycks, D. Utility Engineering : Analyzing and Controlling Emergent Value Systems in AIs , February 2025. URL http://arxiv.org/abs/2502.08640. arXiv:2502.08640 [cs]

    2025 arXiv

  5. [13]

    Large Language Models Often Know When They Are Being Evaluated , June 2025

    Needham, J., Edkins, G., Pimpale, G., Bartsch, H., and Hobbhahn, M. Large Language Models Often Know When They Are Being Evaluated , June 2025. URL http://arxiv.org/abs/2505.23836. arXiv:2505.23836 [cs]

    2025 arXiv

  6. [14]

    and Lawsen, A

    Opus, C. and Lawsen, A. Comment on The Illusion of Thinking : Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity , June 2025. URL http://arxiv.org/abs/2506.09250. arXiv:2506.09250 [cs] version: 1

    2025 arXiv

  7. [15]

    P., Alvarez, A., Bhat, V., Shen, A., Wang, E., and Tramèr, F

    Paleka, D., Sudhir, A. P., Alvarez, A., Bhat, V., Shen, A., Wang, E., and Tramèr, F. Consistency Checks for Language Model Forecasters , January 2025. URL http://arxiv.org/abs/2412.18544. arXiv:2412.18544 [cs]

    2025 arXiv

  8. [16]

    M., Filan, D., and Shai, A

    Piotrowski, M., Riechers, P. M., Filan, D., and Shai, A. S. Constrained belief updates explain geometric structures in transformer representations, February 2025. URL http://arxiv.org/abs/2502.01954. arXiv:2502.01954 [cs]

    2025

  9. [17]

    L., Stickland, A

    Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. GPQA : A Graduate - Level Google - Proof Q & A Benchmark . ArXiv, November 2023. URL https://www.semanticscholar.org/paper/GPQA

    2023

  10. [18]

    The Illusion of Thinking : Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity , June 2025

    Shojaee, P., Mirzadeh, I., Alizadeh, K., Horton, M., Bengio, S., and Farajtabar, M. The Illusion of Thinking : Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity , June 2025. URL http://arxiv.org/abs/2506.06941. arXiv:2506.06941 [cs]

    2025 arXiv

  11. [19]

    Corrigibility

    Soares, N., Fallenstein, B., Armstrong, S., and Yudkowsky, E. Corrigibility. In AAAI Workshop : AI and Ethics , 2015. URL https://scholar.google.com/scholar?cluster=7316501884041445205&hl=en&oi=scholarr

    2015

  12. [20]

    Sondik, E. J. The Optimal Control of Partially Observable Markov Processes Over the Infinite Horizon : Discounted Costs . Operations Research, 26 0 (2): 0 282--304, 1978. ISSN 0030-364X. URL https://www.jstor.org/stable/169635. Publisher: INFORMS

    1978

  13. [21]

    MuSR : Testing the Limits of Chain -of-thought with Multistep Soft Reasoning , March 2024

    Sprague, Z., Ye, X., Bostrom, K., Chaudhuri, S., and Durrett, G. MuSR : Testing the Limits of Chain -of-thought with Multistep Soft Reasoning , March 2024. URL http://arxiv.org/abs/2310.16049. arXiv:2310.16049 [cs]

    2024 arXiv

  14. [22]

    W., Chowdhery, A., Le, Q

    Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., and Wei, J. Challenging BIG - Bench Tasks and Whether Chain -of- Thought Can Solve Them . arXiv, 2022. doi:10.48550/ARXIV.2210.09261. URL https://arxiv.org...

    2022 doi arXiv

  15. [23]

    MMLU - Pro : A More Robust and Challenging Multi - Task Language Understanding Benchmark

    Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., Li, T., Ku, M., Wang, K., Zhuang, A., Fan, R., Yue, X., and Chen, W. MMLU - Pro : A More Robust and Challenging Multi - Task Language Understanding Benchmark . In Globersons, A....

    2024

  16. [24]

    Instruction- Following Evaluation for Large Language Models , November 2023

    Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., and Hou, L. Instruction- Following Evaluation for Large Language Models , November 2023. URL http://arxiv.org/abs/2311.07911. arXiv:2311.07911 [cs]

    2023 arXiv

  17. [25]

    Åström, K. J. Optimal control of Markov processes with incomplete state information. Journal of Mathematical Analysis and Applications, 10 0 (1): 0 174--205, February 1965. ISSN 0022-247X. doi:10.1016/0022-247X(65)90154-X. URL https://www.sciencedirect.com/science/article/pii/...

    1965

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.