REVIEW 4 major objections 5 minor 30 references
Bayesian symbolic regression: Automated equation discovery from a physicists' perspective
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Symbolic regression should rank candidate equations by Bayesian plausibility, and that ranking reduces to minimizing a single description-length score that unifies fit, parameter cost, and prior complexity.
desk verdict A clear, well-written Bayesian manifesto for symbolic regression whose practical criterion inherits the advertised guarantees only under approximations the paper doesn't verify. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the marginal likelihood $p(D|m_i)=\int_{\Theta_i} d\theta_i\, p(D|m_i,\theta_i)p(\theta_i|m_i)$, approximated by Laplace's method. Its negative logarithm plus the model prior defines the description length $L(m_i,D)$, which the paper interprets as the number of nats needed to convey the model and the data, and equivalently as the energy (information Hamiltonian) of a canonical ensemble of models at inverse temperature $\beta=1$. The score $B_1/2-\log p(m_i)$ replaces three heuristic choices of traditional symbolic regression: the loss, the complexity measure, and the Pareto-front elbow selection. The optional $B_2$ term uses the Fisher information matrix to reward models whose predictions are insensitive to small parameter changes, and the prior $p(m_i)$ is made explicit and transparent through a maximum-entropy distribution over operator counts.
What would settle it
Generate data from a model with a flat or multimodal likelihood surface, such as $y=\theta\sin(\omega x)+\epsilon$ with a broad prior over $\omega$ and small $N$, and compare the model ranking produced by $L(m_i,D)$ against the exact marginal likelihood computed by numerical quadrature; a substantial mismatch, or frequent selection of a wrong model where the exact posterior favors the true one, would falsify the claim that the Laplace-based criterion inherits the Bayes-optimality guarantees.
Extended reading notes
Core claim
The central claim is that the complete solution to symbolic regression is the posterior distribution $p(m_i|D)\propto \exp[-L(m_i,D)]$ over candidate closed-form models, and that this distribution can be computed under explicit approximations. Marginalizing each model's parameters with Laplace's method gives $L(m_i,D)=B_1(m_i,D)/2-\log p(m_i)$, where $B_1(m_i,D)=-2\log p(D|m_i,\hat\theta_i)+(k_i+1)\log N$ is the Bayesian information criterion; keeping one more term replaces $B_1$ with $B_2$, which adds $\log\det I(\hat\theta_i)$, the Fisher information at the maximum-likelihood parameters. The paper claims that this one score integrates goodness of fit, parametric complexity, and structural prior plausibility in a single set of units, so the most plausible model is $\arg\min_{m_i} L(m_i,D)$. It further claims that traditional heuristic symbolic regression is implicitly using a prior $\log p_{\mathrm{trad}}\approx \frac{k_i+1}{2}\log N$ that favors more complex models, which explains its dramatic overfitting even on a constant model, and that the Bayesian criterion is consistent as $N\to\infty$ because the marginal likelihood is extensive while the prior is intensive. Finally, it claims that accounting for uncertainty forces one to consider whole ensembles of models, with important consequences including model averaging and a fundamental learnable-to-unlearnable transition.
Load-bearing premise
The practical ranking by $L(m_i,D)$ assumes the Laplace approximation is accurate: for every candidate model, the likelihood must be sharply peaked around its best-fit parameters and the prior smooth there, and when that fails the computed score is not the true log posterior, so the stated consistency and optimality guarantees no longer apply to the implemented method.
Editorial extensions
If this is right
- If the Bayesian criterion is adopted, any symbolic regression method minimizing a heuristic loss is implicitly using a prior that rewards complexity; minimizing $L(m_i,D)$ removes the need for ad hoc Pareto-front and elbow choices.
- The posterior over models, not the single most plausible equation, is the complete output of symbolic regression; predictions should be made by Bayesian model averaging over posterior samples.
- Because the marginal likelihood grows extensively with $N$ while the prior stays intensive, the criterion is consistent: as sample size grows it selects the true model with probability approaching one, as long as no true model has zero prior probability.
- There is a finite-sample learnability transition: above a certain noise level the true generating model ceases to be the minimum of the description-length landscape, so no algorithm can recover it from data alone, and near that transition the set of near-optimal models (a Rashomon set) is diverse and mutually inconsistent.
Reading between the lines
- Editorial inference: the same description-length objective could be grafted onto modern deep-learning symbolic regression by using $L(m_i,D)$ as the training or reward signal for a generator over expressions, turning a heuristic search into an approximately Bayesian sampler; the paper does not test this.
- Editorial inference: because the Laplace approximation is the weakest step, comparing $B_1/B_2$ rankings against exact numerical marginal likelihoods on small expression spaces would quantify how often the implemented criterion diverges from the true posterior, and where the stated consistency guarantee actually holds.
- Editorial inference: the learnability-transition result suggests a practical scientific diagnostic—reporting the spread of description lengths across the Rashomon set for a dataset would tell researchers whether the data are sufficient to pin down a unique law, rather than trusting one fitted equation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that symbolic regression should be founded on Bayesian probability rather than on heuristic objectives. It derives the model selection criterion L(m_i,D) = B1/2 - log p(m_i) (and the B2 variant with a Fisher-information term) from a Laplace approximation to the marginal likelihood, and then gives four arguments for the Bayesian approach: Cox's theorem, consistency, minimum description length, and Dutch-book coherence. It contrasts this with traditional symbolic regression (exemplified by PySR), reporting two synthetic experiments where the Bayesian Machine Scientist (BMS) outperforms PySR, and it discusses model averaging and Rashomon sets as consequences of working with the full posterior over models.
Significance. If the claims were fully established, the paper would provide a useful conceptual foundation for probabilistic symbolic regression and a clear argument that model ensembles, not single equations, are the appropriate output. The derivation of the BIC-type criterion from the Laplace approximation is standard and correctly presented, and the information-theoretic and statistical-physics interpretations are pedagogically valuable. The paper also makes a constructive point: traditional symbolic regression's complexity penalties can be seen as implicit and often unprincipled, whereas the Bayesian criterion makes all assumptions explicit. However, the paper overstates the guarantees it establishes: the strong consistency and optimality claims are proven only for the exact posterior under ideal conditions, not for the implemented BMS approximation, and the empirical evidence is a single run per condition with no repeated trials or code. These gaps are load-bearing because the paper's central argument is that the Bayesian approach 'provides guarantees of performance that heuristic approaches lack.'
major comments (4)
- [Section 4, consistency argument] The sentence 'any alternative that does not coincide with the Bayesian approach in this limit is virtually guaranteed to select the wrong model' is not justified by the preceding argument. Bayesian model selection consistency is known to hold only when the true model is contained in the model space with positive prior probability and under certain regularity conditions; the statement in Section 3a that the prior washes out 'as long as we do not assign p(m_i)=0' ignores the case where the true model is outside the search grammar or has zero prior under the chosen p(m). Without stating these conditions, the universal guarantee is unsubstantiated and should be weakened or qualified.
- [Section 2, Eqs. (2.5)-(2.9)] The consistency and Bayes-optimality arguments in Section 4 apply to the exact posterior p(m_i|D) in Eq. (2.4), but the implemented BMS uses the Laplace-approximated criteria in Eqs. (2.5)-(2.9). The assumptions that the likelihood is sufficiently peaked and the prior is smooth are stated, but the paper does not address model spaces where these fail: redundant parameters, near-collinear terms, and flat likelihood directions can make the Laplace integral dominated by boundaries or saddle points rather than a single interior peak, so the dropped O(1) terms are not negligible. Consequently, the advertised guarantees do not automatically transfer to the implemented BMS; the authors should either prove that the approximation is asymptotically valid for the model classes they consider or explicitly state that the guarantees hold for the exact posterior only.
- [Section 5, Figs. 1-2] The empirical demonstration consists of one dataset per (N,σ) cell, with no seeds, no repeated trials, and no code or configuration details beyond 'default parameters'. The statement that the example 'is sufficient to prove that the heuristics chosen do not lead to consistent model selection' is therefore too strong: a single run of PySR on a single dataset per condition cannot establish a general failure mode. The authors should provide repeated runs with error bars or at least rephrase these results as illustrative rather than proof.
- [Section 5, Eq. (5.4)] The assertion that 'the prior effectively being used by the traditional approach is log p_trad = (ki+1)/2 log N + ...' conflates a model-selection penalty with a Bayesian prior. A complexity penalty in a heuristic loss is not a prior over models, and the ellipsis hides the fact that the approximation terms in B1/B2 are part of the marginal likelihood, not a prior. The claim that traditional approaches 'favor a priori models with more, rather than fewer, parameters' is an analogy that holds only for this particular penalty form and should be presented as such, not as an identified prior.
minor comments (5)
- [Section 5, Eq. (5.1)] The coefficients α_o and β_o in the operator prior are not specified numerically; the paper cites previous work [14] but does not state the values or how they were estimated, which makes the prior non-reproducible from this manuscript alone.
- [Section 1, desideratum (ii)] The paper promises 'no ad hoc parameters or thresholds need to be fixed' to balance fit and complexity, but the prior in Eq. (5.1) contains parameters α_o and β_o fitted to a corpus. While these are not balancing parameters in the same sense, the text should acknowledge that the prior still requires a choice of corpus and estimation procedure.
- [Section 6(a)] Line 'consider a situation in which one whats to predict' contains a typo: 'whats' should be 'wants'.
- [Section 4, Cox argument] The statement that any reasoning system not conforming to Eq. (2.4) 'must violate some of the very basic commonsensical conditions assumed by Cox' is too categorical, given known limitations of Cox's theorem (e.g., requiring real-valued plausibilities and infinite precision). A softer formulation under the standard Cox axioms would be more accurate.
- [Section 6(b)] The learnability transition and the claim that no algorithm can identify the true model above a certain noise level are cited to [16] but not derived or illustrated in this manuscript; for a self-contained perspective, a brief informal derivation or a reference to the precise statement in [16] would help.
Circularity Check
No significant circularity: the derivation of the Bayesian description-length criterion is self-contained, and self-citations are illustrative or point to prior results with independent derivations.
full rationale
The central derivation is not circular. Section 2 starts from Bayes' theorem (Eq. 2.2) and the Gaussian likelihood (Eq. 2.3), marginalizes over parameters (Eq. 2.4), and then obtains L(mi,D)=B1/2-log p(mi) via the Laplace approximation under explicitly stated conditions (Eqs. 2.5-2.7), with the optional B2 correction (Eqs. 2.8-2.9). This is a standard Bayesian model-selection derivation, and the objective is not defined in terms of the outputs it later explains. The consistency and Bayes-optimality arguments in Section 4 rely on external axiomatizations (Cox, de Finetti) and on the standard extensivity/intensity argument for the marginal likelihood versus the prior; the paper even acknowledges the needed caveat that no model may be assigned zero prior. The empirical comparisons in Section 5 use the authors' own BMS implementation (Ref. 14), but the BMS outputs are not fitted to the two synthetic datasets shown, and the prior in Eq. (5.1) is explicitly labeled as an arbitrary but transparent choice. Claims about the learnability transition and Rashomon sets in Section 6 cite prior published work (Refs. 14, 15, 16) that contains the derivations; citing one's own earlier proofs is not circular unless the present argument reduces to the citation, which it does not. The main practical risk, that the Laplace approximation may be inaccurate for some candidate models so that the implemented B1/B2 criterion is not exactly the log posterior, is an approximation-validity limitation rather than a circularity, and the paper presents that approximation as an explicit assumption.
Assumptions & free parameters
free parameters (1)
- α_o, β_o (operator prior coefficients) =
Not given in this paper; fitted to a corpus of formulas in ref [14]
assumptions (4)
- domain assumption The observed data are generated by a true closed-form model m* in the candidate space M plus independent Gaussian noise with unknown variance (Eq. 2.1).
- domain assumption The Laplace approximation is accurate: likelihood sufficiently peaked around the MLE and prior smooth, for every model in M (Section 2).
- standard math Cox's theorem and de Finetti's Dutch book theorem justify probability as the unique consistent framework for degrees of belief (Section 4).
- ad hoc to paper The maximum-entropy operator prior of Eq. (5.1) is a reasonable representation of prior plausibility over scientific formulas.
Cite this review
Pith. "Pith review of Bayesian symbolic regression: Automated equation discovery from a physicists' perspective." pith.science (2026). https://pith.science/paper/EO7X47AA
@misc{pith2026250719540,
author = {Pith},
title = {Pith review of: Bayesian symbolic regression: Automated equation discovery from a physicists' perspective},
year = {2026},
howpublished = {\url{https://pith.science/paper/EO7X47AA}},
note = {Machine review of arXiv:2507.19540}
}
read the original abstract
Symbolic regression automates the process of learning closed-form mathematical models from data. Standard approaches to symbolic regression, as well as newer deep learning approaches, rely on heuristic model selection criteria, heuristic regularization, and heuristic exploration of model space. Here, we discuss the probabilistic approach to symbolic regression, an alternative to such heuristic approaches with direct connections to information theory and statistical physics. We show how the probabilistic approach establishes model plausibility from basic considerations and explicit approximations, and how it provides guarantees of performance that heuristic approaches lack. We also discuss how the probabilistic approach compels us to consider model ensembles, as opposed to single models.
Figures
Reference graph
Works this paper leans on
-
[16]
2023 Fundamental limits to learning closed-form mathematical models from data
Fajardo-Fontiveros O, Reichardt I, De Los Ríos HR, Duch J, Sales-Pardo M, Guimerà R. 2023 Fundamental limits to learning closed-form mathematical models from data. Nat. Comm. 14, 1043
work page 2023
-
[1]
2023 Scientific discovery in the age of artificial intelligence
Wang H, Fu T, Du Y, Gao W, Huang K, Liu Z, Chandak P , Liu S, Van Katwyk P , Deac A, Anandkumar A, Bergen K, Gomes CP , Ho S, Kohli P , Lasenby J, Leskovec J, Liu TY, Manrai A, Marks D, Ramsundar B, Song L, Sun J, Tang J, Veliˇ ckovi´ c P , Welling M, Zhang L, Coley CW, Bengio Y, Zitnik M. 2023 Scientific discovery in the age of artificial intelligence. N...
-
[2]
2023 Combining data and theory for derivable scientific discovery with AI-Descartes
Cornelio C, Dash S, Austel V , Josephson TR, Goncalves J, Clarkson KL, Megiddo N, El Khadir B, Horesh L. 2023 Combining data and theory for derivable scientific discovery with AI-Descartes. Nat. Commun. 14, 1777. 4After the movie Rashomon, in which Akira Kurosawa concatenates a series of contradictory accounts of the same crime. 11royalsocietypublishing.o...
work page 2023
-
[3]
1987 Scientific Discovery: Computational Explorations of the Creative Processes
Langley PW, Simon HA, Bradshaw G, Zytkow JM. 1987 Scientific Discovery: Computational Explorations of the Creative Processes. The MIT Press, Cambridge
work page 1987
-
[4]
2007 Computational Discovery of Scientific Knowledge
Džeroski S, Todorovski L, editors. 2007 Computational Discovery of Scientific Knowledge. Lecture Notes in Artificial Intelligence. Springer
work page 2007
-
[5]
1992 Genetic programming: On the programming of computers by means of natural selection
Koza JR. 1992 Genetic programming: On the programming of computers by means of natural selection. Cambridge, MA: MIT Press
work page 1992
-
[6]
2009 Distilling free-form natural laws from experimental data
Schmidt M, Lipson H. 2009 Distilling free-form natural laws from experimental data. Science 324, 81–5. (10.1126/science.1165893)
-
[7]
2023 Interpretable Machine Learning for Science with PySR and SymbolicRegression.jl
Cranmer M. 2023 Interpretable Machine Learning for Science with PySR and SymbolicRegression.jl. arXiv:2305.01582 (10.48550/arXiv.2305.01582)
Show all 30 references
-
[8]
2016 Discovering governing equations from data by sparse identification of nonlinear dynamical systems
Brunton SL, Proctor JL, Kutz JN. 2016 Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proc. Natl. Acad. Sci. U.S.A. 113, 3932–3937
2016
-
[9]
2021 Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients
Petersen BK, Larma ML, Mundhenk TN, Santiago CP , Kim SK, Kim JT. 2021 Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. In International Conference on Learning Representations
2021
-
[10]
2023 Efficient generator of mathematical expressions for symbolic regression
Mežnar S, Džeroski S, Todorovski L. 2023 Efficient generator of mathematical expressions for symbolic regression. Mach. Learn. 112, 4563–4596
2023
-
[11]
2020 AI Feynman: A physics-inspired method for symbolic regression
Udrescu SM, Tegmark M. 2020 AI Feynman: A physics-inspired method for symbolic regression. Sci. Adv. 6, eaay2631
2020
-
[12]
2021 Contemporary symbolic regression methods and their relative performance
La Cava W, Burlacu B, Virgolin M, Kommenda M, Orzechowski P , de França FO, Jin Y, Moore JH. 2021 Contemporary symbolic regression methods and their relative performance. Adv. Neural Inf. Process. Syst. 2021, 1
2021
-
[13]
2024 Interpretable scientific discovery with symbolic regression: a review
Makke N, Chawla S. 2024 Interpretable scientific discovery with symbolic regression: a review. Artificial Intelligence Review 57, 2. (10.1007/s10462-023-10622-0)
2024 doi
-
[14]
2020 A Bayesian machine scientist to aid in the solution of challenging scientific problems
Guimerà R, Reichardt I, Aguilar-Mogas A, Massucci FA, Miranda M, Pallarès J, Sales-Pardo M. 2020 A Bayesian machine scientist to aid in the solution of challenging scientific problems. Sci. Adv. 6, eaav6971
2020
-
[15]
2020 Bayesian machine scientist to compare data collapses for the Nikuradse dataset
Reichardt I, Pallarès J, Sales-Pardo M, Guimerà R. 2020 Bayesian machine scientist to compare data collapses for the Nikuradse dataset. Phys. Rev. Lett. 124, 084503
2020
-
[17]
1978 Estimating the dimension of a model
Schwarz G. 1978 Estimating the dimension of a model. Ann. Stat. 6, 461–464
1978
-
[18]
2010 Bayesian model selection and statistical modeling
Ando T. 2010 Bayesian model selection and statistical modeling. CRC Press
2010
-
[19]
2024 Exhaustive Symbolic Regression
Bartlett DJ, Desmond H, Ferreira PG. 2024 Exhaustive Symbolic Regression. IEEE Transactions on Evolutionary Computation 28, 950–964. (10.1109/TEVC.2023.3280250)
2024
-
[20]
2007 The Minimum Description Length Principle
Grünwald PD. 2007 The Minimum Description Length Principle. Cambridge, Massachusetts: The MIT Press
2007
-
[21]
2019 Information theory for fields
Enßlin TA. 2019 Information theory for fields. Annalen der Physik 531, 1800127. (10.1002/andp.201800127)
2019 doi
-
[22]
1946 Probability, frequency and reasonable expectation
Cox RT. 1946 Probability, frequency and reasonable expectation. Am. J. Phys. 14, 1–10
1946
-
[23]
2003 Probability Theory: The Logic of Science
Jaynes ET. 2003 Probability Theory: The Logic of Science. Cambridge University Press
2003
-
[24]
2022 Dutch Book Arguments
Vineberg S. 2022 Dutch Book Arguments. In Zalta EN, Nodelman U, editors, The Stanford Encyclopedia of Philosophy , . Metaphysics Research Lab, Stanford University Fall 2022 edition
2022
-
[25]
2023 Priors for symbolic regression
Bartlett D, Desmond H, Ferreira P . 2023 Priors for symbolic regression. In Proceedings of the Companion Conference on Genetic and Evolutionary Computation GECCO ’23 Companion p. 2402–2411 New York, NY, USA. Association for Computing Machinery. (10.1145/3583133.3596327)
2023
-
[26]
2024 Incorporating background knowledge in symbolic regression using a computer algebra system.Machine Learning: Science and Technology 5, 025057
Fox C, Tran ND, Nacion FN, Sharlin S, Josephson TR. 2024 Incorporating background knowledge in symbolic regression using a computer algebra system.Machine Learning: Science and Technology 5, 025057. (10.1088/2632-2153/ad4a1e)
2024 doi
-
[27]
1999 Bayesian model averaging: A tutorial
Hoeting JA, Madigan D, Raftery AE, Volinsky CT. 1999 Bayesian model averaging: A tutorial. Stat. Sci. 14, 382–417
1999
-
[28]
2019 Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead
Rudin C. 2019 Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1, 206–215
2019
-
[29]
2025 Human mobility is well described by closed-form gravity-like models learned automatically from data.Nat
Cabanas-Tirapu O, Danús L, Moro E, Sales-Pardo M, Guimerà R. 2025 Human mobility is well described by closed-form gravity-like models learned automatically from data.Nat. Comm. 16, 1336. 12royalsocietypublishing.org/journal/rsta Phil. T rans. R. Soc. A 0000000
2025
-
[30]
2024 Artificial intelligence needs a scientific method-driven reset
Amaral LAN. 2024 Artificial intelligence needs a scientific method-driven reset. Nat. Phys. 20, 523–524
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.