{"id":"41f7b3d8-3a13-46d6-8428-5d79709c53f7","arxiv_id":"2507.19540","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Bayesian symbolic regression, using the posterior over formulas and explicit priors, is presented as the principled foundation for equation discovery, while heuristic symbolic regression is shown to overfit even constant data.","lead":"Guimerà and Sales-Pardo argue that symbolic regression, the automated search for equations from data, should be built on Bayesian probability rather than heuristic model selection, and illustrate the idea on two simple noisy datasets. The paper matters because it gives scientists a principled checklist for judging automated equation discovery and argues for reporting whole distributions of plausible models instead of a single fitted equation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The exact-Bayesian guarantees do not transfer to the implemented B1/B2 criterion unless the true model lies in the grammar with positive prior and the Laplace approximation holds for every candidate; neither condition is established.","rationale":"The reader's weakest assumption is the Laplace approximation, and that is indeed one load-bearing point: the paper's practical L(mi,D) is only a valid proxy for the exact log posterior when the Laplace integral is accurate, which is not guaranteed for the redundant and sloppy parameterizations common in symbolic regression. My reading agrees with this. However, I see an equally load-bearing concern that the reader does not foreground: the consistency and Bayes-optimality guarantees in Section 4 are conditional on the prior being the true generating distribution and on the true model being in the model class with positive prior, yet the paper later drops this condition when claiming that an unknown prior still yields consistency and that any non-Bayesian alternative is virtually guaranteed to err. The corpus-based prior in Eq. (5.1) is not shown to satisfy the positivity/coverage condition, so the guarantees do not automatically transfer to the implemented BMS. Given the paper is an explicit position statement with clearly stated approximations, I do not think this warrants rejection; but the overclaim should be qualified, and the empirical demonstrations should be made reproducible with repeated trials. Thus I keep the reader's CONDITIONAL verdict and mark partial agreement.","tokens_in":11902,"tokens_out":7006,"duration_ms":88314,"concrete_test":"Compute the exact marginal likelihood in Eq. (2.4) by numerical integration over parameters for a small model space containing a deliberately redundant expression (e.g., y=θ1+θ2+θ3x with θ1, θ2 confounded) and a well-identified expression, at N=50, 100, and 500. Compare the B1/B2 posterior weights with the numerically integrated posterior; if the rank ordering of models changes, the implemented criterion is not the advertised posterior. Separately, rerun Figs. 1–2 with 100 independent seeds per condition and report selection frequencies, since single-run figures cannot substantiate the claim that a non-Bayesian method is 'virtually guaranteed' to select the wrong model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that minimizing L(mi,D)=B1/2−log p(mi) inherits the exact posterior's consistency and Bayes-optimality guarantees. That inheritance fails if (a) the generating model is not in the search grammar or has zero prior under Eq. (5.1), or (b) the Laplace approximation behind Eqs. (2.5)–(2.9) is not accurate. Condition (a) is not a mild technicality: Section 4's 'provided p(m) is known and used as prior' is an assumption the authors drop in the next paragraph, where they say even an unknown prior suffices because it is intensive. But the prior washing out only helps if the true model has positive support; otherwise no amount of data can rescue it. Condition (b) is equally load-bearing. Symbolic-regression model spaces contain expressions with redundant parameters, near-collinear terms, and flat likelihood directions; for these, the Laplace integral is not dominated by a single interior peak, the Fisher information can be singular, and the dropped O(1) terms involving the parameter prior and Hessian determinant are not negligible. The two single-run demonstrations (Figs. 1–2, one dataset per cell, no seeds or code) cannot establish the asymptotic claim that PySR is 'virtually guaranteed' to fail. The advertised guarantees therefore belong to the exact Bayesian ideal, not to the implemented BMS.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that symbolic regression should be founded on Bayesian probability rather than on heuristic objectives. It derives the model selection criterion L(m_i,D) = B1/2 - log p(m_i) (and the B2 variant with a Fisher-information term) from a Laplace approximation to the marginal likelihood, and then gives four arguments for the Bayesian approach: Cox's theorem, consistency, minimum description length, and Dutch-book coherence. It contrasts this with traditional symbolic regression (exemplified by PySR), reporting two synthetic experiments where the Bayesian Machine Scientist (BMS) outperforms PySR, and it discusses model averaging and Rashomon sets as consequences of working with the full posterior over models.","tokens_in":12076,"tokens_out":4110,"duration_ms":48784,"significance":"If the claims were fully established, the paper would provide a useful conceptual foundation for probabilistic symbolic regression and a clear argument that model ensembles, not single equations, are the appropriate output. The derivation of the BIC-type criterion from the Laplace approximation is standard and correctly presented, and the information-theoretic and statistical-physics interpretations are pedagogically valuable. The paper also makes a constructive point: traditional symbolic regression's complexity penalties can be seen as implicit and often unprincipled, whereas the Bayesian criterion makes all assumptions explicit. However, the paper overstates the guarantees it establishes: the strong consistency and optimality claims are proven only for the exact posterior under ideal conditions, not for the implemented BMS approximation, and the empirical evidence is a single run per condition with no repeated trials or code. These gaps are load-bearing because the paper's central argument is that the Bayesian approach 'provides guarantees of performance that heuristic approaches lack.'","major_comments":[{"comment":"The sentence 'any alternative that does not coincide with the Bayesian approach in this limit is virtually guaranteed to select the wrong model' is not justified by the preceding argument. Bayesian model selection consistency is known to hold only when the true model is contained in the model space with positive prior probability and under certain regularity conditions; the statement in Section 3a that the prior washes out 'as long as we do not assign p(m_i)=0' ignores the case where the true model is outside the search grammar or has zero prior under the chosen p(m). Without stating these conditions, the universal guarantee is unsubstantiated and should be weakened or qualified.","section":"Section 4, consistency argument"},{"comment":"The consistency and Bayes-optimality arguments in Section 4 apply to the exact posterior p(m_i|D) in Eq. (2.4), but the implemented BMS uses the Laplace-approximated criteria in Eqs. (2.5)-(2.9). The assumptions that the likelihood is sufficiently peaked and the prior is smooth are stated, but the paper does not address model spaces where these fail: redundant parameters, near-collinear terms, and flat likelihood directions can make the Laplace integral dominated by boundaries or saddle points rather than a single interior peak, so the dropped O(1) terms are not negligible. Consequently, the advertised guarantees do not automatically transfer to the implemented BMS; the authors should either prove that the approximation is asymptotically valid for the model classes they consider or explicitly state that the guarantees hold for the exact posterior only.","section":"Section 2, Eqs. (2.5)-(2.9)"},{"comment":"The empirical demonstration consists of one dataset per (N,σ) cell, with no seeds, no repeated trials, and no code or configuration details beyond 'default parameters'. The statement that the example 'is sufficient to prove that the heuristics chosen do not lead to consistent model selection' is therefore too strong: a single run of PySR on a single dataset per condition cannot establish a general failure mode. The authors should provide repeated runs with error bars or at least rephrase these results as illustrative rather than proof.","section":"Section 5, Figs. 1-2"},{"comment":"The assertion that 'the prior effectively being used by the traditional approach is log p_trad = (ki+1)/2 log N + ...' conflates a model-selection penalty with a Bayesian prior. A complexity penalty in a heuristic loss is not a prior over models, and the ellipsis hides the fact that the approximation terms in B1/B2 are part of the marginal likelihood, not a prior. The claim that traditional approaches 'favor a priori models with more, rather than fewer, parameters' is an analogy that holds only for this particular penalty form and should be presented as such, not as an identified prior.","section":"Section 5, Eq. (5.4)"}],"minor_comments":[{"comment":"The coefficients α_o and β_o in the operator prior are not specified numerically; the paper cites previous work [14] but does not state the values or how they were estimated, which makes the prior non-reproducible from this manuscript alone.","section":"Section 5, Eq. (5.1)"},{"comment":"The paper promises 'no ad hoc parameters or thresholds need to be fixed' to balance fit and complexity, but the prior in Eq. (5.1) contains parameters α_o and β_o fitted to a corpus. While these are not balancing parameters in the same sense, the text should acknowledge that the prior still requires a choice of corpus and estimation procedure.","section":"Section 1, desideratum (ii)"},{"comment":"Line 'consider a situation in which one whats to predict' contains a typo: 'whats' should be 'wants'.","section":"Section 6(a)"},{"comment":"The statement that any reasoning system not conforming to Eq. (2.4) 'must violate some of the very basic commonsensical conditions assumed by Cox' is too categorical, given known limitations of Cox's theorem (e.g., requiring real-valued plausibilities and infinite precision). A softer formulation under the standard Cox axioms would be more accurate.","section":"Section 4, Cox argument"},{"comment":"The learnability transition and the claim that no algorithm can identify the true model above a certain noise level are cited to [16] but not derived or illustrated in this manuscript; for a self-contained perspective, a brief informal derivation or a reference to the precise statement in [16] would help.","section":"Section 6(b)"}],"recommendation":"major_revision","confidential_remarks":"This is a perspective-type paper from established authors, and the core derivation is standard. The main risk is that the strong 'guarantees' language can mislead readers into thinking the implemented BMS inherits exact Bayesian optimality, which is not established. The empirical section is too thin for the claims made. The paper also relies heavily on the authors' own prior work [14,15,16] for the key results on learnability and Rashomon sets; the editor may wish to consider whether the contribution is sufficiently novel for a research journal, or whether it would be better framed explicitly as a review/perspective. The revision should either add the missing regularity conditions and empirical rigor, or substantially weaken the guarantee claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a readable, well-argued position paper for Bayesian symbolic regression. The Bayesian framing itself is standard, and the paper is honest about the Laplace approximation in Section 2, but the headline guarantees in Section 4 outrun the implemented B1/B2 criterion.\n\nWhat's actually new: not much in theory. Bayes' theorem, the Laplace approximation/BIC, MDL equivalence, Cox and Dutch book arguments all appear in the cited references, including the authors' own prior work. The two toy experiments (line with noise, constant) are illustrative demos, not evidence. What the paper does well is exposition: it clearly explains why the marginal likelihood, not the point likelihood, should drive model comparison, why priors over models matter, and why the posterior over models rather than a single 'best' equation is the right output. The warning that Pareto-front elbow selection is a heuristic that can systematically fail is worth making.\n\nSoft spots, in order of importance. First, Section 4 says any alternative that does not coincide with the Bayesian approach in the large-N limit is 'virtually guaranteed to select the wrong model.' That is a theorem-level claim with no proof. The consistency argument assumes the true model sits in the search grammar with positive prior probability; the authors acknowledge this in passing (\"as long as we do not assign p(mi)=0 to any model\"), but then the next paragraph claims the prior washes out even if unknown. That only works with positive support. Second, the Laplace approximation conditions in Section 2—sufficiently peaked likelihood and smooth prior—are dropped when Section 4 transfers guarantees to the implemented L(mi,D). For symbolic-regression model spaces with redundant parameters and flat likelihood directions, that approximation can fail, and the B1/B2 scores are not the exact log posterior. Third, Eq. (2.9) is ambiguous: is I(θhat) the full-sample Fisher information or per-observation? That changes the penalty. Fourth, the experiments are single runs, no seeds, no code, no error bars; the constant-model example is a nice illustration of the elbow heuristic failing, but one run cannot establish 'virtually guaranteed' failure.\n\nThe learnability transition is imported from the authors' own prior work [16], which is fine, but the text presents it as if derived here. The paper would be stronger if it explicitly framed Section 4 as claims about the exact Bayesian ideal, and Sections 2–5 as the approximate implementation.\n\nWho this is for: readers wanting a clear conceptual introduction to Bayesian symbolic regression and a physicist-style argument for posterior-based model averaging. It deserves serious peer review as a perspective piece, but the referee should ask for the overclaims to be qualified and the toy experiments to be made reproducible.","headline":"A clear, well-written Bayesian manifesto for symbolic regression whose practical criterion inherits the advertised guarantees only under approximations the paper doesn't verify.","tokens_in":12696,"tokens_out":1962,"would_cite":true,"duration_ms":22119,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Symbolic regression should rank candidate equations by Bayesian plausibility, and that ranking reduces to minimizing a single description-length score that unifies fit, parameter cost, and prior complexity.","keywords":["symbolic regression","Bayesian model selection","minimum description length","Bayesian information criterion","model averaging","learnability transition","Laplace approximation","Rashomon sets"],"falsifier":"Generate data from a model with a flat or multimodal likelihood surface, such as $y=\\theta\\sin(\\omega x)+\\epsilon$ with a broad prior over $\\omega$ and small $N$, and compare the model ranking produced by $L(m_i,D)$ against the exact marginal likelihood computed by numerical quadrature; a substantial mismatch, or frequent selection of a wrong model where the exact posterior favors the true one, would falsify the claim that the Laplace-based criterion inherits the Bayes-optimality guarantees.","tokens_in":11578,"feed_emoji":"🧮","tokens_out":10597,"duration_ms":99948,"temperature":0.7,"pith_summary":"This paper argues that symbolic regression—automatically discovering closed-form equations from data—should be built on the posterior probability of each candidate model rather than on heuristic loss, complexity, and exploration choices. It shows that under Gaussian noise and the Laplace approximation, the practical model-selection criterion is to minimize $L(m_i,D)=B_1(m_i,D)/2-\\log p(m_i)$, a description length combining goodness of fit, a parameter penalty, and an explicit prior over expressions. The authors claim this criterion is consistent, Bayes-optimal, and free of ad hoc trade-offs, and they demonstrate on simple generated datasets that a standard heuristic implementation systematically overfits while the Bayesian criterion recovers the true model, or underfits only when noise is so high that the true model is undetectable. If the argument is right, symbolic regression should output a distribution over models rather than a single best equation, and prediction should be done by averaging over that distribution.","feed_headline":"One Bayesian score could replace symbolic-regression heuristics","feed_subtitle":"A single description-length criterion unifies fit and complexity and beats systematic overfitting in simple tests.","key_machinery":"The load-bearing object is the marginal likelihood $p(D|m_i)=\\int_{\\Theta_i} d\\theta_i\\, p(D|m_i,\\theta_i)p(\\theta_i|m_i)$, approximated by Laplace's method. Its negative logarithm plus the model prior defines the description length $L(m_i,D)$, which the paper interprets as the number of nats needed to convey the model and the data, and equivalently as the energy (information Hamiltonian) of a canonical ensemble of models at inverse temperature $\\beta=1$. The score $B_1/2-\\log p(m_i)$ replaces three heuristic choices of traditional symbolic regression: the loss, the complexity measure, and the Pareto-front elbow selection. The optional $B_2$ term uses the Fisher information matrix to reward models whose predictions are insensitive to small parameter changes, and the prior $p(m_i)$ is made explicit and transparent through a maximum-entropy distribution over operator counts.","core_discovery":"The central claim is that the complete solution to symbolic regression is the posterior distribution $p(m_i|D)\\propto \\exp[-L(m_i,D)]$ over candidate closed-form models, and that this distribution can be computed under explicit approximations. Marginalizing each model's parameters with Laplace's method gives $L(m_i,D)=B_1(m_i,D)/2-\\log p(m_i)$, where $B_1(m_i,D)=-2\\log p(D|m_i,\\hat\\theta_i)+(k_i+1)\\log N$ is the Bayesian information criterion; keeping one more term replaces $B_1$ with $B_2$, which adds $\\log\\det I(\\hat\\theta_i)$, the Fisher information at the maximum-likelihood parameters. The paper claims that this one score integrates goodness of fit, parametric complexity, and structural prior plausibility in a single set of units, so the most plausible model is $\\arg\\min_{m_i} L(m_i,D)$. It further claims that traditional heuristic symbolic regression is implicitly using a prior $\\log p_{\\mathrm{trad}}\\approx \\frac{k_i+1}{2}\\log N$ that favors more complex models, which explains its dramatic overfitting even on a constant model, and that the Bayesian criterion is consistent as $N\\to\\infty$ because the marginal likelihood is extensive while the prior is intensive. Finally, it claims that accounting for uncertainty forces one to consider whole ensembles of models, with important consequences including model averaging and a fundamental learnable-to-unlearnable transition.","pith_inferences":["Editorial inference: the same description-length objective could be grafted onto modern deep-learning symbolic regression by using $L(m_i,D)$ as the training or reward signal for a generator over expressions, turning a heuristic search into an approximately Bayesian sampler; the paper does not test this.","Editorial inference: because the Laplace approximation is the weakest step, comparing $B_1/B_2$ rankings against exact numerical marginal likelihoods on small expression spaces would quantify how often the implemented criterion diverges from the true posterior, and where the stated consistency guarantee actually holds.","Editorial inference: the learnability-transition result suggests a practical scientific diagnostic—reporting the spread of description lengths across the Rashomon set for a dataset would tell researchers whether the data are sufficient to pin down a unique law, rather than trusting one fitted equation."],"forward_implications":["If the Bayesian criterion is adopted, any symbolic regression method minimizing a heuristic loss is implicitly using a prior that rewards complexity; minimizing $L(m_i,D)$ removes the need for ad hoc Pareto-front and elbow choices.","The posterior over models, not the single most plausible equation, is the complete output of symbolic regression; predictions should be made by Bayesian model averaging over posterior samples.","Because the marginal likelihood grows extensively with $N$ while the prior stays intensive, the criterion is consistent: as sample size grows it selects the true model with probability approaching one, as long as no true model has zero prior probability.","There is a finite-sample learnability transition: above a certain noise level the true generating model ceases to be the minimum of the description-length landscape, so no algorithm can recover it from data alone, and near that transition the set of near-optimal models (a Rashomon set) is diverse and mutually inconsistent."],"supporting_citations":[{"why":"Supplies the heuristic genetic-programming symbolic regression implementation used as the comparison baseline in the two experiments.","marker":"[7]"},{"why":"Implements the Bayesian symbolic regression approach, including the maximum-entropy operator-count prior and marginal-likelihood computation.","marker":"[14]"},{"why":"Establishes the learnable-to-unlearnable transition and the Rashomon-set behavior used in the paper's theoretical arguments.","marker":"[16]"},{"why":"Gives the Bayesian information criterion formula that becomes the leading term of the description-length score $L(m_i,D)$.","marker":"[17]"},{"why":"Supplies the Fisher information extension leading to the $B_2$ approximation with the curvature term.","marker":"[18]"},{"why":"Frames the negative log posterior as the minimum description length, tying model selection to data compression.","marker":"[20]"},{"why":"Provides the consistency axioms used to argue that any non-Bayesian assignment of model plausibility is inconsistent.","marker":"[22]"},{"why":"Supplies the Dutch book argument translating non-Bayesian model plausibility into a guaranteed loss.","marker":"[24]"},{"why":"Gives the model-averaging formula used to define prediction from the posterior over models.","marker":"[27]"},{"why":"Provides the term and concept of Rashomon sets used to describe collections of near-optimal models.","marker":"[28]"}],"fun_headline_variants":["Probability, not heuristics, for equation discovery","One score unifies fit and complexity in symbolic regression","Bayesian criterion ends overfitting in symbolic regression","Ensemble models, not a single fit, for symbolic regression"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The practical ranking by $L(m_i,D)$ assumes the Laplace approximation is accurate: for every candidate model, the likelihood must be sharply peaked around its best-fit parameters and the prior smooth there, and when that fails the computed score is not the true log posterior, so the stated consistency and optimality guarantees no longer apply to the implemented method.","fun_headline_variants_meta":{"raw":{"variants":["Probability, not heuristics, for equation discovery","One score unifies fit and complexity in symbolic regression","Bayesian criterion ends overfitting in symbolic regression","Ensemble models, not a single fit, for symbolic regression"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000936,"raw_usage":{"total_tokens":4002,"prompt_tokens":942,"completion_tokens":3060,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":2997}},"tokens_in":558,"tokens_out":3060,"duration_ms":22790,"temperature":1.0,"reasoning_tokens":2997,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:02:45.119127+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate data from a model with a flat or multimodal likelihood surface, such as $y=\\theta\\sin(\\omega x)+\\epsilon$ with a broad prior over $\\omega$ and small $N$, and compare the model ranking produced by $L(m_i,D)$ against the exact marginal likelihood computed by numerical quadrature; a substantial mismatch, or frequent selection of a wrong model where the exact posterior favors the true one, would falsify the claim that the Laplace-based criterion inherits the Bayes-optimality guarantees.","supporting_citations":[{"cited_title":"2020 A Bayesian machine scientist to aid in the solution of challenging scientific problems","cited_arxiv_id":null,"evidence_quote":"Implements the Bayesian symbolic regression approach, including the maximum-entropy operator-count prior and marginal-likelihood computation."},{"cited_title":"2023 Fundamental limits to learning closed-form mathematical models from data","cited_arxiv_id":null,"evidence_quote":"Establishes the learnable-to-unlearnable transition and the Rashomon-set behavior used in the paper's theoretical arguments."},{"cited_title":"1978 Estimating the dimension of a model","cited_arxiv_id":null,"evidence_quote":"Gives the Bayesian information criterion formula that becomes the leading term of the description-length score $L(m_i,D)$."},{"cited_title":"2010 Bayesian model selection and statistical modeling","cited_arxiv_id":null,"evidence_quote":"Supplies the Fisher information extension leading to the $B_2$ approximation with the curvature term."},{"cited_title":"2007 The Minimum Description Length Principle","cited_arxiv_id":null,"evidence_quote":"Frames the negative log posterior as the minimum description length, tying model selection to data compression."},{"cited_title":"1946 Probability, frequency and reasonable expectation","cited_arxiv_id":null,"evidence_quote":"Provides the consistency axioms used to argue that any non-Bayesian assignment of model plausibility is inconsistent."},{"cited_title":"2022 Dutch Book Arguments","cited_arxiv_id":null,"evidence_quote":"Supplies the Dutch book argument translating non-Bayesian model plausibility into a guaranteed loss."},{"cited_title":"1999 Bayesian model averaging: A tutorial","cited_arxiv_id":null,"evidence_quote":"Gives the model-averaging formula used to define prediction from the posterior over models."},{"cited_title":"2019 Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead","cited_arxiv_id":null,"evidence_quote":"Provides the term and concept of Rashomon sets used to describe collections of near-optimal models."}],"review_version":1}