Pith. sign in

REVIEW 3 major objections 5 minor 29 references

Dimension Reduction for Symbolic Regression

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Prepending a substitution-finding beam search to symbolic regression significantly improves the recovery of formulas from finite samples.

desk verdict A useful, well-engineered extension of AIFeynman's substitution idea that shows real recovery gains, but the reliability claim outruns the evidence because the pipeline's false-positive rate is never measured. read the letter →

arxiv 2506.19537 v1 pith:UNB6U42Q submitted 2025-06-24 cs.LG

Paul Kahlmeyer , Markus Fischer , Joachim Giesen This is my paper · ORCID
classification cs.LG
keywords symbolicregressiondimensionreductionfunctionaldependencesubstitutionsearchbeamexpressionDAGCODECKMAC
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Many natural formulas contain variables that appear only in fixed combinations, and symbolic regression algorithms lose their ability to recover such formulas as complexity grows. This paper proposes finding those combinations automatically by enumerating small candidate expressions and testing each candidate with a functional-dependence score; once a combination is confirmed, it can be replaced by a single new variable. The substitution search is organized as a beam search and can be prepended to any symbolic regression algorithm. The paper reports that this preprocessing reduces the number of variables by roughly half on the Feynman benchmark and by about a third on Wikipedia's eponymous equations, and that it substantially raises recovery rates for several very different regressors.

What carries the argument

The mechanism that carries the argument is an iterative substitution search over small expression DAGs, validated by a functional-dependence score. A candidate substitution transforms the original observations into a lower-dimensional data set; if the score for the transformed data is near 1, the candidate is treated as a valid reduction and becomes a node in a beam search. Two substitution types are used: input substitutions $g(x_I)$, which compress several input variables into one, and out-input substitutions $h(x_I, y)$, which create an auxiliary regression problem that can expose more complex input substitutions. The final formula is reconstructed by solving the transformed equations symbolically with a computer algebra system.

What would settle it

Take a ground-truth formula that contains no small substitution, sample it with increasing noise and decreasing sample size, and run the beam search. If the procedure still reports large reduction rates and high functional-dependence scores for candidate substitutions, then the validity test is accepting false substitutions and the method would send the regressor a reduced problem whose reconstruction does not equal the true formula.

Watch

Extended reading notes

Core claim

The central claim is that symbolic recovery performance is significantly improved by reducing the dimension of a regression problem before handing it to a symbolic regressor. The reduction is achieved by finding valid substitutions: a function $g$ of a subset of the input variables (input substitution), or a function $h$ of some inputs together with the output (out-input substitution), that leaves the data functionally dependent. Candidate substitutions are generated as small expression DAGs, and a functional-dependence measure such as CODEC decides validity. The best-scoring substitutions are followed in a beam search, and a computer algebra system reconstructs the original formula from the substitutions on the chosen path. On the Feynman and Wikipedia eponymous equations benchmarks, the method cuts the variable count by about 50% and 35% respectively and improves recovery for all tested regressors; the authors conclude that combining this beam search with state-of-the-art symbolic regression algorithms significantly boosts their symbolic recovery performance.

Load-bearing premise

The method assumes that a functional-dependence score close to 1 on the finite, noisy sample reliably means the candidate substitution is a true substitution, so the reduced problem is equivalent to the original one and the regressor is not being asked to solve a different problem.

Editorial extensions

If this is right

  • Any symbolic regression algorithm can be preceded by this substitution-finding beam search without changing the algorithm itself, so the gain in recovery is portable across regressor families.
  • Benchmark formulas with known ground truth can be reduced by roughly half their variables, which makes the remaining regression problem easier for search, sampling, and genetic-programming methods alike.
  • Even when a formula is not fully recovered, the returned expression shares more subexpressions with the ground truth when the beam search is used, so the method improves partial interpretability, not just exact recovery.
  • The out-input substitution mechanism finds reductions that the four AIFeynman predicates miss, particularly on the Wikipedia data set, so the method covers a broader class of simplifications than previous predicate-based approaches.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same reduction could serve as a feature-construction step for ordinary (non-symbolic) regression, turning discovered substitutions into interpretable derived variables; this is a natural extension the paper does not test.
  • Since the validity test is a statistical score on a finite sample, high noise or small samples could make the beam search commit to a substitution that is not a true symbolic identity; a conservative score threshold or a verification step on held-out data would address that risk.
  • A testable extension would be to enlarge the DAG search space beyond one intermediary and one output node; the paper keeps this space small for efficiency, and larger budgets might yield even higher reduction rates at higher computational cost.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a dimension-reduction preprocessing step for symbolic regression. Candidate substitutions (input substitutions and out-input substitutions) are enumerated as small expression DAGs, scored with functional dependence measures (CODEC, KMAC), and organized in a beam search. The path with the highest-scoring reduced regression problem is then handed to a downstream symbolic regression algorithm, and the recovered expression is mapped back to the original variables. Experiments on the Wikipedia eponymous equations and Feynman benchmark suites report reduction rates around 34-49% and improved recovery rates for UDFS, DSR, TRANSF, PySR, OPERON, and polynomial regression.

Significance. If the central claim holds, the method is a broadly applicable plug-in preprocessing that improves symbolic recovery for many algorithm families, which would be a useful contribution. The paper is clearly written, the code repository is provided, and the empirical evaluation covers several conceptually different symbolic regressors and two established benchmark suites. The recovery gains are consistent across algorithms. However, the evaluation does not measure the false-positive rate of the unlabeled substitution test, and the recovery comparison includes extra model selection in the beam-search condition. The claim that the method 'reliably identifies valid substitutions' is therefore not yet supported by the experiments as reported.

major comments (3)
  1. [Section 4 and Section 5 (Reduction rate)] The deployed pipeline and the reduction-rate evaluation use different validity criteria. Section 4 states that the regression problems on the path leading to the highest-scoring node are passed to a symbolic regression algorithm, with no rejection threshold on the functional dependence score. Section 5, however, computes the reduction rate by running the beam search and then checking the selected substitutions with SymPy against the known ground truth; the metric records the best dimension reduction obtained after this oracle check. This mismatch means the reported reduction rates measure what a perfect validity oracle could certify, not how often the unlabeled dependence score actually selects a valid substitution. Since the beam size is 1 and a path is always selected, invalid substitutions are not an edge case but the default operating point whenever no valid substitution exists. Please report the precision and recall of the dependence score as a validity classifier at the operating point used by the beam search, including problems that admit no valid substitution, and either specify a threshold or justify why rank-based selection is sufficient.
  2. [Section 4 and Supplementary Section A] The recovery comparison conflates dimension reduction with additional model search. In the beam-search condition, the downstream symbolic regressor is run on each problem along the selected path, and the expression with the best model fit on the original data is returned (Supplementary Section A). The baseline, in contrast, runs the regressor once on the original problem. Multiple independent runs followed by selection on held-out fit can improve recovery by themselves, so the gains in Table 4 may be partly attributable to extra candidate expressions rather than to valid substitutions. Please add a control that gives the baseline the same number of independent regressor runs (or the same compute budget) with the same selection rule, and report the number of regressor calls per problem in both conditions.
  3. [Section 3 and Section 5] The finite-sample reliability of the functional dependence measures at the chosen operating point is not established. Section 3 presents asymptotic guarantees for Chatterjee's coefficient, CODEC, and KMAC, but the implemented validity test is a finite-sample score used only to rank candidates. The paper does not report score distributions for valid versus invalid substitutions under the sample sizes and noise levels of Section 5, nor does it show how the score behaves when no valid substitution exists. Without such calibration, the abstract's claim that the method 'reliably identifies valid substitutions' remains a statement about asymptotic behavior rather than about the implemented procedure. A calibration experiment (score histograms or ROC-style curves for true and false candidates) would directly address this and would also inform the threshold question raised above.
minor comments (5)
  1. [Section 1] The complexity count for Washburn's formula is inconsistent: the text says the expression tree has 13 nodes, but later says the simplified tree is reduced from 12 to 9 nodes; please reconcile these numbers.
  2. [Section 4] The phrase 'in-output substitutions h' appears to be a typo and should read 'out-input substitutions h' to match the terminology used elsewhere.
  3. [Section 5] Tables 2 and 3 do not state the noise level at which the reduction rates are computed; please specify whether these results are for the noise-free setting or a particular noise level.
  4. [Supplementary Section A] The termination criterion of the beam search is not fully specified; please state the maximum number of levels or the stopping condition used in the experiments.
  5. [Figure 3] The x-axis labels for the noise levels are cramped and hard to read; consider using a log-spaced axis with clearer tick labels.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central recovery claim is tested with external regressors on ground-truth benchmarks, and the self-citations are tool reuse rather than load-bearing premises.

full rationale

The paper's main claim, that the beam-search dimension reduction boosts symbolic recovery, is an empirical claim evaluated in Section 5 by running established, independent regressors (DSR, TRANSF, PySR, OPERON) with and without the beam search on the Feynman and Wikipedia benchmark sets. The recovery rate is measured against ground-truth expressions via SymPy, so the result is not forced by the construction of the method. The validity of substitutions in the reduction-rate experiment is likewise checked independently with SymPy rather than with the functional-dependence score. Self-citation of Kahlmeyer et al. (2024) appears as the DAG-enumeration mechanism and as the UDFS baseline; this is code/tool reuse and does not carry the argument's evidential weight, especially since the method improves external algorithms as well. A mild methodological concern is that the beam-size hyperparameter was selected on these same benchmarks, and the reduction-rate metric discards false-positive substitutions instead of measuring their frequency; but these are evaluation limitations, not cases where a prediction reduces by construction to its inputs. No equation in the paper defines the output in terms of the input, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The ledger is small: two search hyperparameters, background results on dependence measures, and the empirical premise about naturally occurring formulas. No new physical entities or fitted model parameters are introduced.

free parameters (2)
  • beam size = 1
    Selected via parameter study on the same two benchmark datasets; larger sizes give only marginal improvement.
  • DAG search space budget = one intermediary node and one output node
    Chosen for efficiency; restricts candidate substitutions to small scalar functions.
assumptions (4)
  • domain assumption The observations arise from a functional dependence y=f(x)
    Fundamental to regression; stated in Section 3 and implicitly in Section 1.
  • standard math CODEC and KMAC converge to 1 for functional dependence and to 0 for independence
    Proven in the cited works (Azadkia and Chatterjee 2021; Deb et al. 2020); relied upon for substitution validity.
  • domain assumption Natural formulas often contain variables in fixed combinations
    Motivates the whole approach; supported empirically by the benchmark datasets and by Guimera et al. 2020.
  • domain assumption SymPy's symbolic simplification and equivalence checks are correct
    Used for validating substitutions in the reduction-rate experiment and for recovering f_g from out-input substitutions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dimension Reduction for Symbolic Regression." pith.science (2026). https://pith.science/paper/UNB6U42Q

@misc{pith2026250619537,
  author       = {Pith},
  title        = {Pith review of: Dimension Reduction for Symbolic Regression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UNB6U42Q}},
  note         = {Machine review of arXiv:2506.19537}
}
read the original abstract

Solutions of symbolic regression problems are expressions that are composed of input variables and operators from a finite set of function symbols. One measure for evaluating symbolic regression algorithms is their ability to recover formulae, up to symbolic equivalence, from finite samples. Not unexpectedly, the recovery problem becomes harder when the formula gets more complex, that is, when the number of variables and operators gets larger. Variables in naturally occurring symbolic formulas often appear only in fixed combinations. This can be exploited in symbolic regression by substituting one new variable for the combination, effectively reducing the number of variables. However, finding valid substitutions is challenging. Here, we address this challenge by searching over the expression space of small substitutions and testing for validity. The validity test is reduced to a test of functional dependence. The resulting iterative dimension reduction procedure can be used with any symbolic regression approach. We show that it reliably identifies valid substitutions and significantly boosts the performance of different types of state-of-the-art symbolic regression algorithms.

Figures

Figures reproduced from arXiv: 2506.19537 by the authors.

Figure 1
Figure 1. Fraction of successfully recovered expressions [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the beam search approach for sym [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Average reduction rates of our beam search, with [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Expression tree corresponding to Washburns formula. If we take the gray node as root of a subtree, we get a subex [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 26 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Azadkia, M.; and Chatterjee, S. 2021. A simple measure of conditional dependence . The Annals of Statistics, 49(6): 3070 -- 3102

    work page 2021

  4. [4]

    J.; Desmond, H.; and Ferreira, P

    Bartlett, D. J.; Desmond, H.; and Ferreira, P. G. 2023. Exhaustive symbolic regression. IEEE Transactions on Evolutionary Computation

    work page 2023

  5. [5]

    Burlacu, B.; Kronberger, G.; and Kommenda, M. 2020. Operon C++: An Efficient Genetic Programming Framework for Symbolic Regression. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, GECCO '20, 1562–1570

    work page 2020

  6. [6]

    Chatterjee, S. 2021. A New Coefficient of Correlation. Journal of the American Statistical Association, 116(536): 2009--2022

    work page 2021

  7. [7]

    Cranmer, M. 2023. Interpretable Machine Learning for Science with PySR and SymbolicRegression.jl

    work page 2023

  8. [8]

    Deb, N.; Ghosal, P.; and Sen, B. 2020. Measuring Association on Topological Spaces Using Kernels and Geometric Graphs

    work page 2020

Show all 29 references
  1. [9]

    P.; Leighton, R

    Feynman, R. P.; Leighton, R. B.; and Sands, M. 2011. The Feynman Lectures on Physics, Vol. II: The New Millennium Edition: Mainly Electromagnetism and Matter. The Feynman Lectures on Physics. Basic Books

    2011

  2. [10]

    A.; Miranda, M.; Pallarès, J.; and Sales-Pardo, M

    Guimerà, R.; Reichardt, I.; Aguilar-Mogas, A.; Massucci, F. A.; Miranda, M.; Pallarès, J.; and Sales-Pardo, M. 2020. A Bayesian machine scientist to aid in the solution of challenging scientific problems. Science Advances, 6(5): eaav6971

    2020

  3. [11]

    Holland, J. H. 1975. Adaptation in Natural and Artificial Systems. University of Michigan Press. Second edition, 1992

    1975

  4. [12]

    Jin, Y.; Fu, W.; Kang, J.; Guo, J.; and Guo, J. 2020. Bayesian Symbolic Regression. arXiv:1910.08892

    2020 arXiv

  5. [13]

    Kahlmeyer, P.; Giesen, J.; Habeck, M.; and Voigt, H. 2024. Scaling Up Unbiased Search-based Symbolic Regression. In Larson, K., ed., Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24 , 4264--4272. International Joint Conference...

    2024

  6. [14]

    Kamienny, P.-A.; d'Ascoli, S.; Lample, G.; and Charton, F. 2022. End-to-end Symbolic Regression with Transformers. In Oh, A. H.; Agarwal, A.; Belgrave, D.; and Cho, K., eds., Advances in Neural Information Processing Systems

    2022

  7. [15]

    M.; Kommenda, M.; and Affenzeller, M

    Kammerer, L.; Kronberger, G.; Burlacu, B.; Winkler, S. M.; Kommenda, M.; and Affenzeller, M. 2020. Symbolic regression by exhaustive search: Reducing the search space using syntactical constraints and efficient semantic structure deduplication. Genetic programming theory and p...

    2020

  8. [16]

    Kommenda, M.; Burlacu, B.; Kronberger, G.; and Affenzeller, M. 2020. Parameter identification for symbolic regression using nonlinear least squares. Genetic Programming and Evolvable Machines, 21(3): 471--501

    2020

  9. [17]

    Koza, J. R. 1994. Genetic programming as a means for programming computers by natural selection. Statistics and computing, 4: 87--112

    1994

  10. [18]

    La Cava, W.; Orzechowski, P.; Burlacu, B.; de Franca, F.; Virgolin, M.; Jin, Y.; Kommenda, M.; and Moore, J. 2021. Contemporary Symbolic Regression Methods and their Relative Performance. In Vanschoren, J.; and Yeung, S., eds., Proceedings of the Neural Information Processing ...

    2021

  11. [19]

    La Cava, W.; Spector, L.; and Danai, K. 2016. Epsilon-Lexicase Selection for Regression. In Proceedings of the Genetic and Evolutionary Computation Conference 2016, GECCO '16, 741–748. New York, NY, USA: Association for Computing Machinery. ISBN 9781450342063

    2016

  12. [20]

    P.; Paprocki, M.; C ert\' i k, O.; Kirpichev, S

    Meurer, A.; Smith, C. P.; Paprocki, M.; C ert\' i k, O.; Kirpichev, S. B.; Rocklin, M.; Kumar, A.; Ivanov, S.; Moore, J. K.; Singh, S.; Rathnayake, T.; Vig, S.; Granger, B. E.; Muller, R. P.; Bonazzi, F.; Gupta, H.; Vats, S.; Johansson, F.; Pedregosa, F.; Curry, M. J.; Terrel,...

    2017

  13. [21]

    Mundhenk, T.; Landajuela, M.; Glatt, R.; Santiago, C.; Faissol, D.; and Petersen, B. 2021. Symbolic Regression via Neural-Guided Genetic Programming Population Seeding

    2021

  14. [22]

    Pearson, K. 1920. Notes on the history of correlation . Biometrika, 13(1): 25--45

    1920

  15. [23]

    K.; Larma, M

    Petersen, B. K.; Larma, M. L.; Mundhenk, T. N.; Santiago, C. P.; Kim, S. K.; and Kim, J. T. 2021. Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. In International Conference on Learning Representations

    2021

  16. [24]

    Schmidt, M.; and Lipson, H. 2009. Distilling Free-Form Natural Laws from Experimental Data. Science, 324(5923): 81--85

    2009

  17. [25]

    Spearman, C. 1904. The proof and measurement of association between two things. The American journal of psychology, 15(1): 72--101

    1904

  18. [26]

    Stephens, T. 2016. Genetic Programming in Python, with a scikit-learn inspired API: gplearn

    2016

  19. [27]

    Udrescu, S.-M.; Tan, A.; Feng, J.; Neto, O.; Wu, T.; and Tegmark, M. 2020. AI Feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity. In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M.; and Lin, H., eds., Advances in Neural Information Processing Syst...

    2020

  20. [28]

    Udrescu, S.-M.; and Tegmark, M. 2020. AI Feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16): eaay2631

    2020

  21. [29]

    Virgolin, M.; Alderliesten, T.; Witteveen, C.; and Bosman, P. A. 2021. Improving model-based genetic programming for symbolic regression of small expressions. Evolutionary computation, 29(2): 211--237

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.