Recognition: no theorem link
SymTorch: Symbolic Distillation of Neural Networks
Pith reviewed 2026-05-15 19:35 UTC · model grok-4.3
The pith
SymTorch distills neural network components into closed-form symbolic expressions that recover physical laws and enable faster hybrid models.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
By feeding neural network components into symbolic regression, SymTorch recovers the exact closed-form functions those components implement. This produces interpretable expressions for pairwise forces in graph networks, full PDE and ODE solutions including constants in physics-informed networks, and the governing equations of chaotic systems like Lorenz, while also supporting hybrid models that replace selected transformer layers with symbolic surrogates for efficiency gains.
What carries the argument
Symbolic distillation, the process of using symbolic regression to replace neural sub-networks with equivalent closed-form mathematical expressions.
If this is right
- Pairwise interaction forces are recovered from graph neural networks trained on n-body observations.
- Exact closed-form PDE and ODE solutions, including constants, are distilled from physics-informed neural networks trained on sparse data.
- Chaotic Lorenz dynamics are uncovered from high-dimensional data, and the symbolic form outperforms the base network on prediction tasks.
- A symbolic extension of LIME produces more accurate local explanations than standard LIME across classification and regression benchmarks.
- Replacing one to seven transformer MLP layers with symbolic approximations yields 2-19 percent throughput gains and up to 18.7 percent VRAM reduction while staying on the Pareto front for perplexity.
Where Pith is reading between the lines
- Accurate symbolic forms could let researchers check whether a network truly learned the intended physics rather than an approximation that breaks under extrapolation.
- Applying the same distillation to networks trained on real experimental data from poorly understood systems might surface candidate new physical laws.
- Hybrid symbolic-neural architectures could improve robustness on out-of-distribution inputs because the symbolic portions enforce exact functional relationships.
- The approach might be extended to compress entire models into compact symbolic programs suitable for deployment on low-resource hardware.
Load-bearing premise
The symbolic regression step finds expressions whose behavior matches the neural network closely enough that physical interpretations and performance claims remain valid.
What would settle it
If the distilled symbolic expression produces predictions that diverge substantially from the original neural network on held-out inputs, or if recovered physical constants differ from known values, the claim that the expressions faithfully represent the learned functions would be falsified.
read the original abstract
What mathematical functions do neural network components learn? Symbolic distillation addresses this question by expressing neural network components with interpretable, closed-form mathematical expressions that expose the functional structure learned during training. We develop symbolic distillation as a systematic, architecture-agnostic methodology, and release our approach as the open-source SymTorch package - a PySR-powered library built natively for the PyTorch ecosystem. Applying this methodology across diverse architectures, we find that SymTorch is successful in the automated discovery of physical laws. Specifically, our approach (1) recovers pairwise interaction forces from graph neural networks trained on empirical $n$-body observations, (2) distills the exact closed-form PDE/ODE solutions of multiple physical systems, including the value of constants, from physics-informed neural networks trained on sparse data, and (3) uncovers the chaotic dynamics of the Lorenz system from high-dimensional data, ultimately outperforming the base neural network on downstream prediction tasks. We further demonstrate the utility of our framework for model interpretability by providing an optimized implementation of SLIME - a symbolic extension to the LIME explainability method. SLIME consistently outperforms LIME across predictive metrics across eight popular classification and regression benchmarks, while still providing an interpretable local symbolic model. Lastly, we investigate replacing transformer MLP layers with symbolic surrogates: replacing 1-7 layers with symbolic approximations yields 2-19\% throughput improvements and up to 18.7\% VRAM reduction, with the resulting hybrid models lying on the Pareto front of throughput versus perplexity among open-source LLMs of comparable scale.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces SymTorch, a PySR-powered open-source PyTorch library for symbolic distillation of neural network components into closed-form mathematical expressions. It claims to recover pairwise interaction forces from graph neural networks on n-body data, distill exact closed-form PDE/ODE solutions including constants from physics-informed networks on sparse data, uncover Lorenz chaotic dynamics while outperforming the base network on prediction tasks, provide an optimized SLIME variant that outperforms LIME on eight classification/regression benchmarks, and yield hybrid transformer models by replacing 1-7 MLP layers with symbolic surrogates for 2-19% throughput gains and up to 18.7% VRAM reduction.
Significance. If the symbolic expressions are shown to be high-fidelity functional approximations to the trained networks, the work provides a practical, architecture-agnostic tool for interpretability in scientific machine learning and efficiency gains in large models, with potential to extract physical insights from black-box networks.
major comments (3)
- [Abstract] Abstract and results sections: the central claims of 'exact' recovery of forces, PDE/ODE solutions, and Lorenz dynamics, as well as outperformance, rest on the assumption that PySR expressions faithfully match network outputs, yet no quantitative fidelity metrics (e.g., max deviation, relative L2 error, or integrated error on held-out grids) are reported between the original network predictions and the distilled symbolic forms.
- [Results on Lorenz and transformers] Lorenz and hybrid transformer experiments: without pointwise or aggregate error statistics between NN outputs and symbolic surrogates, the reported outperformance on prediction tasks and Pareto-front placement could arise from incomplete network optimization or exploitation of approximation discrepancies rather than true distillation.
- [Methods and applications] SLIME and GNN/PINN demonstrations: the manuscript provides no ablation on how the choice of PySR hyperparameters or data sampling density affects recovery accuracy, leaving the robustness of the 'exact' closed-form recoveries (including constants) unverified.
minor comments (2)
- Clarify the precise PySR configuration, loss functions, and data-generation procedure used for each distillation task.
- Add implementation details on how symbolic surrogates are substituted into transformer MLP layers without retraining the rest of the model.
Simulated Author's Rebuttal
We thank the referee for the constructive comments, which highlight important aspects of quantitative validation and robustness. We agree that the manuscript would benefit from explicit fidelity metrics and ablations to strengthen the claims of high-fidelity distillation. We will revise the paper accordingly, as detailed in the point-by-point responses below.
read point-by-point responses
-
Referee: [Abstract] Abstract and results sections: the central claims of 'exact' recovery of forces, PDE/ODE solutions, and Lorenz dynamics, as well as outperformance, rest on the assumption that PySR expressions faithfully match network outputs, yet no quantitative fidelity metrics (e.g., max deviation, relative L2 error, or integrated error on held-out grids) are reported between the original network predictions and the distilled symbolic forms.
Authors: We agree that quantitative fidelity metrics are essential to rigorously support the claims of exact or high-fidelity recovery. In the revised manuscript, we will add a dedicated subsection (and corresponding tables in the results) reporting max absolute deviation, relative L2 error, and integrated error over held-out grids or test trajectories for the GNN force recovery, PINN PDE/ODE solutions, and Lorenz dynamics. These metrics will be computed directly between the original network outputs and the PySR symbolic expressions on unseen data. revision: yes
-
Referee: [Results on Lorenz and transformers] Lorenz and hybrid transformer experiments: without pointwise or aggregate error statistics between NN outputs and symbolic surrogates, the reported outperformance on prediction tasks and Pareto-front placement could arise from incomplete network optimization or exploitation of approximation discrepancies rather than true distillation.
Authors: We acknowledge the need for explicit error statistics to confirm that outperformance arises from true distillation rather than optimization artifacts. The revised version will include pointwise and aggregate error metrics (MSE, max error, and relative error) between the base network outputs and the symbolic surrogates for both the Lorenz prediction tasks and the hybrid transformer layers. We will also add details confirming that the base networks were trained to convergence with standard optimizers and early stopping, and that the symbolic forms improve extrapolation on held-out chaotic regimes. revision: yes
-
Referee: [Methods and applications] SLIME and GNN/PINN demonstrations: the manuscript provides no ablation on how the choice of PySR hyperparameters or data sampling density affects recovery accuracy, leaving the robustness of the 'exact' closed-form recoveries (including constants) unverified.
Authors: We agree that ablations on PySR hyperparameters and sampling density would better verify robustness. In the revised manuscript, we will add an ablation study (in the methods or appendix) systematically varying key PySR parameters (population size, number of iterations, parsimony coefficient) and data sampling densities for the GNN, PINN, and SLIME experiments. Results will show that the recovered expressions, including constants, remain stable and accurate across these variations, with only minor degradation at extreme low-density settings. revision: yes
Circularity Check
No significant circularity; empirical applications of external symbolic regression tool
full rationale
The paper presents SymTorch as a PySR-based methodology for distilling neural network components into symbolic expressions across several empirical case studies (n-body forces, PINN solutions, Lorenz dynamics, and hybrid transformers). No load-bearing derivation chain exists that reduces predictions or results to fitted parameters by construction, self-definitional loops, or self-citation chains. The reported recoveries and performance gains are outcomes of applying an independent symbolic regression procedure to data generated from trained networks; these outcomes remain falsifiable against held-out network evaluations and do not tautologically follow from the paper's own equations or prior author work. Self-citation to PySR is present but functions as tool usage rather than the sole justification for any central claim.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Symbolic regression via PySR can recover the functional form learned by a neural network component
Forward citations
Cited by 1 Pith paper
-
The finite expression method for turbulent dynamics with high-order moment recovery
A two-stage symbolic regression plus generative model framework recovers governing interaction terms and forcing in stochastic triad models while accurately predicting statistical moments up to order five.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.