{"id":"02359509-8dcf-43f4-90e1-19534b924236","arxiv_id":"2411.15111","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Comparing fixed and learnable activations in PINNs across five PDEs shows learnable bases help in small networks, destabilize large ones, and low spectral bias does not guarantee accuracy.","lead":"This paper benchmarks seven fixed and learnable activation functions in physics-informed neural networks across five partial differential equations. It finds that the best activation depends on the problem, and that a broader frequency spectrum does not guarantee better accuracy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"A2 KAN failures are observed under one fixed training protocol; without per-model learning-rate/input-normalization checks, the 'curse of functional dimensionality' explanation is not causally supported.","rationale":"The paper is a broad empirical benchmark and, taken as such, its main negative finding—that low spectral bias alone does not guarantee accuracy—is supported by the reported counterexamples: Fourier and Jacobi bases show flat NTK spectra but Hessian instability and poor test errors on several PDEs, while B-splines show steep NTK decay but good accuracy on problems with local features. I found no internal derivation that reduces to a fitted parameter, and the public code and pretrained models are positive evidence for reproducibility. The weakest link is the scaling and causality claim: the A2 failures are dramatic (losses around 1e10 or larger), but they occur under one fixed optimizer schedule designed for MLPs, and 'curse of functional dimensionality' is a label rather than a demonstrated mechanism. This matches the reader's weakest assumption and is the right thing to test. A single additional experiment—per-model learning-rate and input-normalization tuning on the two worst cases—would settle whether the scalability conclusion is intrinsic or a training-budget artifact. If the artifact interpretation wins, the paper's headline should be weakened to 'under a common protocol, X happens' rather than 'due to higher functional dimensionality.' Therefore I keep the reader's CONDITIONAL verdict unchanged; the concern does not overturn the empirical findings, but it does mean the causal explanation should not be accepted without the proposed control experiment.","tokens_in":24499,"tokens_out":7297,"duration_ms":70963,"concrete_test":"Retrain A2 Fourier, Chebyshev, Jacobi, and B-spline KANs on Helmholtz and Wave under a per-model tuning protocol: sweep learning rates {1e-2, 1e-3, 1e-4, 1e-5} with cosine decay or a lower fixed LR, normalize inputs to [-1,1] for Chebyshev/Jacobi and scale Fourier frequencies or GRBF centers to the data range, run 200,000 iterations or early stopping, and repeat with 3–5 seeds. If any of the currently failing A2 models reaches a test error comparable to its A1 counterpart (e.g., under 10% relative L2 on Helmholtz or Wave), then the A2 failures are artifacts of the fixed training budget and the 'curse of functional dimensionality' claim fails; if all remain above 90% or divergent across the sweep, the scalability conclusion is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central scaling conclusion—that learnable activations, especially KAN basis functions, become intractable in A2 due to a 'curse of functional dimensionality'—rests entirely on experiments in which every model is trained with Adam at LR 1e-3 for 60,000 iterations, batch size 128, with no per-basis input normalization and no repeated seeds (Section 5.1). Under this fixed protocol, A2 Fourier, Chebyshev, and Jacobi KANs diverge to physics losses of order 1e10–1e12 (Table 5: Helmholtz, Wave, Klein-Gordon, Convection-Diffusion), while B-spline A2 and GRBF A2 mostly do not. This pattern is exactly what one would expect from learning-rate and input-scale mismatch: high-degree polynomial and Fourier bases have large or unbounded function values and gradients outside their well-conditioned domain, and the paper itself notes that Chebyshev and Jacobi polynomials are sensitive to input scale and that results could be improved with hyperparameter tuning. Since 'curse of functional dimensionality' is never defined or derived, the observed failures do not distinguish an intrinsic scalability defect from a fixed-budget optimization artifact. The claim that low spectral bias does not guarantee accuracy is less affected, but the causal story attached to it—'functional dimensionality' and 'convergence instability' from broad NTK spectra—is tied to the same protocol. Single runs without seeds further weaken the empirical ranking. Thus the load-bearing condition, that the fixed protocol isolates architecture behavior, is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a systematic empirical comparison of fixed and learnable activation functions in physics-informed neural networks (PINNs), covering MLPs with Tanh and parametric-Tanh and KANs with B-spline, GRBF, Fourier, Chebyshev, and Jacobi bases, across five PDEs (Helmholtz, Klein-Gordon, Wave, Convection-diffusion, Cavity). The authors report relative L2 errors, training losses, empirical NTK eigenvalue spectra, and maximum Hessian eigenvalues. They conclude that learnable activations help in smaller architectures but scale poorly in larger ones due to an alleged 'curse of functional dimensionality,' that low spectral bias does not guarantee better accuracy, and that activation function selection remains inherently problem-specific.","tokens_in":24698,"tokens_out":7958,"duration_ms":66510,"significance":"If the empirical findings are robust, the paper provides a valuable counterexample to the common assumption that flatter NTK spectra imply better PINN accuracy, and it offers one of the more extensive comparisons of KAN basis functions on PDEs to date. The public code and Zenodo release support reproducibility. However, the scaling conclusion and the spectral-bias interpretation currently overreach relative to the controlled evidence, so the significance depends on whether the authors can address the training-protocol confounds and the post-hoc selection issues.","major_comments":[{"comment":"The conclusion that learnable activations 'encounter scalability issues in complex networks due to the curse of functional dimensionality' is not supported by the data in Table 4: B-spline(A2) achieves the lowest error on Cavity (4.09%) and 3.03% on Helmholtz, and GRBF(A2) achieves 8.11% on Helmholtz and 14.7% on Wave, so the A2 failures are specific to Fourier, Chebyshev, and Jacobi bases rather than to learnable activations in general. Moreover, all models are trained under a fixed budget (Adam, LR 1e-3, 60,000 iterations, batch 128, no per-basis input normalization; Section 5.1), and the paper itself notes that results could be improved with hyperparameter tuning. The divergences in Table 5 (Lphy ~1e10 to 1e12 for Fourier/Chebyshev/Jacobi A2) are consistent with learning-rate or input-scale mismatch for polynomial/Fourier bases, as the paper acknowledges in Section 3.3. The 'curse of functional dimensionality' is never defined or derived. Please either provide additional experiments (e.g., lower learning rates, input normalization, or grid-size tuning for the failing A2 models) or substantially soften the scalability claim to be about the fixed-protocol behavior of these specific bases.","section":"Section 7 and Section 5.1"},{"comment":"All results are single runs; no variance estimates or multiple seeds are reported. Since the paper ranks models (e.g., 'the lowest velocity error' in Section 5.2.1) and draws qualitative conclusions, the absence of error bars makes the ranking claims statistically unsupported. Please report mean and standard deviation over at least three independent seeds, or explicitly limit claims to the single-run observations.","section":"Tables 4 and 5"},{"comment":"The NTK and Hessian analysis is performed only on 'best-performing models' selected post-hoc from Table 4, and the experiments use only the basis function without the SiLU term (as stated in Section 6). This selection makes the claimed association between broad NTK spectra and convergence instability (Section 7) vulnerable to selection bias; the paper should either compute spectra for all models or present the claim as a case-specific observation. Additionally, Equations (21)-(26) present infinite-width NTK gradient-flow theory, but the experiments use finite-width empirical NTK with Adam and minibatches; the approximation involved should be stated explicitly.","section":"Section 6"},{"comment":"The text states that 'KAN models with A2 architecture exhibit instability and failure to solve the Helmholtz equation,' but Table 4 reports relative L2 errors of 3.03% for B-spline(A2) and 8.11% for GRBF(A2) on Helmholtz. This is a direct contradiction and misleads the reader about the scope of the A2 failures; please correct the narrative to align with the data in Table 4.","section":"Section 5.2.1"}],"minor_comments":[{"comment":"The boundary loss terms are written as norms of sums (e.g., \\|u(t,0)+u(t,1)\\|), which is not equivalent to enforcing both boundary conditions separately; this appears to be a typo and should be corrected for reproducibility.","section":"Equations (8) and (14)"},{"comment":"The Kolmogorov-Arnold representation theorem applies to continuous functions on the unit cube; the domain notation R^n is imprecise and should be refined.","section":"Equation (3)"},{"comment":"The caption 'including the zero eigenvalues' is unclear on a log-scale plot; please clarify how zero eigenvalues are handled in the spectral plots.","section":"Figure 13 caption"},{"comment":"The notation 'weight decay of 10e-6' is unusual; please use '1e-5' for clarity.","section":"Section 5.1"},{"comment":"The text in Section 5.2.3 refers to 'GRBF(A1)' while Table 4 and the figure caption refer to 'GRBF(A2)' for the Wave equation; please reconcile this inconsistency.","section":"Table 4 and Section 5.2.3"},{"comment":"The power iteration method in Equation (28) is standard, but the number of iterations and the convergence criterion used in the experiments are not reported; please provide these details.","section":"Section 6.2"}],"recommendation":"major_revision","confidential_remarks":"The paper self-cites its own Zenodo release, which is appropriate for reproducibility. The main concern is that the central scaling claim is overgeneralized and not causally supported by the controlled experiments; this is fixable with additional experiments or careful reframing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a broad empirical comparison: seven activation/basis families in MLP and KAN PINNs across five PDEs, with NTK spectra and Hessian eigenvalue tracking. That cross-product alone is more systematic than most prior KAN-versus-MLP studies, and the headline observation—flat NTK spectra do not guarantee better test accuracy, with Fourier and Jacobi bases showing broad spectra but poor convergence—is a genuine counterpoint to the spectral-bias folklore. The B-spline result, performing well on localized-feature problems despite high measured spectral bias, is believable and useful for practitioners. Credit where due: the code and pretrained models are public, and the empirical findings are direct measurements, not curve-fits in disguise.\n\nThe soft spots are real but not fatal to the main benchmark. Single runs without error bars weaken every ranking; the top-vs-bottom ordering in Tables 4 and 5 is probably robust only for the large gaps. The A2 scaling story is the weakest part. Fourier, Chebyshev, and Jacobi KANs diverge to physics losses of 1e10–1e12 under one fixed Adam schedule with no per-basis input normalization. The paper itself notes these bases are sensitive to input scale. So the 'curse of functional dimensionality' explanation is not causally supported; a learning-rate or normalization mismatch would produce exactly this pattern. That section needs rework or heavy qualification. The claim that low spectral bias does not guarantee accuracy is much less affected, because it rests on the A1 results and the NTK/Hessian diagnostics, not on the A2 failures.\n\nAlso minor: the FBPINN analogy for B-spline performance is post-hoc interpretation, and the paper's own statements about needing hyperparameter tuning undercut the fixed-budget 'fair comparison'. Still, the central benchmark is sound enough that a careful reader gets a useful map of which activations work where.\n\nWho is this for? Anyone choosing an activation or KAN basis for PINNs on wave-like or convection-dominated problems. It deserves a serious referee—the comparison is valuable and the negative result about NTK spectra is worth publishing, but the scaling conclusion should be reframed as a fixed-protocol observation, not an intrinsic property. I would send it to peer review with a request for repeated runs and a per-model learning-rate sanity check before the A2 claims can stand.","headline":"A useful, honest empirical benchmark of learnable activations and KAN bases for PINNs, with a real cautionary finding about NTK spectra, but the scaling conclusion is over-read from a single fixed training protocol.","tokens_in":25325,"tokens_out":852,"would_cite":true,"duration_ms":10236,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Low spectral bias alone does not guarantee accurate PINN solutions, because broader NTK spectra can bring convergence instability.","keywords":["physics-informed neural networks","Kolmogorov-Arnold networks","learnable activation functions","spectral bias","neural tangent kernel","Hessian eigenvalues","partial differential equations","training stability"],"falsifier":"Retrain the Fourier and Jacobi KANs on Helmholtz, Cavity, and Wave with per-model tuned learning rates, grid sizes, and iteration budgets chosen so their maximum Hessian eigenvalues stay as low as the B-spline model's; if they then match or beat B-spline test errors, the broad-spectrum instability trade-off is not a general law of these architectures.","tokens_in":24197,"feed_emoji":"🧮","tokens_out":8236,"duration_ms":74303,"temperature":0.7,"pith_summary":"This paper asks whether learnable activation functions—parametric Tanh in MLPs and trainable basis functions in KANs—can overcome the two persistent PINN problems: spectral bias, the tendency to learn low-frequency features slowly, and unstable convergence caused by the multi-term physics loss. Comparing seven model variants on five PDEs with different spectral content, the authors find a trade-off between expressivity and training stability. Fourier and Jacobi bases keep the broadest NTK eigenvalue spectra during training yet show the largest Hessian eigenvalues and unstable losses, while B-splines, which show steep spectral decay, still achieve the best errors on Helmholtz and Cavity problems through localized frequency representation. The upshot is that activation choice is inherently problem-specific and low spectral bias is not a reliable proxy for accuracy.","feed_headline":"Low spectral bias is no guarantee of PINN accuracy","feed_subtitle":"Neural PDE solvers must balance frequency reach with training stability; the best activation depends on the equation.","key_machinery":"The central object is the pair of diagnostics applied to the same trained models: the empirical NTK eigenvalue spectrum, whose decay rate quantifies spectral bias, and the maximum Hessian eigenvalue \\(\\lambda_{\\max}\\), estimated by power iteration, which quantifies convergence instability from sharp curvature in the PINN loss. The models under comparison are MLPs with fixed or parametric Tanh and KANs with B-spline, Gaussian RBF, Fourier, Chebyshev, and Jacobi learnable basis functions. All are trained on five PDEs under identical settings, and the two diagnostics are tracked at initialization, mid-training, and the end of training.","core_discovery":"Using the eigenvalue decay of the empirical NTK to measure spectral bias and the maximum Hessian eigenvalue \\(\\lambda_{\\max}\\) to measure convergence stability, the paper shows that these two diagnostics pull in opposite directions. Fourier and Jacobi KAN bases maintain broad NTK spectra throughout training—evidence of high-frequency representational capacity—yet their \\(\\lambda_{\\max}\\) values are large and fluctuating, losses oscillate, and test errors are high or even diverge on several equations. B-splines, by contrast, exhibit rapid NTK eigenvalue decay, indicating high spectral bias, but they obtain the lowest errors on Helmholtz and Cavity because their local support confines high-frequency content to small regions, similar to domain decomposition. The paper concludes that low spectral bias alone does not guarantee better accuracy and that activation selection remains inherently problem-specific.","pith_inferences":["Under the paper's fixed training budget, per-architecture tuning of learning rate, grid size, and iterations remains untested; if such tuning rescued the larger Fourier or Jacobi KANs, the 'curse of functional dimensionality' would need to be reinterpreted as an optimization-budget artifact.","Because B-spline local support mimics finite-basis domain decomposition, pairing B-spline KAN layers with multigrid-style coarse-to-fine schedules is a natural next experiment for multi-scale PDEs.","The diagnostics suggest a cheap selection heuristic: after a few hundred training steps, measure NTK flatness and \\(\\lambda_{\\max}\\) on the physics residual and pick the activation with the best stability-adjusted bandwidth; the paper's data imply B-spline or Tanh would win on most of the tested equations.","Extending the comparison to inverse problems and operator learning would test whether the expressivity-stability trade-off transfers beyond forward PDE solving, which the paper does not address."],"forward_implications":["Flat NTK spectra should not be used alone as a design target for PINN accuracy; frequency reach must be balanced against training stability.","B-spline KANs can outperform Fourier KANs on oscillatory and boundary-layer problems despite higher measured spectral bias, because local support plays the role of domain decomposition.","Learnable activations that work well in small networks can destabilize in larger ones; scaling KAN width and depth raises optimization difficulty faster than expressivity helps.","Activation selection should follow the PDE's spectral character: Tanh-type smooth activations for wave-like problems, locally supported bases for sharp gradients, and cosines only when global periodicity and stable training align.","Hessian diagnostics expose that the physics-loss term stays ill-conditioned across all activations, so activation changes alone will not resolve the optimizer's burden."],"supporting_citations":[{"why":"Supplies the PINN formulation (physics loss plus boundary and initial losses) that every model in the study optimizes.","marker":"[1]"},{"why":"Introduces KANs with B-spline learnable basis functions, the architecture class the paper generalizes to other bases.","marker":"[16]"},{"why":"Establishes spectral bias—low-frequency learning dominance—which the paper measures via NTK spectra.","marker":"[10]"},{"why":"Provides the NTK-based training dynamics for PINNs that motivates the eigenvalue-decay analysis.","marker":"[61]"},{"why":"Gives the NTK perspective on PINN training failure that the Hessian analysis is designed to complement.","marker":"[62]"},{"why":"Supplies the NTK convergence theory linking eigenvalue decay to exponential learning rates.","marker":"[67]"},{"why":"Shows Fourier feature mappings widen the NTK spectrum, the expectation the paper tests and complicates.","marker":"[69]"},{"why":"Demonstrates adaptive activation functions accelerate PINN convergence, the class of methods under comparison.","marker":"[33]"},{"why":"Domain-decomposition FBPINNs are used to explain why B-spline local support captures local oscillations.","marker":"[14]"}],"fun_headline_variants":["PINN accuracy isn't just about spectral bias","Expressivity vs stability: the PINN trade-off","Why broad frequency reach can backfire in PINNs","Activation functions: the real PINN performance lever"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The scaling conclusion rests on the assumption that training every model with the same learning rate, 60,000 iterations, batch size, and no per-KAN tuning is a fair comparison that isolates architecture behavior; if larger KANs simply need lower learning rates or longer schedules, the reported failures could be an artifact of the fixed budget.","fun_headline_variants_meta":{"raw":{"variants":["PINN accuracy isn't just about spectral bias","Expressivity vs stability: the PINN trade-off","Why broad frequency reach can backfire in PINNs","Activation functions: the real PINN performance lever"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00026,"raw_usage":{"total_tokens":1603,"prompt_tokens":975,"completion_tokens":628,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":566}},"tokens_in":591,"tokens_out":628,"duration_ms":6595,"temperature":1.0,"reasoning_tokens":566,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:29:03.038880+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the Fourier and Jacobi KANs on Helmholtz, Cavity, and Wave with per-model tuned learning rates, grid sizes, and iteration budgets chosen so their maximum Hessian eigenvalues stay as low as the B-spline model's; if they then match or beat B-spline test errors, the broad-spectrum instability trade-off is not a general law of these architectures.","supporting_citations":[{"cited_title":"Understanding and mitigating gradient flow pathologies in physics-informed neural networks.SIAM Journal on Scientific Computing, 43(5):A3055–A3081, 2021","cited_arxiv_id":null,"evidence_quote":"Provides the NTK-based training dynamics for PINNs that motivates the eigenvalue-decay analysis."}],"review_version":1}