{"id":"7a3108ce-fe8d-4160-9ec8-ad95bc3353c4","arxiv_id":"2412.09752","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new forward-pass algorithm computes exact n-th derivatives of deep networks in time linear in network size, avoiding repeated autodifferentiation.","lead":"This paper defines a faster way to compute high-order derivatives of neural networks, using a classical formula to pass derivative values through the network in one forward sweep. The method could make physics-informed neural networks practical for differential equations that need many derivatives.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section III-A's exponential baseline for autodifferentiation is not the state of the art; Taylor-mode forward AD computes n-th input derivatives in polynomial time, so the central exponential-to-quasilinear contrast collapses.","rationale":"The reader's weakest_assumption identifies precisely this issue: the claimed O(M^n) cost of autodifferentiation is not proven and is contradicted by Taylor-mode AD. My stress-test confirms this is the most load-bearing concern. The paper's derivation of the Faà di Bruno recurrence appears mathematically sound, and the empirical PINN results are plausible, but the central significance claim—that n-TangentProp converts 'exponential' autodiff into 'quasilinear'—depends entirely on the baseline being exponential. Since Taylor-mode AD already computes input derivatives in polynomial time, the paper's asymptotic comparison is against a strawman, and its own runtime is actually worse than polynomial in n. This means the paper's theoretical framing is incorrect, though its engineering contribution (single forward pass, O(nM) memory, speedups over naive PyTorch autodiff) could still be valuable after major revision. The reader's verdict of CONDITIONAL already incorporates this concern; my analysis strengthens it but does not change the verdict. The concrete test—benchmarking against JAX jet—would settle whether the claimed advantage exists in practice and would guide the required revision.","tokens_in":13538,"tokens_out":7936,"duration_ms":84184,"concrete_test":"Benchmark n-TangentProp against Taylor-mode AD (e.g., JAX's jax.jet, or a custom univariate jet propagation) on the same feed-forward networks and derivative orders used in Figures 1 and 4 (n = 3, 5, 7, 9, 12), measuring wall-clock time and peak memory for both forward and backward passes. If Taylor-mode completes in polynomial time and is faster than n-TangentProp for n ≥ 9, the claimed exponential-to-quasilinear advantage is refuted. Additionally, tabulate per-neuron operation counts: n-TangentProp uses roughly n * p(n) multiplications, while jet propagation uses O(n^2); plot these for n = 10, 20, 50 to show the asymptotic crossover where n-TangentProp becomes slower than the polynomial baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (abstract, Section I, Section III-A) is that autodifferentiation costs O(M^n) to compute the n-th input derivative, because there are O(M^n) n-th order partial derivatives. This conflates the number of possible partial derivatives with the cost of computing one specific input derivative. Taylor-mode forward AD (univariate jets) propagates truncated Taylor series through each elementary operation, costing O(n^2) per operation and O(n^2 M) for a network with M parameters—polynomial in n, not exponential. Implementations such as JAX's jax.jet make this concrete. Against this true polynomial baseline, n-TangentProp's runtime O(n p(n) M) = O(e^{Theta(sqrt(n))} M) is asymptotically slower than any polynomial in n. Thus the abstract's contrast between O(e^{sqrt(n)} M) and O(e^{sqrt(n)}/n M^n) is a comparison against a strawman: the exponential blowup is an artifact of naive repeated reverse-mode graph building, not of autodifferentiation generally. The algorithm may still be a useful engineering contribution for moderate n (e.g., a single forward pass and linear memory in nM), but the theoretical novelty claim—'quasilinear instead of exponential'—is unsupported and likely false against the actual state of the art. The paper also miscites the Hardy–Ramanujan bound: p(n) = O(exp(pi sqrt(2n/3))/n), giving O(exp(2.565 sqrt(n)) M), not O(exp(sqrt(n)) M).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes n-TangentProp, an extension of the TangentProp formalism that computes the n-th derivative of a dense feed-forward neural network with respect to its input in a single forward pass. The derivation applies Faà di Bruno's formula layer-by-layer, leading to the recurrence in Eq. (5b) and Algorithm 1, with a claimed runtime of O(e^{sqrt(n)} M) and memory O(nM). The authors contrast this with an alleged O(e^{sqrt(n)}/n M^n) cost for autodifferentiation, and they present empirical scaling experiments and an end-to-end PINN application to unstable self-similar Burgers profiles, with code available on GitHub.","tokens_in":13814,"tokens_out":6425,"duration_ms":69479,"significance":"If the quasilinear claim were correct, the algorithm would be a significant contribution to physics-informed neural networks, where high-order input derivatives are a bottleneck. The Faà di Bruno-based recurrence is exact and self-contained, and the released code is a concrete engineering artifact. The empirical forward-pass timings for moderate n are consistent with the paper's own complexity formula when compared against PyTorch reverse-mode autodifferentiation. However, the central theoretical significance is undermined by the fact that Taylor-mode forward automatic differentiation already computes input-only derivatives in polynomial time per layer; against that state of the art, the proposed algorithm is asymptotically slower, not faster. The paper's contribution is therefore best understood as an empirical acceleration over a specific implementation (PyTorch reverse-mode), not as a quasilinear alternative to automatic differentiation in general.","major_comments":[{"comment":"The claimed exponential baseline for autodifferentiation is not the relevant baseline for input-only derivatives. Taylor-mode forward automatic differentiation (univariate jets) propagates a truncated Taylor series through each elementary operation and computes d^n f/dx^n in O(n^2 M) time for a network with M parameters, without enumerating all partial derivatives. The paper's assertion that repeated autodifferentiation necessarily costs O(M^n) because there are O(M^n) n-th order partial derivatives conflates the number of possible derivatives with the cost of computing one specific input derivative. Against this true polynomial baseline, the proposed O(e^{O(sqrt(n))} M) runtime is asymptotically slower, so the central 'quasilinear instead of exponential' claim is unsupported and likely false. The authors should either compare against Taylor-mode AD explicitly or substantially reframe the contribution as an engineering acceleration over a particular reverse-mode implementation.","section":"Abstract; Section III-A"},{"comment":"The displayed recurrence is not index-consistent. The left-hand side is (xi^(n))^{ell-1}_j, while the right-hand side evaluates sigma and xi at layer ell (sigma^{(|p|)}(a^ell_j) and (xi^(m))^ell_j). Following Eq. (3b), the correct recurrence should use the layer-(ell-1) quantities, namely sigma^{(|p|)}(a^{ell-1}_j) and the derivatives of a^{ell-1}_j, not those of a^ell_j. As printed, the equations do not directly imply the update steps in Algorithm 1; the indexing convention must be corrected or explicitly defined before the derivation can be verified.","section":"Section III-B, Eq. (5b)"},{"comment":"The Hardy-Ramanujan asymptotic is misstated. The correct classical bound is p(n) = O(exp(pi sqrt(2n/3))/n), which gives n p(n) = O(exp(2.565 sqrt(n))), not O(exp(sqrt(n))). Consequently the refined runtime O(np(n)M) is O(exp(2.565 sqrt(n)) M), and the term 'quasilinear' is inaccurate for a superpolynomial (though subexponential) function. Even with the corrected constant, the runtime is asymptotically worse than any polynomial in n, which further undermines the comparison with Taylor-mode AD.","section":"Section III-B, Hardy-Ramanujan bound"},{"comment":"The empirical evaluation compares the proposed method only against PyTorch reverse-mode autodifferentiation. Since the theoretical baseline is invalid, these experiments do not establish an improvement over the state of the art in higher-order AD. The observed speedups for n >= 3 may still support a narrower engineering claim, but the paper should clearly state that the comparison is against a specific implementation and should not imply that the algorithm outperforms all existing exact higher-order differentiation methods.","section":"Section IV-B and IV-C"}],"minor_comments":[{"comment":"The phrase 'for it's implementation' should be 'for its implementation'.","section":"Section I, contributions"},{"comment":"There are several typographical issues, including 'classifictation' in Section III-B and the broken formatting of 'Fa `a di Bruno' in the same section.","section":"Throughout"},{"comment":"The main text states that the fourth profile was 'able to compute' and reports timings, while Appendix A says the method 'failed to adequately converge' to the fourth profile. This contradiction should be resolved and clearly reported.","section":"Section IV-C1 and Appendix A"},{"comment":"The variable names y, a, and the use of 'BELL' are only loosely described in the text; a short explanation of the data structures and of how the Bell coefficients are precomputed would improve reproducibility.","section":"Algorithm 1"}],"recommendation":"reject","confidential_remarks":"The manuscript's central theoretical claim is not valid against the actual state of the art in automatic differentiation, and the Hardy-Ramanujan and recurrence-index errors compound the problem. The empirical results may support a more modest engineering contribution, but that would require a thorough reframing of the paper's title, abstract, and claims, which is beyond a routine revision. I recommend rejection, though the authors are encouraged to resubmit a version that honestly positions the method as an implementation-level acceleration for moderate n against reverse-mode frameworks."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Kyle,\n\nShort version: this paper is a mixed bag. The Faà di Bruno recurrence in Eq. (5b) is correct, and the authors do a decent job showing wall-clock speedups over repeated PyTorch reverse-mode autodiff for derivative orders 3–9. That part is real engineering. But the paper's central claim—that autodiff costs O(M^n) to compute d^n/dx^n f(x) and that n-TangentProp is the first quasilinear alternative—is not supported. Taylor-mode forward AD (univariate jets, implemented as jax.jet) already computes the nth input derivative in O(n^2 M) time, polynomial in n. Against that baseline, n-TangentProp's O(e^{√n} M) is asymptotically slower, not faster. The exponential blowup the paper attributes to autodiff is an artifact of repeatedly building reverse-mode graphs, not a property of autodiff generally.\n\nThe Hardy–Ramanujan bound is also misquoted: p(n) ~ exp(π√(2n/3))/(4n√3), so the stated O(e^{√n}/n) drops a factor of 2.565 in the exponent. The 'quasilinear' label is wrong for e^{√n}; it's sub-exponential, not quasilinear.\n\nMore troubling is the fourth-profile experiment. The main text says it was computed in under two hours, but the appendix admits 'our method did not satisfactorily solve for the fourth profile' and that the code 'failed to adequately converge.' That's not a minor discrepancy—it undercuts the most spectacular empirical claim.\n\nWhat's genuinely useful: the authors provide code, they're transparent about their implementation's lack of GPU optimization, and they correctly identify that PINN loss functions need input derivatives, not parameter derivatives. If the paper were reframed as 'an efficient forward-mode implementation for high-order PINN derivatives, with a comparison against naive autodiff,' it could be a solid engineering note. As written, the novelty and complexity claims are overstated, and the missing Taylor-mode comparison is a serious gap.\n\nWorth a serious referee? Yes—the idea is practically motivated and the recurrence is correct, so a careful reviewer should ask for a JAX jet benchmark and a corrected complexity analysis. But I would not accept it as-is, and I would not cite it in its current form.","headline":"A correct but well-known recurrence is marketed as a novel algorithm against a strawman baseline; the PINN engineering results are real, but the complexity claims do not survive contact with Taylor-mode AD.","tokens_in":14350,"tokens_out":3951,"would_cite":false,"duration_ms":40074,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that n-TangentProp computes exact high-order input derivatives of dense feed-forward networks in quasilinear time, reducing the exponential cost it attributes to autodifferentiation and making derivative-heavy PINN…","keywords":["higher-order derivatives","automatic differentiation","Faà di Bruno's formula","physics-informed neural networks","feed-forward neural networks","quasilinear algorithm","TangentProp","partition function"],"falsifier":"Run a Taylor-mode automatic differentiation implementation that propagates truncated Taylor series layer by layer on the same dense feed-forward benchmark and compare its runtime for $d^n/dx^n f(x)$ with n-TangentProp across $n$ and $M$; a polynomial-in-$(M,n)$ runtime comparable to or faster than n-TangentProp's would falsify the paper's claim of an exponential-to-quasilinear improvement over autodifferentiation.","tokens_in":13301,"feed_emoji":"📉","tokens_out":10423,"duration_ms":87882,"temperature":0.7,"pith_summary":"The paper tries to establish that the exponential cost of computing high-order derivatives during physics-informed neural network (PINN) training is not inherent. It proposes n-TangentProp, an extension of the TangentProp formalism, which computes the exact derivative $d^n/dx^n f(x)$ of a dense feed-forward network with a smooth, parameter-free activation in a single forward pass. The claimed runtime is quasilinear, $O(e^{\\sqrt{n}}M)$, against $O(e^{\\sqrt{n}}/n\\,M^n)$ for autodifferentiation, with memory reduced from exponential to linear. If the claim holds, derivative-heavy PINN losses—Sobolev training, high-order Burgers profiles, and similar PDE residual terms—become substantially cheaper to evaluate.","feed_headline":"Neural-net derivative cost drops from exponential to quasilinear","feed_subtitle":"It cuts the cost of high-order derivatives so PINN training with many derivatives fits on one GPU.","key_machinery":"The load-bearing object is Faà di Bruno's formula, the chain rule generalized to $n$-th derivatives of a composition. The paper inserts it into the TangentProp forward pass: instead of differentiating the network by building and traversing a computational graph, each activation layer directly produces all derivatives $(\\xi^{(m)})$ of its input, and the weight layers combine them linearly. The sum over integer partitions of $n$ with Bell-polynomial coefficients $C_p$ is what couples lower-order derivatives into the $n$-th one, and the derivatives must be computed in increasing order because $\\xi^{(n)}$ depends on all $\\xi^{(k)}$ with $k<n$. The partition function $p(n)$ therefore sets the combinatorial size of each activation step, and its asymptotic behavior is exactly what turns the recurrence into a quasilinear algorithm.","core_discovery":"The paper's central claim, stated on its own terms, is that applying Faà di Bruno's formula layer by layer converts the forward pass into a complete derivative computation. For each layer, the activation's $n$-th derivative is obtained by summing over all integer partitions of $n$: $(\\xi^{(n)})^{\\ell-1}_j = \\sum_p C_p \\sigma^{(|p|)}(a^\\ell_j) \\prod_{m=1}^n (\\xi^{(m)})^\\ell_j{}^{p_m}$, then the linear weights propagate it by $(\\gamma^{(n)})^\\ell_i = \\sum_j w^\\ell_{ij}(\\xi^{(n)})^{\\ell-1}_j$. Since the number of partition terms is the partition number $p(n)$, and the classical bound $p(n)=O(e^{\\sqrt{n}}/n)$ holds, the total cost for a network with $M$ parameters is $O(n p(n) M)=O(e^{\\sqrt{n}}M)$. The authors emphasize that the method is exact, computes only the input derivatives actually needed, and that autodifferentiation instead implicitly pays an $O(e^{\\sqrt{n}}/n\\,M^n)$ cost because it builds all partial derivatives.","pith_inferences":["Inference: the asymptotic comparison is against reverse-mode computational-graph autodifferentiation as implemented in common deep learning libraries; Taylor-mode automatic differentiation, which propagates truncated Taylor series layer by layer, also computes input derivatives without materializing all $O(M^n)$ partials, so a direct benchmark against Taylor-mode AD would likely shrink the headlin","Inference: the recurrence is stated for a scalar input and parameter-free activations, but the same Faà di Bruno summation should extend to mixed partial derivatives and multi-dimensional inputs, which would broaden the method to general PDE residuals.","Inference: the practical crossover point depends on implementation constants; the reported advantage already appears around three derivatives for a small PINN architecture, and a compiled or fully vectorized implementation would likely make the method win at lower derivative orders and larger batch sizes."],"forward_implications":["Physics-informed training with Sobolev losses can use higher derivative orders, such as $m=4$ or more, at practical cost instead of stopping at one or two extra derivatives.","The third and fourth self-similar smooth Burgers profiles become computable on a single GPU; the paper reports the third profile in under an hour and projects the autodifferentiation route to exceed 25 hours.","Memory use drops from exponential, $O(M^n)$, to linear, $O(nM)$, so more derivatives fit on the same hardware than autodifferentiation can store.","Because the method is exact, it can replace autodifferentiation for derivative computation inside a PINN training loop without changing the loss definition or the solution accuracy.","For optimizers such as L-BFGS, which evaluate many forward passes during line search, the cheap forward pass compounds into large end-to-end training speedups."],"supporting_citations":[{"why":"Supplies the TangentProp forward-propagation formalism that n-TangentProp extends to arbitrary derivative order.","marker":"[34]"},{"why":"Supplies Faà di Bruno's formula used to derive the per-layer derivative recurrence.","marker":"[31]"},{"why":"Supplies the classical bound $p(n)=O(e^{\\sqrt{n}}/n)$ used to get the quasilinear runtime.","marker":"[16]"},{"why":"Supplies the automatic differentiation survey and the computational-graph model used as the comparison baseline.","marker":"[3]"},{"why":"Introduces physics-informed neural networks and the autodifferentiation-based loss that motivates the algorithm.","marker":"[29]"},{"why":"Provides the self-similar Burgers profile application whose high-order derivative requirements motivate the experiments.","marker":"[40]"},{"why":"Supplies the reduced ODE and the smoothness conditions used in the Burgers profile experiment.","marker":"[12]"},{"why":"Provides the deep learning framework used for the implementation and the timing benchmarks.","marker":"[26]"}],"fun_headline_variants":["Exact high-order derivatives in quasilinear time","Neural-net derivative cost drops to quasilinear","Quasilinear derivatives speed PINN training","From exponential to quasilinear: neural derivatives","Exact derivatives, quasilinear cost: new algorithm"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The speedup claim rests on the assumption that autodifferentiation must compute and store all partial derivatives with respect to every input and parameter, making $O(M^n)$ time and memory unavoidable; if a polynomial-time method such as Taylor-mode automatic differentiation already computes the same input derivatives, the exponential baseline being beaten is not the true state of the art.","fun_headline_variants_meta":{"raw":{"variants":["Exact high-order derivatives in quasilinear time","Neural-net derivative cost drops to quasilinear","Quasilinear derivatives speed PINN training","From exponential to quasilinear: neural derivatives","Exact derivatives, quasilinear cost: new algorithm"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000275,"raw_usage":{"total_tokens":1656,"prompt_tokens":974,"completion_tokens":682,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":612}},"tokens_in":590,"tokens_out":682,"duration_ms":7693,"temperature":1.0,"reasoning_tokens":612,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:45:23.830453+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a Taylor-mode automatic differentiation implementation that propagates truncated Taylor series layer by layer on the same dense feed-forward benchmark and compare its runtime for $d^n/dx^n f(x)$ with n-TangentProp across $n$ and $M$; a polynomial-in-$(M,n)$ runtime comparable to or faster than n-TangentProp's would falsify the paper's claim of an exponential-to-quasilinear improvement over autodifferentiation.","supporting_citations":[{"cited_title":"S IMARD , B","cited_arxiv_id":null,"evidence_quote":"Supplies the TangentProp forward-propagation formalism that n-TangentProp extends to arbitrary derivative order."},{"cited_title":"R OMAN , The formula of faa di bruno , The American Mathematical Monthly, 87 (1980), pp","cited_arxiv_id":null,"evidence_quote":"Supplies Faà di Bruno's formula used to derive the per-layer derivative recurrence."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the classical bound $p(n)=O(e^{\\sqrt{n}}/n)$ used to get the quasilinear runtime."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the automatic differentiation survey and the computational-graph model used as the comparison baseline."},{"cited_title":"R AISSI , P","cited_arxiv_id":null,"evidence_quote":"Introduces physics-informed neural networks and the autodifferentiation-based loss that motivates the algorithm."},{"cited_title":"W ANG , C.-Y","cited_arxiv_id":null,"evidence_quote":"Provides the self-similar Burgers profile application whose high-order derivative requirements motivate the experiments."},{"cited_title":"E GGERS AND M","cited_arxiv_id":null,"evidence_quote":"Supplies the reduced ODE and the smoothness conditions used in the Burgers profile experiment."},{"cited_title":"P ASZKE , S","cited_arxiv_id":null,"evidence_quote":"Provides the deep learning framework used for the implementation and the timing benchmarks."}],"review_version":1}