REVIEW 3 major objections 3 minor 1 cited by
On Faster Marginalization with Squared Circuits via Orthonormalization
T0 review · 3 major / 3 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Orthonormal squared circuits are automatically normalized and their marginalization cost drops from quadratic in layer size to a mixed linear-plus-quadratic bound that can be much smaller.
desk verdict Theorem 1 and Proposition 1 are sound and useful, but Theorem 2's polynomial-time orthonormalization is unproven and likely false as stated due to exponential Hadamard-to-Kronecker width blow-up. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the orthonormal tensorized circuit: a computational graph whose input layers output orthonormal function vectors and whose sum layers are parameterized by matrices with orthonormal rows. Because squaring turns each layer into $W \otimes W^*$ products, integrating a squared input layer over its whole domain yields the vectorized identity $\mathrm{vec}(I_K)$, and (semi-)unitarity carries identities through sum layers. Two algorithms do the work: the marginalization algorithm skips layers that depend only on marginalized variables, evaluates layers that depend only on kept variables without squaring them, and squares only the mixed layers; the orthonormalization algorithm QR-factorizes the effective weight matrix at each sum layer, pushes the residual upper-triangular factor upward, and replaces Hadamard products by Kronecker products. Structured decomposability—disjoint scopes at product layers and a tree-structured factorization—is what lets the integrals factor and identity propagation hold.
What would settle it
Run the orthonormalization algorithm on a family of deep circuits built only from Hadamard product layers with orthonormal inputs, and track the width of the residual matrix returned at each recursive level; the polynomial-time claim fails if that width grows exponentially in depth while the input width stays polynomial.
Extended reading notes
Core claim
The central claim is that orthonormal circuits—structured-decomposable tensorized circuits whose input layers form orthonormal function vectors and whose sum layers use (semi-)unitary matrices—encode, after squaring, a distribution with partition function equal to 1. The proof works by propagating identity matrices upward through the squared circuit: integrating the squared input layers gives $\mathrm{vec}(I_K)$, and unitary sum weights and decomposable products preserve that identity structure, so the whole integral collapses to 1. For marginals, the paper's algorithm shows that layers whose scope lies entirely inside the marginalized set never need to be evaluated, layers whose scope lies entirely in the kept set are evaluated without squaring, and only the mixed layers in $\phi_{Y,Z}$ are squared, yielding Theorem 1's complexity. Finally, Theorem 2 claims a bottom-up QR-based orthonormalization: residual triangular factors are pushed toward the output, Hadamard product layers are replaced once by Kronecker product layers, and the result is an orthonormal circuit equal to the original up to the scalar $Z^{-1/2}$, computable in polynomial time.
Load-bearing premise
The load-bearing premise is that every tensorized circuit whose inputs are orthonormal can be reparameterized in polynomial time into an orthonormal circuit without the intermediate steps blowing up the circuit's size; without that bound, the faster marginalization result only helps circuits that are already given in orthonormal form.
Editorial extensions
If this is right
- For an already-orthonormal circuit, computing the partition function becomes free: $Z = 1$, so the normalizing constant no longer costs $O(L S^2)$.
- Marginalizing a variable subset costs $O(|\phi_Y| S + |\phi_{Y,Z}| S^2)$, which for tree-structured circuits with a localized query set can be close to linear in the circuit size rather than quadratic.
- Layers whose scope depends only on the marginalized variables are skipped entirely during inference, not merely evaluated faster.
- Since any tensorized circuit with orthonormal input functions can be orthonormalized in polynomial time up to the normalizing constant, the faster marginalization applies to the whole representable class, not only to models already built with unitary parameters.
- The orthonormalization algorithm returns the reciprocal square root of the partition function as a byproduct, so normalization and orthonormalization happen in one pass.
Reading between the lines
- A natural testable extension is to learn sum layers under the semi-unitary constraint via manifold gradient methods and compare density-estimation quality and marginalization wall-clock time against unconstrained squared circuits on high-dimensional image data.
- When the marginalized set is spatially scattered, the mixed layer set $\phi_{Y,Z}$ can include many branches, so the speedup over $O(L S^2)$ shrinks; quantifying how $\phi_{Y,Z}$ grows with the query pattern would let practitioners predict when the algorithm pays off.
- The construction suggests a unified marginalization recipe for tensor networks that are not in a known canonical form: orthonormalize the circuit counterpart first, then apply the marginalization algorithm, instead of deriving a separate canonical form for each network topology.
- Because the complexity of orthonormalization depends on the sizes of the residual matrices after face-splitting products, a closed-form bound on the orthonormalized circuit's size as a function of depth and width is the cleanest way to decide in advance whether conversion is worthwhile.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes orthonormal tensorized circuits, defined by orthonormal input functions and (semi-)unitary sum-layer matrices, as a parameterization of squared probabilistic circuits. Proposition 1 argues that squaring a structured-decomposable orthonormal circuit yields a normalized distribution, and Theorem 1 gives a marginalization algorithm whose complexity O(|phi_Y|S + |phi_Y,Z|S^2) improves on the prior O(LS^2) for squared circuits. Section 5 then claims, via Theorem 2 and Algorithm 1, that any circuit with orthonormal input layers can be converted to an orthonormal circuit in polynomial time up to a global constant, so the parameterization has no expressiveness loss. The normalization and marginalization results are argued in detail; the orthonormalization theorem is the main weak point.
Significance. If the main results stand, the paper makes a useful conceptual transfer from tensor-network canonical forms to squared circuits: Proposition 1 and Theorem 1 are carefully argued, the identity-matrix propagation argument in Appendix B.1 is convincing, and Algorithm B.1 is a non-trivial contribution that can speed up marginals when many layers depend only on the marginalized variables. The paper also provides concrete algorithms rather than only existence statements. However, Theorem 2 is the only bridge from 'orthonormal circuits are convenient' to 'we can always orthonormalize without losing expressiveness', and that bridge is currently not established; the practical significance of the paper is therefore conditional.
major comments (3)
- [Appendix B.3, Theorem 2 / Algorithm 1] The proof of Theorem 2 does not bound the size of the orthonormalized circuit. In Case (iii), an Hadamard product layer is replaced by a Kronecker product layer: if the two recursively orthonormalized children have output widths w1 and w2, the new Kronecker layer has width w1*w2, and the returned residual R1•R2 has K1 rows and K2*K3 columns. No invariant in the proof prevents widths from compounding across nested Hadamard layers. For example, a balanced Hadamard-product tree of depth d over K-dimensional orthonormal leaves has O(2^d) layers each of size O(K), but Algorithm 1 produces a Kronecker layer of width K^{2^d}, and the final QR decomposition at the output sum then acts on a vector of that length. The stated complexity O(Lsum*J^3 + Lprod*J^4), with J the original maximum layer size, is therefore not a valid bound for the circuit actually built by Algorithm 1. The 'polynomial time' and 'no expressiveness loss' claims in Section 5 need either a new proof with an explicit output-size invariant or a substantial restriction on the circuit class.
- [Appendix B.3, Case (iv)] The scalar-case algebra at the end of the proof is inconsistent with the QR convention used earlier. From V† = QR with Q†Q = 1, the 1x1 factor R has modulus ||V†||, so the residual R† returned by Algorithm 1 is ||V||, not ||V||^{-1}. The proof first writes 'r11 such that ||r11 V†||2 = 1', then states beta = r11 = Z^{-1/2}, and concludes Z = sum_i |v_i1|^2; only one of these relations can be correct. The intended relation is that the orthonormalized output satisfies c' = Z^{-1/2} c, so the returned scalar residual is sqrt(Z) rather than Z^{-1/2}. This needs to be corrected, since it directly affects the statement of Theorem 2.
- [Definition 1 / Algorithm A.1 / Algorithm 1] Definition 1 allows product layers with N inputs, and the size formulas explicitly depend on N, but Algorithm A.1, Algorithm 1, and the proofs in Appendices A and B treat every product layer as having exactly two inputs. The paper should either restrict all definitions and theorems to binary product layers, or provide a reduction from N-ary layers that does not change the claimed complexity. This is not merely cosmetic: for N > 2, the orthonormalization width blow-up in Case (iii) interacts with the number of product layers, and the binary decomposition changes the layer count and sizes used in the complexity statements.
minor comments (3)
- [Appendix B.2, Hadamard product case] The proof writes 'ℓ2_1(y′) = ℓ1(y′) ⊙ ℓ1(y′)∗'; the operation should be the Kronecker product, since squaring a layer produces ℓ ⊗ ℓ∗. The surrounding formula uses the correct ⊗ operation, so this appears to be a typo.
- [Appendix B.2, Case (ii)] The line defining the variable partitions repeats 'Y′2 = X′2 ∩ Y' twice; the second occurrence should be 'Z′2 = X′2 ∩ Z'. This makes the subsequent case analysis harder to follow.
- [Theorem 2 statement] The theorem statement says 'Let c be a tensorized circuit', while Algorithm 1 and the proof assume c is structured-decomposable. The statement should include the structured-decomposability assumption explicitly, as is done for Proposition 1 and Theorem 1.
Circularity Check
No circularity: the normalization and marginalization results are proved self-contained from explicit definitions; Theorem 2's size-blow-up issue is a proof gap, not a circular step.
full rationale
The paper's derivation chain is self-contained. Proposition 1 is proved by structural induction over the squaring construction in Algorithm A.1: orthonormal input functions integrate to identity vectors, and semi-unitary sum parameters propagate identity matrices through the squared circuit, yielding Z = 1. This is a direct consequence of the definitions, not an assumption of the conclusion. Theorem 1 follows from the same identity-propagation mechanism, with the complexity bound obtained by classifying layers into those depending only on kept variables, only on marginalized variables, or on both; the proof does not fit any parameter to data and does not invoke the desired marginalization cost as an input. Theorem 2 is the only potentially load-bearing step whose proof may be incomplete: the claimed polynomial-time guarantee is not backed by a bound on the width of the orthonormalized circuit after Hadamard layers are replaced by Kronecker layers, so the output size could grow exponentially in depth. However, that is a correctness or verification gap, not circularity: the theorem is not reduced to its own conclusion, nor is it justified by an unverified self-citation. The self-citations to Loconte et al. are used as background or as the squaring baseline whose properties are re-proved in the appendices, and no fitted-input-called-prediction or definitional circularity appears. The central claims therefore do not reduce to their inputs by construction.
Assumptions & free parameters
assumptions (5)
- domain assumption The circuit is structured-decomposable, so the squaring construction yields a decomposable circuit and integration factorizes across product layers.
- domain assumption Input functions can be chosen from an orthonormal basis, such as delta basis for discrete variables or Fourier/Hermite bases for continuous variables.
- standard math QR decomposition of complex matrices yields a semi-unitary factor Q and an upper triangular R, and the mixed-product properties of Kronecker and face-splitting products hold.
- domain assumption Product layers can be treated as binary in the algorithms, even though Definition 1 allows N-ary product layers.
- standard math Matrix products and QR decompositions take cubic time in matrix dimensions.
Cite this review
Pith. "Pith review of On Faster Marginalization with Squared Circuits via Orthonormalization." pith.science (2026). https://pith.science/paper/R52PPIMV
@misc{pith2026241207883,
author = {Pith},
title = {Pith review of: On Faster Marginalization with Squared Circuits via Orthonormalization},
year = {2026},
howpublished = {\url{https://pith.science/paper/R52PPIMV}},
note = {Machine review of arXiv:2412.07883}
}
read the original abstract
Squared tensor networks (TNs) and their generalization as parameterized computational graphs -- squared circuits -- have been recently used as expressive distribution estimators in high dimensions. However, the squaring operation introduces additional complexity when marginalizing variables or computing the partition function, which hinders their usage in machine learning applications. Canonical forms of popular TNs are parameterized via unitary matrices as to simplify the computation of particular marginals, but cannot be mapped to general circuits since these might not correspond to a known TN. Inspired by TN canonical forms, we show how to parameterize squared circuits to ensure they encode already normalized distributions. We then use this parameterization to devise an algorithm to compute any marginal of squared circuits that is more efficient than a previously known one. We conclude by formally showing the proposed parameterization comes with no expressiveness loss for many circuit classes.
Figures
Forward citations
Cited by 1 Pith paper
-
A Quantum Information Theoretic Approach to Tractable Probabilistic Models
Positive unital circuits are a new tractable probabilistic model family built from quantum POVMs, and their decomposable variant is the first non-monotone circuit class not requiring structured decomposability.
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Abramowitz, M.; Stegun, I. A.; and Miller, D. 1965. Handbook of Mathematical Functions With Formulas, Graphs and Mathematical Tables (National Bureau of Standards Applied Mathematics Series No. 55). Journal of Applied Mechanics, 32: 239--239
work page 1965
-
[4]
Absil, P.-A.; Mahony, R. E.; and Sepulchre, R. 2007. Optimization Algorithms on Matrix Manifolds. Princeton University Press
work page 2007
-
[5]
Ahmed, K.; Teso, S.; Chang, K.-W.; Van den Broeck, G.; and Vergari, A. 2022. Semantic probabilistic layers for neuro-symbolic learning. In Advances in Neural Information Processing Systems 35 (NeurIPS), volume 35, 29944--29959. Curran Associates, Inc
work page 2022
-
[6]
Arjovsky, M.; Shah, A.; and Bengio, Y. 2015. Unitary Evolution Recurrent Neural Networks. In International Conference on Machine Learning
work page 2015
-
[7]
Bansal, N.; Chen, X.; and Wang, Z. 2018. Can We Gain More from Orthogonality Regularizations in Training Deep Networks? In Advances in Neural Information Processing Systems, volume 31
work page 2018
-
[8]
Biamonte, J. D.; and Bergholm, V. 2017. Tensor Networks in a Nutshell. arXiv: Quantum Physics
work page 2017
Show all 43 references
-
[9]
M.; Blei, D.; and Saul, L
Cai, D.; Modi, C.; Margossian, C.; Gower, R. M.; Blei, D.; and Saul, L. K. 2024. Eigen VI : score-based variational inference with orthogonal function expansions. In The Thirty-eighth Annual Conference on Neural Information Processing Systems ( NeurIPS )
2024
-
[10]
L.; and Mart \'i nez-Rubio, D
Casado, M. L.; and Mart \'i nez-Rubio, D. 2019. Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group. ArXiv, abs/1901.08428
2019 arXiv
-
[11]
Cheng, S.; Wang, L.; Xiang, T.; and Zhang, P. 2019. Tree tensor networks for generative modeling. Physical Review B, 99(15): 155131
2019
-
[12]
Choi, Y.; Vergari, A.; and Van den Broeck, G. 2020. Probabilistic Circuits: A Unifying Framework for Tractable Probabilistic Modeling. Technical report, University of California, Los Angeles (UCLA)
2020
-
[13]
Darwiche, A.; and Marquis, P. 2002. A knowledge compilation map. Journal of Artificial Intelligence Research (JAIR), 17: 229--264
2002
-
[14]
Gala, G.; de Campos, C.; Peharz, R.; Vergari, A.; and Quaeghebeur, E. 2024 a . Probabilistic Integral Circuits. In AISTATS 2024
2024
-
[15]
Gala, G.; de Campos, C.; Vergari, A.; and Quaeghebeur, E. 2024 b . Scaling Continuous Latent Variable Models as Probabilistic Integral Circuits. arXiv preprint arXiv:2406.06494
2024 arXiv
-
[16]
Glasser, I.; Pancotti, N.; and Cirac, J. I. 2018. From Probabilistic Graphical Models to Generalized Tensor Networks for Supervised Learning. IEEE Access, 8: 68169--68182
2018
-
[17]
Glasser, I.; Sweke, R.; Pancotti, N.; Eisert, J.; and Cirac, I. 2019. Expressive power of tensor-network factorizations for probabilistic modeling. In Advances in Neural Information Processing Systems 32 (NeurIPS), 1498--1510. Curran Associates, Inc
2019
-
[18]
V.; and Haegeman, J
Hauru, M.; Damme, M. V.; and Haegeman, J. 2020. Riemannian optimization of isometric tensor networks. SciPost Physics
2020
-
[19]
W.; and Li, B
Huang, L.; Liu, X.; Lang, B.; Yu, A. W.; and Li, B. 2017. Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks. In AAAI Conference on Artificial Intelligence
2017
-
[20]
Jackson, D. 1941. Fourier series and orthogonal polynomials
1941
-
[21]
Jackson, D. 1982. The theory of approximation
1982
-
[22]
Liu, A.; Mandt, S.; and Van den Broeck, G. 2022. Lossless Compression with Probabilistic Circuits. In International Conference on Learning Representations
2022
-
[23]
S.; Mengel, S.; Trapp, M.; Solin, A.; Gillis, N.; and Vergari, A
Loconte, L.; Aleksanteri, M. S.; Mengel, S.; Trapp, M.; Solin, A.; Gillis, N.; and Vergari, A. 2024 a . Subtractive Mixture Models via Squaring: Representation and Learning. In The Twelfth International Conference on Learning Representations ( ICLR )
2024
-
[24]
Loconte, L.; Di Mauro , N.; Peharz, R.; and Vergari, A. 2023. How to Turn Your Knowledge Graph Embeddings into Generative Models via Probabilistic Circuits. In Advances in Neural Information Processing Systems 37 (NeurIPS). Curran Associates, Inc
2023
-
[25]
Loconte, L.; Mari, A.; Gala, G.; Peharz, R.; de Campos, C.; Quaeghebeur, E.; Vessio, G.; and Vergari, A. 2024 b . What is the Relationship between Tensor Factorizations and Circuits (and How Can We Exploit it)? arXiv:2409.07953
2024 arXiv
-
[26]
Loconte, L.; Mengel, S.; and Vergari, A. 2024. Sum of Squares Circuits. arXiv:2408.11778
2024 arXiv
-
[27]
A.; Ryzhov, A.; Filippov, S
Luchnikov, I. A.; Ryzhov, A.; Filippov, S. N.; and Ouerdane, H. 2021. QGOpt: Riemannian optimization for quantum technologies. SciPost Physics
2021
-
[28]
S.; Panov, M
Novikov, G. S.; Panov, M. E.; and Oseledets, I. V. 2021. Tensor-train density estimation. In 37th Conference on Uncertainty in Artificial Intelligence (UAI), volume 161 of Proceedings of Machine Learning Research, 1321--1331. PMLR
2021
-
[29]
Or \'u s, R. 2013. A Practical Introduction to Tensor Networks: Matrix Product States and Projected Entangled Pair States. Annals of Physics, 349: 117--158
2013
-
[30]
Peharz, R.; Tschiatschek, S.; Pernkopf, F.; and Domingos, P. M. 2015. On Theoretical Properties of Sum-Product Networks . In International Conference on Artificial Intelligence and Statistics
2015
-
[31]
M.; and Cirac, J
P \'e rez-Garc \'i a, D.; Verstraete, F.; Wolf, M. M.; and Cirac, J. I. 2007. Matrix Product State Representations. Quantum Information and Computing, 7(5): 401–430
2007
-
[32]
Pinheiro, A.; and Vidakovic, B. 1997. Estimating the square root of a density via compactly supported wavelets. Computational Statistics and Data Analysis, 25(4): 399--415
1997
-
[33]
Pipatsrisawat, K.; and Darwiche, A. 2008. New Compilation Languages Based on Structured Decomposability. In 23rd Conference on Artificial Intelligence (AAAI), volume 8, 517--522
2008
-
[34]
Roman, S. 1984. The Umbral Calculus
1984
-
[35]
Schollwoeck, U. 2010. The density-matrix renormalization group in the age of matrix product states. Annals of Physics, 326: 96--192
2010
-
[36]
Shao, X.; Molina, A.; Vergari, A.; Stelzner, K.; Peharz, R.; Liebig, T.; and Kersting, K. 2022. Conditional sum-product networks: Modular probabilistic circuits via gate functions. International Journal of Approximate Reasoning, 140: 298--313
2022
-
[37]
Shi, Y.-Y.; Duan, L.-M.; and Vidal, G. 2006. Classical simulation of quantum many-body systems with a tree tensor network. Physical Review A, 74: 22320
2006
-
[38]
Shpilka, A.; and Yehudayoff, A. 2010. Arithmetic Circuits: A survey of recent results and open questions. Founddations and Trends in Theoretical Computer Science, 5: 207--388
2010
-
[39]
Stoudenmire, E.; and Schwab, D. J. 2016. Supervised Learning with Tensor Networks. In Advances in Neural Information Processing Systems 29 (NeurIPS), 4799--4807. Curran Associates, Inc
2016
-
[40]
Vergari, A.; Choi, Y.; Liu, A.; Teso, S.; and Van den Broeck, G. 2021. A Compositional Atlas of Tractable Circuit Operations for Probabilistic Inference. In Advances in Neural Information Processing Systems 34 (NeurIPS), 13189--13201. Curran Associates, Inc
2021
-
[41]
Vergari, A.; Di Mauro , N.; and Esposito, F. 2019. Visualizing and understanding sum-product networks. Machine Learning, 108(4): 551--573
2019
-
[42]
Yang, Y.; Mandt, S.; and Theis, L. 2022. An Introduction to Neural Data Compression. Foundations and Trends in Computer Graphics and Vision, 15: 113--200
2022
-
[43]
Zhang, H.; Dang, M.; Peng, N.; and Van den Broeck, G. 2023. Tractable Control for Autoregressive Language Generation. In 40th International Conference on Machine Learning (ICML), volume 202 of Proceedings of Machine Learning Research, 40932--40945. PMLR
2023
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.