REVIEW 4 minor 55 references
Random sampling maps turn size differences into a metric, giving explicit rates for training small and evaluating large any-dimensional models.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-10 18:19 UTC pith:65CESEZE
load-bearing objection Clean, usable theory of size generalization and sketching via FinSet sampling maps, with improved rates for the architectures people actually use.
Any-Dimensional Learning by Sampling
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
If a sequence of compact sets is closed under a compatible sampling map of finite degree, the induced sampling metric makes the space of all sizes precompact; every function that is Lipschitz continuous with respect to that metric then satisfies explicit high-probability sketching bounds |f(x)−f(S_n(x))| and uniform generalization bounds e_∞(f,ĝ)≤e_n(f,ĝ)+R_n with R_n→0. The rates become O(n^{-1}) or O(n^{-1/2}) as soon as f depends only on the law or moments of a fixed-size sample.
What carries the argument
The sampling metric dsamp(x,y)=∑_k 2^{-k} W_1(S_k(x),S_k(y)), built from random sampling maps S_k that arise from actions of maps between finite sets on index sets of finite degree; compactness of the completion plus concentration of dsamp(x,S_n(x)) produce the rates.
Load-bearing premise
The sets of objects of each size must be compact and closed under the chosen sampling maps, and the functions of interest must be Lipschitz continuous with respect to the resulting sampling metric.
What would settle it
Construct a sequence of compact sets closed under sampling and an L-Lipschitz function for which the observed |f(x)−f(S_n(x))| fails to decay at the claimed rate (or fails to concentrate) as n grows, or exhibit a natural any-dimensional model whose continuity modulus with respect to the sampling metric is infinite.
If this is right
- A model trained only up to size n automatically controls error on every larger size once the training error and the explicit remainder R_n are known.
- Evaluating a permutation-invariant transformer or GNN on an N-token or N-node input can be replaced, with high probability, by evaluation on a fixed-size random sketch whose size depends only on the desired accuracy.
- Any-dimensional data distributions are precisely the sequences obtained by sampling a single random limit object; average-error generalization rates follow for free.
- Moment polynomials, graph densities/numbers, DeepSets, PointNet and polynomial GNNs inherit the improved 1/n or 1/√n rates without further analysis.
Where Pith is reading between the lines
- The same sampling metric may furnish a practical early-stopping or curriculum criterion: keep increasing training size until the empirical remainder R_n falls below a target tolerance.
- Domains whose natural symmetries mix duplication and zero-padding (e.g., sparse graphs with occasional dense cores) may require hybrid sampling maps not yet covered by the three families.
- Because the rates are uniform over all Lipschitz functions, they immediately supply dimension-free sketching guarantees for any future architecture proved continuous in one of the three sampling metrics.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper develops a unified sampling-based framework for any-dimensional generalization and sketching of functions defined on objects of varying size (sequences, graphs, tensors, etc.). It introduces three families of sampling maps—sampling with replacement, random binning, and species sampling—via FinSet-compatible actions on index sets of finite degree, and defines a sampling metric via Wasserstein distances of the resulting samples. Compactness of the resulting metric spaces (Prop. 2.3, Thm. 2.1) is shown to be equivalent to uniform approximation, sketching, and generalization properties. Explicit high-probability sketching rates and uniform generalization rates e_∞(f,ĝ)≤e_n(f,ĝ)+R_n are proved for L-Lipschitz functions on compact sets closed under the sampling maps (Thms. 3.1, 3.7, 4.2, 4.5); the rates improve to O(n^{-1}) or O(n^{-1/2}) when the function factors through a fixed-dimensional law or moment (Cors. 3.8–3.10, 4.6–4.9). Concrete applications include moment polynomials, graph homomorphism densities/numbers, permutation-invariant transformers, DeepSets, PointNet, and graph neural networks. Any-dimensional data distributions are characterized via projection- or equipartition-consistent sequences (Thms. 3.4, 4.3).
Significance. The work supplies a clean, first-principles reduction of any-dimensional learning to classical finite-dimensional generalization once a suitable sampling metric is fixed, together with explicit, non-asymptotic rates that improve substantially on earlier bounds for the same function classes (e.g., O(n^{-1}) versus O(n^{-1/d}) or O((log n)^{-1/4}) for polynomials). The FinSet-action formalism unifies several previously separate literatures (graphons, exchangeable arrays, random quotients, species sampling) and yields immediately usable sketching maps whose size depends only on the target accuracy, not on ambient dimension. The proofs are complete and self-contained (bounded-difference concentration, covering-number estimates, finite de Finetti lemmas from the authors’ prior work), and the standing hypotheses are stated clearly and verified for the motivating families. This is a solid contribution to the mathematical foundations of size-generalization and sketching.
minor comments (4)
- The informal statements of Theorems 1.5 and 1.8 in the introduction omit the precise dependence of the constants on the degree D and the ambient radius r; a short parenthetical reference to the formal statements would help readers who stop at the introduction.
- In Corollary 3.10 the infinite-precision self-attention and mean-pooling assumptions are stated only in a footnote; elevating them to the main text would make the applicability conditions more transparent.
- The covering-number argument in the proof of Theorem 3.1 (after Lemma 3.12) invokes a generic bound from Kolmogorov–Tikhomirov; a one-line remark that the same rates hold under any entropy integral of order o(n^{1/2}) would clarify robustness.
- A few typographical inconsistencies appear (e.g., “Wehavestated” on p. 2, missing spaces after periods in several places). A light copy-edit pass would remove them.
Circularity Check
No significant circularity: rates follow from concentration of sampling metrics under explicit Lipschitz and closedness hypotheses; self-citations supply independent lemmas.
specific steps
-
self citation load bearing
[Theorem 3.4 / proof of Theorem 3.4 (and analogously Theorem 4.3)]
"Conversely, if (µn) is projection-consistent then by [15, Thm. 4.15] there is a sequence νi ∈ P(Ω∞) such that Law(Rn(Xi)) → µn weakly for each n … The last claim follows by Proposition 2.7, which applies by Theorem 3.1."
The characterization of projection-consistent sequences as sampling representations of limit objects relies on a finite de Finetti theorem proved in the authors’ concurrent paper [15]. The citation is load-bearing for the de-Finetti-type statement of Theorem 3.4, but the main sketching/generalization rates of Theorems 3.1/3.7 do not depend on it; they follow from the concentration arguments given in the present paper. Hence the circularity is minor and non-central.
full rationale
The paper’s central claims (Theorems 3.1, 3.7, 4.2, 4.5 and their corollaries) are derived from first principles once the FinSet-compatible sampling maps, the sampling metric dsamp, and the standing hypotheses (compact sets closed under sampling, L-Lipschitz continuity) are fixed. Theorem 2.1 reduces any-dimensional sketching/generalization to precompactness of the completion; Proposition 2.3 obtains precompactness from Tychonoff; Theorems 3.1 and 4.2 establish concentration of dsamp(x,Sn(x)) via bounded-difference inequalities, covering-number estimates, and the elementary empirical-measure bound (27); Theorems 3.7 and 4.5 transfer those rates to Lipschitz functions, with the improved O(n^{-1}) and O(n^{-1/2}) rates following once the function factors through a fixed-dimensional law or moment (Corollaries 3.8–3.10, 4.6–4.9). The self-citations to the authors’ earlier de-Finetti papers [15,13] supply independently stated and proved lemmas (finite de Finetti theorems, polynomial representations) that are used as black-box tools; they do not redefine the target rates. No step reduces a claimed prediction to a fitted input or to a uniqueness theorem that is itself circular. The single minor self-citation pattern is therefore non-load-bearing, yielding score 1.
Axiom & Free-Parameter Ledger
axioms (4)
- standard math Wasserstein-1 distance metrizes weak convergence on compact metric spaces
- standard math Bounded-difference inequality / McDiarmid’s inequality for concentration of sampling maps
- domain assumption Objects of different sizes that differ by permutation + duplication (resp. zero-padding) are regarded as equivalent
- domain assumption The target function classes are Lipschitz continuous with respect to the sampling metric
invented entities (1)
-
Compatible sequence of index sets with FinSet action of finite degree
independent evidence
read the original abstract
Many machine learning models are defined for inputs of different sizes, such as point clouds containing different numbers of points, sequences of tokens of different lengths, and graphs on different numbers of nodes. Such models are trained on finitely-many examples of necessarily limited sizes. How well do these models generalize from inputs of small size to larger inputs of size not seen during training? Furthermore, evaluating such models on large inputs is often expensive. How can we sketch large inputs to obtain smaller ones on which the model takes similar values? At the heart of both questions is the need to compare inputs of different sizes and to approximate large inputs by small ones. We present a unified approach to address these questions by using random sampling maps to compare inputs of different sizes. The sampling maps we consider are generalizations of sampling with replacement, random binning, and species sampling. We characterize the application domains in which each type of sampling is appropriate in terms of the symmetries and relations between problem instances of different sizes in the domain. Our framework yields explicit generalization and sketching rates for function classes continuous with respect to a chosen notion of sampling, encompassing large families of functions defined on sequences, graphs, and tensors of different sizes. Specific examples include moment polynomials on measures, homomorphism densities and numbers of graphs, permutation-invariant transformers, and graph neural networks.
Reference graph
Works this paper leans on
-
[1]
Limits of dense graph sequences.Journal of Combinatorial Theory, Series B, 96(6):933–957, 2006
László Lovász and Balázs Szegedy. Limits of dense graph sequences.Journal of Combinatorial Theory, Series B, 96(6):933–957, 2006
work page 2006
-
[2]
C. Borgs, J.T. Chayes, L. Lovász, V.T. Sós, and K. Vesztergombi. Convergent sequences of dense graphs I: Subgraph frequencies, metric properties and testing.Advances in Mathematics, 219(6):1801– 1851, 2008
work page 2008
-
[3]
Graph limits and exchangeable random graphs
Persi Diaconis and Svante Janson. Graph limits and exchangeable random graphs.arXiv preprint arXiv:0712.2749, 2007. 39
work page internal anchor Pith review Pith/arXiv arXiv 2007
-
[4]
Romain Abraham, Jean-François Delmas, and Julien Weibel. Probability-graphons: Limits of large dense weighted graphs.Innovations in Graph Theory, 2:25–117, March 2025
work page 2025
-
[5]
Graham Cormode and S. Muthukrishnan. An improved data stream summary: the count-min sketch and its applications.Journal of Algorithms, 55(1):58–75, 2005
work page 2005
-
[6]
Hash kernels for structured data.Journal of Machine Learning Research, 10(11), 2009
Qinfeng Shi, James Petterson, Gideon Dror, John Langford, Alex Smola, and SVN Vishwanathan. Hash kernels for structured data.Journal of Machine Learning Research, 10(11), 2009
work page 2009
-
[7]
Mohammad Hossein Yassaee, Mohammad Reza Aref, and Amin Gohari. Achievability proof via output statistics of random binning.IEEE Transactions on Information Theory, 60(11):6760–6786, 2014
work page 2014
-
[8]
Nonparametric bayesian inference
Abel Rodriguez and Peter Müller. Nonparametric bayesian inference. InNSF-CBMS Regional Confer- ence Series in Probability and Statistics, volume 9, pages i–110. JSTOR, 2013
work page 2013
-
[9]
Jim Pitman. Exchangeable and partially exchangeable random partitions.Probability theory and related fields, 102(2):145–158, 1995
work page 1995
-
[10]
J. F. C. Kingman. The representation of partition structures.Journal of the London Mathematical Society, s2-18(2):374–380, 10 1978
work page 1978
-
[11]
Random partitions in population genetics.Proceedings of the Royal Society of London
John Frank Charles Kingman. Random partitions in population genetics.Proceedings of the Royal Society of London. A. Mathematical and Physical Sciences, 361(1704):1–20, 05 1978
work page 1978
-
[12]
American Mathematical Soc., 2012
László Lovász.Large networks and graph limits, volume 60. American Mathematical Soc., 2012
work page 2012
-
[13]
Limits of weighted graphs via random quotients.arXiv preprint arXiv:2512.23149, 2025
Eitan Levin and Venkat Chandrasekaran. Limits of weighted graphs via random quotients.arXiv preprint arXiv:2512.23149, 2025
-
[14]
Vladimir Vapnik.The Nature of Statistical Learning Theory. Springer, 1999
work page 1999
-
[15]
Eitan Levin and Venkat Chandrasekaran. Any-dimensional polynomial optimization via de Finetti theorems.arXiv preprint arXiv:2507.15632, 2025
-
[16]
Eitan Levin, Yuxin Ma, Mateo Díaz, and Soledad Villar. On transferring transferability: Towards a theory for size generalization.arXiv preprint arXiv:2505.23599, 2025
-
[17]
Takashi Furuya, Maarten V. de Hoop, and Gabriel Peyré. Transformers are universal in-context learners. InThe Thirteenth International Conference on Learning Representations, 2025
work page 2025
-
[18]
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017
work page 2017
-
[19]
Qi, Hao Su, Kaichun Mo, and Leonidas J
Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017
work page 2017
-
[20]
The graph neural network model.IEEE Transactions on Neural Networks, 20(1):61–80, 2009
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model.IEEE Transactions on Neural Networks, 20(1):61–80, 2009
work page 2009
-
[21]
Christian Bueno and Alan Hylton. On the representation power of set pooling networks.Advances in Neural Information Processing Systems, 34:17170–17182, 2021
work page 2021
-
[22]
Invariant and equivariant graph networks
Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. InInternational Conference on Learning Representations, 2019
work page 2019
-
[23]
Graphon neural networks and the transferability of graph neural networks
Luana Ruiz, Luiz Chamon, and Alejandro Ribeiro. Graphon neural networks and the transferability of graph neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1702–1712. Curran Associates, Inc., 2020. 40
work page 2020
-
[24]
A survey of transformers.AI Open, 3:111–132, 2022
Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. A survey of transformers.AI Open, 3:111–132, 2022
work page 2022
-
[25]
Trans- formers in time series: a survey
Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Trans- formers in time series: a survey. InProceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI ’23, 2023
work page 2023
-
[26]
Sohir Maskey, Ron Levie, and Gitta Kutyniok. Transferability of graph neural networks: an extended graphon approach.Applied and Computational Harmonic Analysis, 63:48–83, 2023
work page 2023
-
[27]
Generalization analysis of message passing neural networks on large random graphs
Sohir Maskey, Ron Levie, Yunseok Lee, and Gitta Kutyniok. Generalization analysis of message passing neural networks on large random graphs. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors,Advances in Neural Information Processing Systems, volume 35, pages 4805–4817. Curran Associates, Inc., 2022
work page 2022
-
[28]
Limits, approximation and size transferability for gnns on sparse graphs via graphops
Thien Le and Stefanie Jegelka. Limits, approximation and size transferability for gnns on sparse graphs via graphops. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 41305–41342. Curran Associates, Inc., 2023
work page 2023
-
[29]
Trans- formers can achieve length generalization but not robustly
Yongchao Zhou, Uri Alon, Xinyun Chen, Xuezhi Wang, Rishabh Agarwal, and Denny Zhou. Trans- formers can achieve length generalization but not robustly. InICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024
work page 2024
-
[30]
A formal framework for understanding length generalization in trans- formers
Xinting Huang, Andy Yang, Satwik Bhattamishra, Yash Sarrof, Andreas Krebs, Hattie Zhou, Preetum Nakkiran, and Michael Hahn. A formal framework for understanding length generalization in trans- formers. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors,International Conference on Learning Representations, volume 2025, pages 58095–58179, 2025
work page 2025
-
[31]
The impact of positional encoding on length generalization in transformers
Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, volume 36, pages 24892–24928. Curran Associates, Inc., 2023
work page 2023
-
[32]
Length Generalization Bounds for Transformers
Andy Yang, Pascal Bergsträßer, Georg Zetzsche, David Chiang, and Anthony W Lin. Length general- ization bounds for transformers.arXiv preprint arXiv:2603.02238, 2026
work page internal anchor Pith review Pith/arXiv arXiv 2026
-
[33]
Function graph transformers universally approximate operators between function spaces
Takashi Furuya, David Mis, Ivan Dokmanić, Maarten V de Hoop, and Matti Lassas. Func- tion graph transformers universally approximate operators between function spaces.arXiv preprint arXiv:2605.17968, 2026
work page internal anchor Pith review Pith/arXiv arXiv 2026
-
[34]
Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions? InInternational Conference on Learning Representations, 2020
work page 2020
-
[35]
Invariant Kernels: Rank Stabilization and Generalization Across Dimensions
Mateo Díaz, Dmitriy Drusvyatskiy, Jack Kendrick, and Rekha R Thomas. Invariant kernels: Rank stabilization and generalization across dimensions.arXiv preprint arXiv:2502.01886, 2025
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[36]
Nicolas Fournier. Convergence of the empirical measure in expected wasserstein distance: non- asymptotic explicit bounds inRd.ESAIM: Probability and Statistics, 27:749–775, 2023
work page 2023
-
[37]
Szemerédi’s lemma for the analyst.GAFA Geometric And Functional Analysis, 17(1):252–270, 2007
László Lovász and Balázs Szegedy. Szemerédi’s lemma for the analyst.GAFA Geometric And Functional Analysis, 17(1):252–270, 2007
work page 2007
- [38]
-
[39]
Spivak.Polynomial Functors: A Mathematical Theory of Interaction
Nelson Niu and David I. Spivak.Polynomial Functors: A Mathematical Theory of Interaction. London Mathematical Society Lecture Note Series. Cambridge University Press, 2025. 41
work page 2025
-
[40]
Antonio Ortega, Pascal Frossard, Jelena Kovačević, José M. F. Moura, and Pierre Vandergheynst. Graph signal processing: Overview, challenges, and applications.Proceedings of the IEEE, 106(5):808– 828, 2018
work page 2018
-
[41]
Pierre Cardaliaguet. Notes on mean field games. Technical report, Technical report, 2010
work page 2010
-
[42]
Zhiyi Zhang and Michael Grabchak. Entropic representation and estimation of diversity indices.Journal of Nonparametric Statistics, 28(3):563–575, 2016
work page 2016
-
[43]
P. Diaconis and D. Freedman. Finite Exchangeable Sequences.The Annals of Probability, 8(4):745 – 764, 1980
work page 1980
-
[44]
Edwin Hewitt and Leonard J. Savage. Symmetric measures on cartesian products.Transactions of the American Mathematical Society, 80(2):470–501, 1955
work page 1955
-
[45]
Classes of equivalent random quantities.Uspekhi Matematicheskikh Nauk, 8(2):125–130, 1953
Evgenii Borisovich Dynkin. Classes of equivalent random quantities.Uspekhi Matematicheskikh Nauk, 8(2):125–130, 1953
work page 1953
-
[46]
A. J. Stam. Distance between sampling with and without replacement.Statistica Neerlandica, 32(2):81– 91, 1978
work page 1978
-
[47]
Cambridge university press, 2019
Martin J Wainwright.High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge university press, 2019
work page 2019
-
[48]
Peter Bartlett. Theoretical statistics, lecture 14.https://www.stat.berkeley.edu/~bartlett/ courses/2013spring-stat210b/notes/14notes.pdf, 2013
work page 2013
-
[49]
Andrei Nikolaevich Kolmogorov and Vladimir Mikhailovich Tikhomirov.ε-entropy andε-capacity of sets in function spaces.Uspekhi Matematicheskikh Nauk, 14(2):3–86, 1959
work page 1959
-
[50]
UlrikevonLuxburgandOlivierBousquet. Distance-basedclassificationwithLipschitzfunctions.Journal of Machine Learning Research, 5(Jun):669–695, 2004
work page 2004
-
[51]
Approximation theory for lipschitz continuous transformers.arXiv preprint arXiv:2602.15503, 2026
Takashi Furuya, Davide Murari, and Carola-Bibiane Schönlieb. Approximation theory for lipschitz continuous transformers.arXiv preprint arXiv:2602.15503, 2026
-
[52]
The ring of multisymmetric functions.Annales de l’Institut Fourier, 55(3):717– 731, 2005
Francesco Vaccarino. The ring of multisymmetric functions.Annales de l’Institut Fourier, 55(3):717– 731, 2005
work page 2005
-
[53]
Stanley and Sergey Fomin.Enumerative Combinatorics
Richard P. Stanley and Sergey Fomin.Enumerative Combinatorics. Cambridge Studies in Advanced Mathematics. Cambridge University Press, 1999
work page 1999
-
[54]
Ian Grant Macdonald.Symmetric functions and Hall polynomials. Oxford university press, 1998
work page 1998
-
[55]
On edge exchangeable random graphs.Journal of statistical physics, 173(3):448–484, 2018
Svante Janson. On edge exchangeable random graphs.Journal of statistical physics, 173(3):448–484, 2018. 42
work page 2018
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.