Pith. sign in

REVIEW 4 major objections 5 minor 38 references

A single local update rule can grow usable weights for MLPs, CNNs, and ResNets of many sizes, including architectures never seen in training.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 19:53 UTC pith:EJ6QGVUP

load-bearing objection Real extension of HyperNCA to pre-specified multi-architecture graphs with honest MNIST heatmaps; CIFAR gap and hand-mapped neighborhoods keep the generalization claim capacity-band limited. the 4 major comments →

arxiv 2607.07743 v1 pith:EJ6QGVUP submitted 2026-07-08 cs.LG cs.AI

Architecture Generalization with MetaNCA

Meet Barot , Daniel Berenberg , Sina Khajehabdollahi This is my paper
classification cs.LG cs.AI
keywords neural cellular automatalocal weight updatesarchitecture generalizationhypernetworksWeight Transformerself-organizationmeta-learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the parameters of a neural network do not have to be found by back-propagation. Instead, a compact local rule—meta-trained once on a handful of architectures—can iteratively rewrite every weight using only information from that weight’s immediate forward and backward neighbors on the computation graph. After a fixed number of such local steps the resulting task network classifies MNIST or CIFAR-100 at usable accuracy, even when the architecture itself was never seen during meta-training. The same rule scales from tiny MLPs to two-million-parameter CNNs and ResNets. Because the rule never needs the global gradient of the task network, a single small generator can produce an entire family of networks. Architectural diversity during meta-training measurably strengthens this generalization.

Core claim

A learned local rule network, operating solely on forward and backward neighborhoods of each weight, self-organizes the parameters of task networks (MLPs, CNNs, ResNets) into classifiers that generalize across architectures never encountered in meta-training, reaching millions of parameters without back-propagating through the task network.

What carries the argument

The Weight Transformer: a linear-attention block that builds a perception vector for each weight from its scalar value, hidden state, and the attention-aggregated signals of its forward and backward neighbors on the computation graph; an MLP head then emits the local updates that are applied for a fixed number of steps.

Load-bearing premise

That a hand-specified definition of “forward and backward neighbors” for each architecture family, plus meta-training on only two to five concrete networks, is enough for the local rule to capture weight-generation dynamics that work outside that narrow capacity band.

What would settle it

Meta-train the identical Weight Transformer rule on the same two or three ResNet widths used in the paper, then measure test accuracy on architectures whose base channel count lies well below the training band (c ≤ 14); if accuracy remains near the conventional baseline rather than collapsing, the claimed capacity-range independence is supported.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces MetaNCA, a graph neural cellular automaton whose local rule (a Weight Transformer using linear attention over forward/backward neighborhoods on the computation graph, plus per-weight hidden states) iteratively self-organizes the weights of a task network. A single rule is meta-trained by BPTT on task classification loss across a small set of architectures (2–5), then used at inference without back-propagation through the task net. Experiments show generation of usable MLPs and CNNs on MNIST (up to ~2M parameters, competitive with Adam) and ResNets on CIFAR-100 (up to 29.9% vs ~41–42% Adam), with heatmaps indicating generalization to held-out architectures within each family and improved coverage when more training architectures are used.

Significance. If the results hold under stronger controls, MetaNCA is a genuine step beyond fixed-architecture hypernetworks and grid-based HyperNCA: a single compact local rule (≈82–126k parameters) that operates directly on computation-graph edges and can be amortized across a family of architectures. Strengths include an explicit, reproducible method (code linked), honest reporting of the CIFAR-100 gap and capacity-range degradation, clear architecture-grid heatmaps with multi-sample averages, and a well-specified linear-attention Weight Transformer. The work is relevant to weight-space learning, developmental programs, and compressible model generation. The contribution is incremental rather than paradigm-shifting, but the combination of local graph updates with multi-architecture meta-training is novel and worth publishing once claims are scoped to the evidence.

major comments (4)
  1. [Abstract; Method (Local rule network inputs, Fig. 2); Discussion] The central claim of architecture generalization is load-bearing on a hand-instantiated neighborhood map that is not architecture-agnostic. Method and Fig. 2 define distinct forward/backward neighborhoods for dense vs. convolutional layers (channel/kernel structure); Discussion correctly notes residual connectivity would require a further mapping. Abstract and Introduction nevertheless present the rule as “by construction not tied to any particular architecture.” Either provide a single automatic neighborhood construction that works across families without per-family engineering, or revise the claim language throughout to “within-family generalization under a family-specific neighborhood definition.”
  2. [Results (Figs. 3–5); Discussion] Generalization is capacity-range-specific rather than universal, which undercuts the strongest reading of the claim. Fig. 5 (ResNets, trained only on c=16 and c=20) shows sharp collapse for c≤14; Fig. 3 shows that adding training points mainly improves interpolation and boundary coverage. The paper acknowledges this, but does not quantify how far one can extrapolate (e.g., relative parameter count or width ratio) nor test whether training on a deliberately wider capacity band removes the cliff. A controlled ablation—train on {small, mid, large} vs. only mid-range, report accuracy vs. distance from training set—is needed before claiming that “architectural diversity strengthens generalization” as a general principle.
  3. [Related Work; Results (Adam comparison paragraph)] Baselines are limited to Adam-trained networks of the same architecture. There is no experimental comparison to HyperNCA, layer-wise hypernetworks, HyperNEAT-style indirect encodings, or even a non-local MLP that maps positional encodings to weights. Without these, it is impossible to isolate the benefit of iterative local graph updates versus simply meta-learning a weight generator. At minimum, add (i) a static hypernetwork baseline conditioned on the same positional features and (ii) a same-family HyperNCA-style readout if feasible, on the MNIST MLP/CNN grids.
  4. [Results (Architecture Generalization: ResNets on CIFAR-100, Fig. 5)] CIFAR-100 results (max 29.9% vs Adam 41.4–42.1%) leave a large absolute gap that is only briefly noted. With only two training widths and no hyperparameter sweep reported for the rule on this task, it is unclear whether the gap is fundamental (local rule expressivity) or an under-training artifact. Either expand the ResNet training set and report a learning curve / architecture-count ablation, or explicitly demote CIFAR-100 to a stress test and center the paper’s claims on the MNIST regime where MetaNCA is competitive.
minor comments (5)
  1. [Method; Complexity Analysis] T=10, p=0.8, neighborhood dropout, and hidden dimension h are free parameters listed only in prose; a short hyperparameter table (or appendix) would aid reproducibility.
  2. [Fig. 3] Fig. 3 caption notes that (30,0) and (0,30) are the same architecture marked twice; a single marker convention or a note in the figure itself would reduce confusion.
  3. [Weight Structure, Fig. 6] Weight-structure comparison (Fig. 6) is qualitative; a simple sparsity or spectral statistic would make the “more sparse / symmetric” claim precise.
  4. [Future Work (Multitask MetaNCA)] Multitask MetaNCA is mentioned only in Future Work with no numbers; either move a small quantitative result into the main experiments or drop the claim until data exist.
  5. [Weight Transformer (Multihead Linear Attention)] Eqs. (1)–(5) and Algorithm 1 are clear, but the rotary multi-head assignment (one head per positional dimension) is easy to miss; a one-sentence explicit statement would help.

Circularity Check

0 steps flagged

No significant circularity: empirical meta-learning method with BPTT training and held-out architecture evaluation; claims rest on measured accuracies, not definitional reductions.

full rationale

MetaNCA is an empirical machine-learning paper. The local rule network is optimized end-to-end by back-propagation through time on the task classification loss for a small set of training architectures; once trained, it is applied iteratively (T=10 steps) to randomly initialized weights of held-out architectures and the resulting classifiers are evaluated on standard test splits. No equation equates a claimed prediction to a fitted constant by construction, no uniqueness theorem is imported from overlapping authors to force the design, and the neighborhood definitions (forward/backward edges on the computation graph, specialized for convolutions in Fig. 2) are explicit design choices rather than self-referential definitions of the reported accuracies. Self-citations (HyperNCA, Growing NCA, plasticity rules) appear only as background motivation. The generalization heatmaps and CIFAR-100 numbers are ordinary experimental outcomes, not tautologies. Consequently the derivation chain is self-contained against external benchmarks and exhibits zero circular steps of the enumerated kinds.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 3 invented entities

The central empirical claim rests on standard ML training assumptions plus several design choices that are free parameters of the method (update steps, mask fraction, hidden-state dimension, number and placement of meta-training architectures) and on the domain assumption that local forward/backward neighborhoods on the computation graph carry enough information for a single rule to generate high-performing weights across architectures. The Weight Transformer and MetaNCA framework themselves are invented entities whose only evidence is the experiments in this paper.

free parameters (5)
  • T (local-rule update steps) = 10
    Fixed at T=10 for all experiments; controls how many iterative applications of the rule are unrolled during meta-training and inference.
  • stochastic update fraction p = 0.8
    Fraction of weights updated each step (asynchronous simulation); set to 0.8.
  • number and choice of meta-training architectures = 2–5 architectures
    2–5 architectures per experiment; selection (including deliberately adding worst/largest configs) materially affects measured generalization coverage.
  • hidden-state dimension h and rule-network size = ~82k / ~126k rule params
    Per-weight latent size and overall rule capacity (~82k params for MLPs, ~126k for conv/ResNet); chosen by authors, not derived.
  • neighborhood dropout rate
    Random whole-vector dropout of neighbors before attention; reduces dependence on individual neighbors; value not exhaustively ablated in text.
axioms (4)
  • domain assumption Forward and backward neighbors on the computation graph (plus per-weight hidden states and sinusoidal positional encodings) constitute a sufficient local observation for learning a general weight-generation rule.
    Core modeling premise stated in Method; neighborhood must still be hand-mapped per architecture family (Discussion).
  • domain assumption Back-propagation through time through T unrolled local-rule steps, with task loss only, yields a rule that converges to high-performing weight attractors from random initialization.
    Standard meta-learning / BPTT assumption used throughout training description.
  • ad hoc to paper Linear attention with ELU feature map and rotary positional encodings is an adequate aggregator of neighbor signals.
    Architectural choice of the Weight Transformer; alternatives not systematically compared.
  • standard math Standard classification losses (cross-entropy on MNIST / CIFAR-100) and the Muon optimizer are appropriate for meta-training the rule.
    Conventional ML practice; no novel claim attached.
invented entities (3)
  • MetaNCA framework no independent evidence
    purpose: Unifies local NCA-style updates with multi-architecture weight generation on computation-graph edges.
    Name and overall pipeline introduced in this work; evidence is the reported experiments only.
  • Weight Transformer local rule no independent evidence
    purpose: Linear multi-head attention over forward/backward weight neighborhoods plus MLP head that emits Δw and Δh.
    Novel architecture block; no external validation outside this paper’s figures.
  • Per-weight hidden state vector carried across update steps no independent evidence
    purpose: Gives the local rule memory beyond the scalar weight value.
    Standard in NCA literature but here attached to every edge of an arbitrary task graph; dimension is a free design choice.

pith-pipeline@v1.1.0-grok45 · 15209 in / 3426 out tokens · 37180 ms · 2026-07-10T19:53:34.581905+00:00 · methodology

0 comments
read the original abstract

Self-organization is an emergent property of life, driven by the collective behavior of individual components acting on local information. Biological neurons, through local interactions transmitted through synapses, are able to learn efficiently and can adapt their connections over an organism's lifespan. Motivated by these desirable properties of adaptability and local interaction, neural cellular automata (NCA) models have been successful at learning morphogenesis solely through local update rules, demonstrating stability over many updates and robustness to perturbations. In this work, we introduce Meta Neural Cellular Automata (MetaNCA), a framework that learns local rules which self-organize the weights of artificial neural networks. A learned rule network iteratively updates the weights of a task network using only local interactions on the computation graph. We propose a novel Weight Transformer architecture for the local rule network, which uses linear attention to aggregate signals from neighboring weights and hidden states. Once trained, the rule network generates task networks of diverse architectures without backpropagation. We show that MetaNCA generates weights for feedforward MLPs, CNNs, and ResNets on MNIST and CIFAR-100, scaling to networks of 2 million parameters. We further show that MetaNCA generalizes to architectures not seen during meta-training, and that architectural diversity in the training phase strengthens this generalization.

Figures

Figures reproduced from arXiv: 2607.07743 by Daniel Berenberg, Meet Barot, Sina Khajehabdollahi.

Figure 1
Figure 1. Figure 1: Diagram of the Weight Transformer local rule net [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Diagram showing a weight w and its neighborhood in an example CNN’s first 3 layers. The backward neighborhood Nb(w) consists of all current filters’ weights operating on the same input channel as w, as well as the filter from the previous layer that produces this input channel. The forward neighborhood Nf (w) consists of all weights in the current filter of w, as well as all filters in the next layer from … view at source ↗
Figure 3
Figure 3. Figure 3: Heatmaps of weight transformer local rule net gen [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Heatmaps of the Weight Transformer local rule network generating convolutional task networks across three kernel [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: CIFAR-100 test accuracy for ResNet architectures [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Weight matrices (50×50) connecting the two hidden layers of MNIST dense networks. (a) Generated by MetaNCA after 10 local rule updates. (b) Trained with Adam for 50 epochs. Both achieve 97% test accuracy. vidual task networks, through transferring the capabilities of larger models to smaller ones, or different topologies al￾lowing the learning rule to capture more useful signals for optimization. We note t… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 38 canonical work pages · 8 internal anchors

  1. [1]

    Reviews of Modern Physics , volume=

    Statistical mechanics of cellular automata , author=. Reviews of Modern Physics , volume=. 1983 , publisher=

    work page 1983

  2. [2]

    Mathematical Games: The fantastic combinations of

    Gardner, Martin , journal=. Mathematical Games: The fantastic combinations of. 1970 , publisher=

    work page 1970

  3. [3]

    Training Deep Nets with Sublinear Memory Cost

    Training deep nets with sublinear memory cost , author=. arXiv preprint arXiv:1604.06174 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    2000 IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks , pages=

    Evolution and design of distributed learning rules , author=. 2000 IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks , pages=. 2000 , organization=

    work page 2000

  5. [5]

    Distill , volume=

    Growing neural cellular automata , author=. Distill , volume=

    work page

  6. [6]

    Randazzo, Ettore and Niklasson, Eyvind and Mordvintsev, Alexander , journal=

    work page

  7. [7]

    Advances in Neural Information Processing Systems , volume=

    Learning graph cellular automata , author=. Advances in Neural Information Processing Systems , volume=

    work page

  8. [8]

    Variational Neural Cellular Automata

    Variational neural cellular automata , author=. arXiv preprint arXiv:2201.12360 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Neural cellular automata manifold , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    work page

  10. [10]

    Evolutionary computation , volume=

    Evolving neural networks through augmenting topologies , author=. Evolutionary computation , volume=. 2002 , publisher=

    work page 2002

  11. [11]

    Advances in neural information processing systems , volume=

    Attention is all you need , author=. Advances in neural information processing systems , volume=

    work page

  12. [12]

    Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    Deep residual learning for image recognition , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    work page

  13. [13]

    Image Generation With Neural Cellular Automatas

    Image generation with neural cellular automatas , author=. arXiv preprint arXiv:2010.04949 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  14. [14]

    HyperNetworks

    Hypernetworks , author=. arXiv preprint arXiv:1609.09106 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    Deng, Li , journal=. The. 2012 , publisher=

    work page 2012

  16. [16]

    Annals of eugenics , volume=

    The use of multiple measurements in taxonomic problems , author=. Annals of eugenics , volume=. 1936 , publisher=

    work page 1936

  17. [17]

    2024 , publisher=

    Su, Jianlin and Ahmed, Murtadha and Lu, Yu and Pan, Shengfeng and Bo, Wen and Liu, Yunfeng , journal=. 2024 , publisher=

    work page 2024

  18. [18]

    Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

    Clevert, Djork-Arn. Fast and accurate deep network learning by exponential linear units (. arXiv preprint arXiv:1511.07289 , volume=

    work page internal anchor Pith review Pith/arXiv arXiv

  19. [19]

    Nature communications , volume=

    A critique of pure learning and what artificial neural networks can learn from animal brains , author=. Nature communications , volume=. 2019 , publisher=

    work page 2019

  20. [20]

    Proceedings of the National Academy of Sciences , volume=

    Encoding innate ability through a genomic bottleneck , author=. Proceedings of the National Academy of Sciences , volume=. 2024 , publisher=

    work page 2024

  21. [21]

    Hu, Edward J and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Liang and Chen, Weizhu and others , journal=

    work page

  22. [22]

    Distill , year =

    Randazzo, Ettore and Mordvintsev, Alexander and Niklasson, Eyvind and Levin, Michael and Greydanus, Sam , title =. Distill , year =

    work page

  23. [23]

    Distill , year =

    Niklasson, Eyvind and Mordvintsev, Alexander and Randazzo, Ettore and Levin, Michael , title =. Distill , year =

    work page

  24. [24]

    Artificial Life Conference Proceedings 37 , volume=

    Guichard, Etienne and Reimers, Felix and Kvalsund, Mia and Lepper. Artificial Life Conference Proceedings 37 , volume=. 2025 , organization=

    work page 2025

  25. [25]

    Continual learning with hypernetworks

    Continual learning with hypernetworks , author=. arXiv preprint arXiv:1906.00695 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 1906

  26. [26]

    Brock, Andrew and Lim, Theodore and Ritchie, James M and Weston, Nick , journal=

    work page

  27. [27]

    Artificial Intelligence Review , volume=

    A brief review of hypernetworks in deep learning , author=. Artificial Intelligence Review , volume=. 2024 , publisher=

    work page 2024

  28. [28]

    arXiv preprint arXiv:2603.10090 , year=

    A survey of weight space learning: Understanding, representation, and generation , author=. arXiv preprint arXiv:2603.10090 , year=

    work page arXiv

  29. [29]

    Najarro, Elias and Sudhakaran, Shyam and Glanois, Claire and Risi, Sebastian , journal=

    work page

  30. [30]

    Artificial Life Conference Proceedings 35 , volume=

    Towards self-assembling artificial neural networks through neural developmental programs , author=. Artificial Life Conference Proceedings 35 , volume=. 2023 , organization=

    work page 2023

  31. [31]

    Artificial Life Conference Proceedings 36 , volume=

    Evolving self-assembling neural networks: from spontaneous activity to experience-dependent learning , author=. Artificial Life Conference Proceedings 36 , volume=. 2024 , organization=

    work page 2024

  32. [32]

    International Conference on Machine Learning , pages=

    Differentiable plasticity: training plastic neural networks with backpropagation , author=. International Conference on Machine Learning , pages=. 2018 , organization=

    work page 2018

  33. [33]

    Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity

    Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity , author=. arXiv preprint arXiv:2002.10585 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 2002

  34. [34]

    Artificial life , volume=

    A hypercube-based encoding for evolving large-scale neural networks , author=. Artificial life , volume=. 2009 , publisher=

    work page 2009

  35. [35]

    2024 , url =

    Keller Jordan and Yuchen Jin and Vlado Boza and Jiacheng You and Franz Cesista and Laker Newhouse and Jeremy Bernstein , title =. 2024 , url =

    work page 2024

  36. [36]

    Neural Network Diffusion

    Neural network diffusion , author=. arXiv preprint arXiv:2402.13144 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  37. [37]

    Advances in Neural Information Processing Systems , year=

    Hyper-representations as generative models: Sampling unseen neural network weights , author=. Advances in Neural Information Processing Systems , year=

    work page

  38. [38]

    Advances in Neural Information Processing Systems , volume=

    Meta-learning through hebbian plasticity in random networks , author=. Advances in Neural Information Processing Systems , volume=

    work page