Pith. sign in

REVIEW 3 major objections 5 minor 36 references

ARC-NCA: Towards Developmental Solutions to the Abstraction and Reasoning Corpus

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read Neural cellular automata match ChatGPT 4.5 on ARC tasks at 1/1000 the cost.

desk verdict Real first: NCAs on ARC-AGI, but the ChatGPT parity claim rests on unmatched evaluation sets and needs a fix before it holds. read the letter →

arxiv 2505.08778 v1 pith:KH6L5T5J submitted 2025-05-13 cs.AI cs.NE

classification cs.AIcs.NE
keywords AbstractionandReasoningCorpusNeuralCellularAutomataEngramNCAdevelopmentalcomputationtest-timetrainingARC-AGIfew-shotvisual
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a developmental model—a neural cellular automaton grown anew for each puzzle—can handle the few-shot visual reasoning of the ARC-AGI benchmark. It reports that a standard NCA solves 10.7% of the 262 non-resizing public tasks, the best EngramNCA variant solves 12.9%, and the union of variants reaches 17.6%, while ChatGPT 4.5 scores 10.3% on the private evaluation set. The same runs cost roughly $0.0003–$0.0005 per task versus about $0.29 for the LLM. A sympathetic reader should take this as a proof of concept that developmental local-update rules, not just large pretrained transformers, can produce exact solutions to some ARC tasks from a handful of examples.

What carries the argument

The central object is the Neural Cellular Automaton (NCA): a grid of cells, each holding a continuous state vector updated by a shared convolutional neural network that reads local neighborhoods, so global patterns emerge from identical local rules. EngramNCA extends this with dual public and private cell states and two cooperating networks, GeneCA and GenePropCA, to encode and propagate hidden memory; the ARC-specific versions add learnable sensing kernels, a toroidal/non-toroidal split, and local patch training. The machinery operates by test-time training: for every task a new CA is trained from scratch on the example pairs by backpropagating pixel-wise MSE through the developmental steps, and a task counts as solved when $\log(\mathrm{MSE}) \le -7$.

What would settle it

Run the official ARC-AGI checker on the outputs of the best EngramNCA and of the union of models for the 262 public tasks, replacing the $\log(\mathrm{MSE}) \le -7$ rule with the checker's accept/reject. If the official solve rate is materially below 12.9% (or below 17.6% for the union), the threshold-based exact-solution claim is overoptimistic; evaluating the NCA and ChatGPT 4.5 on the same identical tasks would settle the parity claim.

Watch

Extended reading notes

Core claim

ARC-NCA's central claim is that test-time training of small neural cellular automata—one freshly initialized and trained per ARC problem on its 2–3 training pairs—can produce exact outputs for 10–13% of the 262 non-resizing public tasks, and that combining several NCA variants reaches 17.6%. The best single variant, EngramNCA v3, adds learnable sensing filters and splits toroidal and non-toroidal propagation, solving 12.9% at a threshold of $\log(\mathrm{MSE}) \le -7$. Because the cost per task is about $0.0004, the authors argue this is comparable to, and sometimes better than, ChatGPT 4.5's 10.3% solve rate at roughly 1000 times lower cost.

Load-bearing premise

The central claim rests on treating a pixel-error threshold as equivalent to an exact ARC solution and on treating the public and private ARC evaluation sets as comparable; if either assumption fails, the reported solve rates and the parity with ChatGPT 4.5 would need revision.

Editorial extensions

If this is right

  • A cheap per-task developmental learner can match a frontier LLM on a nontrivial slice of ARC-AGI, so solving ARC-style tasks does not necessarily require massive pretraining for every task type.
  • Combining diverse NCA variants in unions raises the solve rate from 12.9% to 17.6%, so ensembling developmental programs is a direct and validated lever.
  • Loosening the exactness threshold from $-7$ to $-6$ increases individual solve rates by 2–6 percentage points, suggesting that small architecture or post-processing changes may convert near-solutions into exact solutions.
  • Larger hidden states and maximal grid padding preserve or improve solve rates (16.1% and up to 27% at the looser threshold), indicating that grid-size-changing ARC tasks are tractable within the NCA paradigm.
  • The roughly three-orders-of-magnitude cost advantage makes per-task developmental training practical to run at scale and to combine with more expensive correction mechanisms.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that the parity claim is the least secure part of the paper: the ChatGPT 4.5 score comes from the private evaluation set while the NCA scores come from the public set, and the exact-solution threshold is not checked against the official ARC checker. A like-for-like evaluation on identical tasks would settle how much of the apparent parity is real.
  • If the developmental mechanism itself is what carries these solutions, then pretraining NCA on primitive transformations or initializing near criticality could raise solve rates without changing the test-time-training paradigm; this is a direct next experiment the paper leaves open.
  • The same per-task developmental training could be applied to ARC-AGI-2, where current AI systems score in single digits; a nonzero solve rate there would show the mechanism transfers beyond the original corpus.
  • A hybrid in which an LLM corrects near-solutions might preserve much of the cost advantage, since the NCA would supply most of the structure and only a few LLM calls per task would be needed.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ARC-NCA, a developmental test-time training approach in which a neural cellular automaton (standard NCA or one of several EngramNCA variants) is trained from scratch on the few training examples of each ARC-AGI task and then run to produce an output grid. On the 262 public ARC-AGI tasks that do not require resizing, individual models solve between 6.5% and 12.9% of tasks at an MSE-based threshold, with union rates up to 17.6%; a larger EngramNCA v3 model reaches 16.1%, and a maximally padded version reaches 27% at a loosened threshold. The paper reports a cost of roughly 3e-4 to 5e-4 USD per task, compared with 0.29 USD per task for ChatGPT 4.5, and claims its results are comparable to or better than ChatGPT 4.5 (10.3%) at a fraction of the cost. Qualitative examples and project code and videos are provided.

Significance. If the central comparative claim were supported, this would be a noteworthy result: a simple per-task developmental model achieving ChatGPT-4.5-level performance on ARC-style tasks at roughly a thousandth of the cost. The use of NCAs for 2D ARC tasks is, to my knowledge, novel, and the released code and videos are valuable for reproducibility and for the artificial-life community. The paper is best read as a proof of concept for a developmental baseline. However, the headline comparison to ChatGPT 4.5 is not supported by the reported evaluation protocol, because the baseline score comes from a different (private) evaluation set than the one used for ARC-NCA. The contribution is interesting but needs a fair comparison or a substantially softened claim.

major comments (3)
  1. [Results, Table 2 (and Table 3, Summary and Discussion)] The central comparative claim is not supported as stated. The footnote to Table 2 states that the ChatGPT 4.5 result (10.3%) was obtained on the ARC-AGI private evaluation set, whereas all ARC-NCA results are from the public evaluation set and, in the main experiments, only from the 262 tasks that do not require resizing. No evidence is offered that the 262-task public subset is comparable in difficulty to the full private set, and the cost comparison in Table 3 inherits this mismatch. The abstract and summary restate the claim that ARC-NCA is 'comparable to, and sometimes surpass[es]' ChatGPT 4.5 without this caveat. Please either evaluate both methods on the same task set (for example, by evaluating ChatGPT on the same 262 public tasks or submitting ARC-NCA outputs to the private leaderboard), or remove the cross-model comparison and reframe the results as a public-subset benchmark.
  2. [Training, 'Determining the Quality of Solutions'] The solve-rate metric relies on the assertion that log(MSE) <= -7 corresponds to an exact solution and that this threshold was 'experimentally determined,' but no comparison against the official ARC checker or ground-truth labels is shown. All reported solve rates, including the loosened-threshold results in Tables 5, 6, and 8, depend on this unvalidated mapping from MSE to correctness. A calibration analysis is needed: for a sample of threshold-passing outputs, verify exact equality with the official evaluation, and report the false-positive rate. Without this, the internal validity of the reported percentages is not established.
  3. [Results, 'CA Union Results', Tables 4 and 6] The union solve rates count a task as solved if any model in the union produces a correct output, and the text states that taking the union yields a valid submission because 'two answers may be submitted.' This reasoning is incorrect for tasks with more than one test input: with four models producing one output per test input, only two of those outputs can be submitted per input, and without a selection mechanism the union is an upper bound on achievable performance, not a valid submission score. Please report per-model rates as the primary results and either propose a concrete selection procedure for the union or label the union as an oracle/upper-bound metric.
minor comments (5)
  1. [Figure 7] The caption for Figure 7 says 'EngramNCA v3,' but the prose and the panel label refer to 'EngramNCA v1'; please correct the mismatch.
  2. [Table 3] The header contains the typo 'Modle' instead of 'Model,' and the paper uses both 'Chat GPT' and 'ChatGPT'; please standardize.
  3. [Abstract] The abstract contains a typo ('abilites') and the phrase 'while only few (with median count of three) correct examples are presented' is awkward; please rephrase for clarity.
  4. [Further Experiments, Table 8] The text says 'All results will be reported on the 262 problems that do not require resizing,' but the maximally padded experiment in Section 8 appears to address all tasks; please state explicitly which task set was used for Table 8 and how many tasks it covers.
  5. [Future Works] The paper acknowledges that results are documented on single trials. Given that each NCA is trained from scratch, a brief discussion of run-to-run variability (or a small repeated-run study) would strengthen the robustness of the reported solve rates.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the core results are produced by from-scratch per-task training on the external ARC-AGI benchmark, so the derivation chain is self-contained.

full rationale

The paper's load-bearing claims are evaluated against the ARC-AGI public evaluation set, which is an external benchmark with ground-truth input-output grids. For every task, a new NCA or EngramNCA is trained from scratch on the task's training examples and then evaluated on a held-out test grid; the reported solve rates therefore are not fitted to the conclusion. The EngramNCA architecture is drawn from the authors' prior work, but that citation supplies an architecture and training scheme, not the ARC results themselves, and the paper explicitly tests the suitability of the developmental approach on an external benchmark. The solve threshold log(MSE) <= -7 is a measurement criterion, not a fitted parameter whose value predetermines the comparison with ChatGPT 4.5; whether it exactly matches the official ARC checker is a correctness or validation concern, not a circular one. Similarly, the mismatch between the public 262-task subset and ChatGPT 4.5's private-set score is an apples-to-oranges comparison problem, not a case of the result being equivalent to its inputs by construction. No equation in the paper reduces to a fitted parameter that defines the target being predicted, and no load-bearing argument rests solely on an unverified self-citation. The paper is self-contained against an external benchmark, so the appropriate circularity score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No fundamentally new physical or mathematical entities are introduced. EngramNCA and its hidden memory states are borrowed from the authors' prior work (Guichard et al., 2025), and ARC-AGI is an external benchmark. The main uncharged assumptions are representational and evaluative: the color encoding, the MSE threshold, the public-private set comparability, and the assumption that local NCA updates can express ARC rules.

free parameters (4)
  • Exact-solution MSE threshold = log(MSE) <= -7
    Experimentally determined cutoff used to count a task as solved; directly sets all claimed solve rates. No independent verification that this threshold exactly matches official ARC correctness.
  • Near-solution MSE threshold = log(MSE) <= -6
    Loosened cutoff used for the 'almost solved' tables; inflates solve rates by 2 to 11 percentage points.
  • Per-task training schedule = 3000 iterations, LR 1e-3, 66% LR decay at 2000
    Chosen hyperparameters for all models; not swept, but convergence and final loss depend on them.
  • Architecture channel and hidden sizes = 50 channels; hidden 64 for NCA, (32,32) or (132,132) for EngramNCA
    Capacity choices that affect which ARC tasks can be represented and solved; the larger hidden size improves solve rate in Further Experiments.
assumptions (5)
  • domain assumption The integer-to-HSL-to-RGB-alpha encoding preserves all task-relevant information
    Equations 1-10 assume constant alpha and equal color spacing. This arbitrary representation could distort edge or color semantics, but the paper does not test alternative encodings.
  • domain assumption Pixel-wise MSE is a valid training objective and exactness proxy for ARC solutions
    MSE over visible channels is used both as loss and as the solution criterion; no external checker verifies that low MSE implies the correct abstract rule.
  • domain assumption The public evaluation set is comparable to the private leaderboard set for the ChatGPT 4.5 comparison
    This assumption is explicitly contradicted by the Table 2 footnote, which states ChatGPT 4.5 results are from the private evaluation set while ARC-NCA results are from the public set. The paper still uses the comparison as a headline.
  • domain assumption A fresh NCA trained from scratch on 2-3 examples can infer the intended ARC transformation
    The whole test-time training scheme assumes the optimizer can extract a generalizing rule from a handful of examples using MSE regression; solved examples provide partial support, but many tasks fail.
  • domain assumption Small convolutional NCA update rules have sufficient expressive capacity for ARC transformations
    The method assumes a compact local update rule can represent the needed global transformations; the moderate solve rates and 'reasoning pitfalls' suggest this capacity is often insufficient.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ARC-NCA: Towards Developmental Solutions to the Abstraction and Reasoning Corpus." pith.science (2026). https://pith.science/paper/KH6L5T5J

@misc{pith2026250508778,
  author       = {Pith},
  title        = {Pith review of: ARC-NCA: Towards Developmental Solutions to the Abstraction and Reasoning Corpus},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KH6L5T5J}},
  note         = {Machine review of arXiv:2505.08778}
}
read the original abstract

The Abstraction and Reasoning Corpus (ARC), later renamed ARC-AGI, poses a fundamental challenge in artificial general intelligence (AGI), requiring solutions that exhibit robust abstraction and reasoning capabilities across diverse tasks, while only few (with median count of three) correct examples are presented. While ARC-AGI remains very challenging for artificial intelligence systems, it is rather easy for humans. This paper introduces ARC-NCA, a developmental approach leveraging standard Neural Cellular Automata (NCA) and NCA enhanced with hidden memories (EngramNCA) to tackle the ARC-AGI benchmark. NCAs are employed for their inherent ability to simulate complex dynamics and emergent patterns, mimicking developmental processes observed in biological systems. Developmental solutions may offer a promising avenue for enhancing AI's problem-solving capabilities beyond mere training data extrapolation. ARC-NCA demonstrates how integrating developmental principles into computational models can foster adaptive reasoning and abstraction. We show that our ARC-NCA proof-of-concept results may be comparable to, and sometimes surpass, that of ChatGPT 4.5, at a fraction of the cost.

Figures

Figures reproduced from arXiv: 2505.08778 by the authors.

Figure 1
Figure 1. Example ARC task, adapted from (Chollet, 2019). [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Diagram depicting one pass of the Growing NCA update step and its neural network model. Adapted from (Mordv [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Diagram depicting one pass of the EngramNCA GeneCA update step and its neural network model. Adapted from [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Diagram depicting one pass of the EngramNCA GenePropCA update step and its neural network model. Adapted [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: One backpropagation step of training EngramNCA for solving ARC problems. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: An example of solution generated by standard [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: An example of solution generated by EngramNCA [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: An example of solution generated by EngramNCA [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: An example of solution generated by EngramNCA [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 12
Figure 12. Figure 12: An example of partial reasoning success in a solution of ARC generated by EngramNCA v4 CA Results Model Solve Rate @ -7 Solve Rate @-6 Cost ($/Task) EngramNCA v3 16.1% 19.8% ≈ 5e-4 EngramNCA v3 Padded 16% 27% ≈ 7e-4 Chat GPT 4.5⋆ 10.3% 10.3% 0.29 [PITH_FULL_IMAGE:fig…
Figure 11
Figure 11. Figure 11: An example of a near solution produced by Engram￾NCA v1 layer of EngramNCA v3, and solving all ARC-AGI prob￾lems by use of maximal padding as described in Dealing with changing grid sizes. CA Architecture Details CA type Augmentations Channels, Hid￾den Size EngramNCA …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 24 canonical work pages

  1. [1]

    ARC Team, P. (2025). Arc prize leaderboard. https://arcprize.org/leaderboard

  2. [2]

    M., Niv, Y., and Barto, A

    Botvinick, M. M., Niv, Y., and Barto, A. G. (2009). Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective. cognition , 113(3):262--280

  3. [3]

    Burtsev, M. (2024). Learning elementary cellular automata with transformers. arXiv preprint arXiv:2412.01417

  4. [4]

    Chollet, F. (2019). On the measure of intelligence. arXiv preprint arXiv:1911.01547

  5. [5]

    Chollet, F. (2025). Openai o3 breakthrough high score on arc-agi-pub. https://arcprize.org/blog/oai-o3-pub-breakthrough

  6. [6]

    Chollet, F., Knoop, M., Kamradt, G., and Landers, B. (2024). Arc prize 2024: Technical report. arXiv preprint arXiv:2412.04604

  7. [7]

    Chollet, F., Knopp, M., and Kamradt, G. (2025). Arc-agi-2. https://arcprize.org/arc-agi \# arc-agi-2

  8. [8]

    and Cully, A

    Faldor, M. and Cully, A. (2024). Cax: Cellular automata accelerated in jax. arXiv preprint arXiv:2410.02651

Show all 36 references
  1. [9]

    Fischer, R., Jakobs, M., M \"u cke, S., and Morik, K. (2020). Solving abstract reasoning tasks with grammatical evolution. In LWDA , pages 6--10

  2. [10]

    Friston, K. (2003). Learning and inference in the brain. Neural Networks , 16(9):1325--1352

  3. [11]

    Gilpin, W. (2019). Cellular automata as convolutional neural networks. Physical Review E , 100(3):032402

  4. [12]

    Guichard, E. (2024). Critically pre-trained neural cellular automata as robot controllers

  5. [13]

    Guichard, E., Reimers, F., Kvalsund, M., Lepper d, M., and Nichele, S. (2025). Engramnca: a neural cellular automaton model of memory transfer. arXiv preprint arXiv:2504.11855

  6. [14]

    Kamradt, G. (2025). Analyzing o3 and o4-mini with arc-agi. https://arcprize.org/blog/analyzing-o3-with-arc-agi

  7. [15]

    O., Glette, K., Pontes-Filho, S., and Lepper d, M

    Kvalsund, M.-K., Ellefsen, K. O., Glette, K., Pontes-Filho, S., and Lepper d, M. E. (2024). Sensor movement drives emergent attention and scalability in active neural cellular automata. bioRxiv , pages 2024--12

  8. [16]

    Langton, C. G. (1990). Computation at the edge of chaos: Phase transitions and emergent computation. Physica D: nonlinear phenomena , 42(1-3):12--37

  9. [17]

    B., Dunsmore, K., and Gavelek, J

    McVee, M. B., Dunsmore, K., and Gavelek, J. R. (2005). Schema theory revisited. Review of educational research , 75(4):531--566

  10. [18]

    Menta, A., Archetti, A., and Matteucci, M. (2024). Latent neural cellular automata for resource-efficient image restoration. In ALIFE 2024: Proceedings of the 2024 Artificial Life Conference . MIT Press

  11. [19]

    Meunier, D., Lambiotte, R., Fornito, A., Ersche, K., and Bullmore, E. T. (2009). Hierarchical modularity in human brain functional networks. Frontiers in neuroinformatics , 3:571

  12. [20]

    Millidge, B., Seth, A., and Buckley, C. L. (2021). Predictive coding: a theoretical and experimental review. arXiv preprint arXiv:2107.12979

  13. [21]

    Mordvintsev, A., Randazzo, E., Niklasson, E., and Levin, M. (2020). Growing neural cellular automata. Distill , 5(2):e23

  14. [22]

    Neumann, K. L. and Kopcha, T. J. (2018). The use of schema theory in learning, design, and technology. TechTrends , 62:429--431

  15. [23]

    B., Risi, S., and Tufte, G

    Nichele, S., Ose, M. B., Risi, S., and Tufte, G. (2017). Ca-neat: evolved compositional pattern producing networks for cellular automata morphogenesis and replication. IEEE Transactions on Cognitive and Developmental Systems , 10(3):687--700

  16. [24]

    Pontes-Filho, S., Nichele, S., and Lepper d, M. (2023). Critical neural cellular automata

  17. [25]

    Pontes-Filho, S., Walker, K., Najarro, E., Nichele, S., and Risi, S. (2022). A single neural cellular automaton for body-brain co-evolution. In Proceedings of the Genetic and Evolutionary Computation Conference Companion , pages 148--151

  18. [26]

    and Mordvintsev, A

    Randazzo, E. and Mordvintsev, A. (2023). Biomaker ca: a biome maker project using cellular automata. arXiv preprint arXiv:2307.09320

  19. [27]

    Reimers, F., Jain, S., Shrestha, A., and Nichele, S. (2023). Pathfinding neural cellular automata with local self-attention. Zenoob

  20. [28]

    Seth, A. K. (2014). The cybernetic bayesian brain. In Open mind . Open MIND. Frankfurt am Main: MIND Group

  21. [29]

    and Levin, M

    Shomrat, T. and Levin, M. (2013). An automated training paradigm reveals long-term memory in planarians and its persistence through head regeneration. Journal of Experimental Biology , 216(20):3799--3810

  22. [30]

    Stovold, J. (2023). Neural cellular automata can respond to signals. In ALIFE 2023: Ghost in the Machine: Proceedings of the 2023 Artificial Life Conference . MIT Press

  23. [31]

    Sudhakaran, S., Grbic, D., Li, S., Katona, A., Najarro, E., Glanois, C., and Risi, S. (2021). Growing 3d artefacts and functional machines with neural cellular automata. In Artificial Life Conference Proceedings 33 , volume 2021, page 108. MIT Press One Rogers Street, Cambridg...

  24. [32]

    Tesfaldet, M., Nowrouzezahrai, D., and Pal, C. (2022). Attention-based neural cellular automata. Advances in Neural Information Processing Systems , 35:8174--8186

  25. [33]

    Variengien, A., Nichele, S., Glover, T., and Pontes-Filho, S. (2021). Towards self-organized control: Using neural cellular automata to robustly control a cart-pole agent. arXiv preprint arXiv:2106.15240

  26. [34]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems , 30

  27. [35]

    Wolfram, S. (1997). New kind of science

  28. [36]

    Xu, Y., Li, W., Vaezipoor, P., Sanner, S., and Khalil, E. B. (2023). Llms and the abstraction and reasoning corpus: Successes, failures, and the importance of object-based representations. arXiv preprint arXiv:2305.18354

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.