REVIEW 3 major objections 5 minor 36 references
ARC-NCA: Towards Developmental Solutions to the Abstraction and Reasoning Corpus
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Neural cellular automata match ChatGPT 4.5 on ARC tasks at 1/1000 the cost.
desk verdict Real first: NCAs on ARC-AGI, but the ChatGPT parity claim rests on unmatched evaluation sets and needs a fix before it holds. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Neural Cellular Automaton (NCA): a grid of cells, each holding a continuous state vector updated by a shared convolutional neural network that reads local neighborhoods, so global patterns emerge from identical local rules. EngramNCA extends this with dual public and private cell states and two cooperating networks, GeneCA and GenePropCA, to encode and propagate hidden memory; the ARC-specific versions add learnable sensing kernels, a toroidal/non-toroidal split, and local patch training. The machinery operates by test-time training: for every task a new CA is trained from scratch on the example pairs by backpropagating pixel-wise MSE through the developmental steps, and a task counts as solved when $\log(\mathrm{MSE}) \le -7$.
What would settle it
Run the official ARC-AGI checker on the outputs of the best EngramNCA and of the union of models for the 262 public tasks, replacing the $\log(\mathrm{MSE}) \le -7$ rule with the checker's accept/reject. If the official solve rate is materially below 12.9% (or below 17.6% for the union), the threshold-based exact-solution claim is overoptimistic; evaluating the NCA and ChatGPT 4.5 on the same identical tasks would settle the parity claim.
Extended reading notes
Core claim
ARC-NCA's central claim is that test-time training of small neural cellular automata—one freshly initialized and trained per ARC problem on its 2–3 training pairs—can produce exact outputs for 10–13% of the 262 non-resizing public tasks, and that combining several NCA variants reaches 17.6%. The best single variant, EngramNCA v3, adds learnable sensing filters and splits toroidal and non-toroidal propagation, solving 12.9% at a threshold of $\log(\mathrm{MSE}) \le -7$. Because the cost per task is about $0.0004, the authors argue this is comparable to, and sometimes better than, ChatGPT 4.5's 10.3% solve rate at roughly 1000 times lower cost.
Load-bearing premise
The central claim rests on treating a pixel-error threshold as equivalent to an exact ARC solution and on treating the public and private ARC evaluation sets as comparable; if either assumption fails, the reported solve rates and the parity with ChatGPT 4.5 would need revision.
Editorial extensions
If this is right
- A cheap per-task developmental learner can match a frontier LLM on a nontrivial slice of ARC-AGI, so solving ARC-style tasks does not necessarily require massive pretraining for every task type.
- Combining diverse NCA variants in unions raises the solve rate from 12.9% to 17.6%, so ensembling developmental programs is a direct and validated lever.
- Loosening the exactness threshold from $-7$ to $-6$ increases individual solve rates by 2–6 percentage points, suggesting that small architecture or post-processing changes may convert near-solutions into exact solutions.
- Larger hidden states and maximal grid padding preserve or improve solve rates (16.1% and up to 27% at the looser threshold), indicating that grid-size-changing ARC tasks are tractable within the NCA paradigm.
- The roughly three-orders-of-magnitude cost advantage makes per-task developmental training practical to run at scale and to combine with more expensive correction mechanisms.
Reading between the lines
- We infer that the parity claim is the least secure part of the paper: the ChatGPT 4.5 score comes from the private evaluation set while the NCA scores come from the public set, and the exact-solution threshold is not checked against the official ARC checker. A like-for-like evaluation on identical tasks would settle how much of the apparent parity is real.
- If the developmental mechanism itself is what carries these solutions, then pretraining NCA on primitive transformations or initializing near criticality could raise solve rates without changing the test-time-training paradigm; this is a direct next experiment the paper leaves open.
- The same per-task developmental training could be applied to ARC-AGI-2, where current AI systems score in single digits; a nonzero solve rate there would show the mechanism transfers beyond the original corpus.
- A hybrid in which an LLM corrects near-solutions might preserve much of the cost advantage, since the NCA would supply most of the structure and only a few LLM calls per task would be needed.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ARC-NCA, a developmental test-time training approach in which a neural cellular automaton (standard NCA or one of several EngramNCA variants) is trained from scratch on the few training examples of each ARC-AGI task and then run to produce an output grid. On the 262 public ARC-AGI tasks that do not require resizing, individual models solve between 6.5% and 12.9% of tasks at an MSE-based threshold, with union rates up to 17.6%; a larger EngramNCA v3 model reaches 16.1%, and a maximally padded version reaches 27% at a loosened threshold. The paper reports a cost of roughly 3e-4 to 5e-4 USD per task, compared with 0.29 USD per task for ChatGPT 4.5, and claims its results are comparable to or better than ChatGPT 4.5 (10.3%) at a fraction of the cost. Qualitative examples and project code and videos are provided.
Significance. If the central comparative claim were supported, this would be a noteworthy result: a simple per-task developmental model achieving ChatGPT-4.5-level performance on ARC-style tasks at roughly a thousandth of the cost. The use of NCAs for 2D ARC tasks is, to my knowledge, novel, and the released code and videos are valuable for reproducibility and for the artificial-life community. The paper is best read as a proof of concept for a developmental baseline. However, the headline comparison to ChatGPT 4.5 is not supported by the reported evaluation protocol, because the baseline score comes from a different (private) evaluation set than the one used for ARC-NCA. The contribution is interesting but needs a fair comparison or a substantially softened claim.
major comments (3)
- [Results, Table 2 (and Table 3, Summary and Discussion)] The central comparative claim is not supported as stated. The footnote to Table 2 states that the ChatGPT 4.5 result (10.3%) was obtained on the ARC-AGI private evaluation set, whereas all ARC-NCA results are from the public evaluation set and, in the main experiments, only from the 262 tasks that do not require resizing. No evidence is offered that the 262-task public subset is comparable in difficulty to the full private set, and the cost comparison in Table 3 inherits this mismatch. The abstract and summary restate the claim that ARC-NCA is 'comparable to, and sometimes surpass[es]' ChatGPT 4.5 without this caveat. Please either evaluate both methods on the same task set (for example, by evaluating ChatGPT on the same 262 public tasks or submitting ARC-NCA outputs to the private leaderboard), or remove the cross-model comparison and reframe the results as a public-subset benchmark.
- [Training, 'Determining the Quality of Solutions'] The solve-rate metric relies on the assertion that log(MSE) <= -7 corresponds to an exact solution and that this threshold was 'experimentally determined,' but no comparison against the official ARC checker or ground-truth labels is shown. All reported solve rates, including the loosened-threshold results in Tables 5, 6, and 8, depend on this unvalidated mapping from MSE to correctness. A calibration analysis is needed: for a sample of threshold-passing outputs, verify exact equality with the official evaluation, and report the false-positive rate. Without this, the internal validity of the reported percentages is not established.
- [Results, 'CA Union Results', Tables 4 and 6] The union solve rates count a task as solved if any model in the union produces a correct output, and the text states that taking the union yields a valid submission because 'two answers may be submitted.' This reasoning is incorrect for tasks with more than one test input: with four models producing one output per test input, only two of those outputs can be submitted per input, and without a selection mechanism the union is an upper bound on achievable performance, not a valid submission score. Please report per-model rates as the primary results and either propose a concrete selection procedure for the union or label the union as an oracle/upper-bound metric.
minor comments (5)
- [Figure 7] The caption for Figure 7 says 'EngramNCA v3,' but the prose and the panel label refer to 'EngramNCA v1'; please correct the mismatch.
- [Table 3] The header contains the typo 'Modle' instead of 'Model,' and the paper uses both 'Chat GPT' and 'ChatGPT'; please standardize.
- [Abstract] The abstract contains a typo ('abilites') and the phrase 'while only few (with median count of three) correct examples are presented' is awkward; please rephrase for clarity.
- [Further Experiments, Table 8] The text says 'All results will be reported on the 262 problems that do not require resizing,' but the maximally padded experiment in Section 8 appears to address all tasks; please state explicitly which task set was used for Table 8 and how many tasks it covers.
- [Future Works] The paper acknowledges that results are documented on single trials. Given that each NCA is trained from scratch, a brief discussion of run-to-run variability (or a small repeated-run study) would strengthen the robustness of the reported solve rates.
Circularity Check
No circularity found: the core results are produced by from-scratch per-task training on the external ARC-AGI benchmark, so the derivation chain is self-contained.
full rationale
The paper's load-bearing claims are evaluated against the ARC-AGI public evaluation set, which is an external benchmark with ground-truth input-output grids. For every task, a new NCA or EngramNCA is trained from scratch on the task's training examples and then evaluated on a held-out test grid; the reported solve rates therefore are not fitted to the conclusion. The EngramNCA architecture is drawn from the authors' prior work, but that citation supplies an architecture and training scheme, not the ARC results themselves, and the paper explicitly tests the suitability of the developmental approach on an external benchmark. The solve threshold log(MSE) <= -7 is a measurement criterion, not a fitted parameter whose value predetermines the comparison with ChatGPT 4.5; whether it exactly matches the official ARC checker is a correctness or validation concern, not a circular one. Similarly, the mismatch between the public 262-task subset and ChatGPT 4.5's private-set score is an apples-to-oranges comparison problem, not a case of the result being equivalent to its inputs by construction. No equation in the paper reduces to a fitted parameter that defines the target being predicted, and no load-bearing argument rests solely on an unverified self-citation. The paper is self-contained against an external benchmark, so the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (4)
- Exact-solution MSE threshold =
log(MSE) <= -7
- Near-solution MSE threshold =
log(MSE) <= -6
- Per-task training schedule =
3000 iterations, LR 1e-3, 66% LR decay at 2000
- Architecture channel and hidden sizes =
50 channels; hidden 64 for NCA, (32,32) or (132,132) for EngramNCA
assumptions (5)
- domain assumption The integer-to-HSL-to-RGB-alpha encoding preserves all task-relevant information
- domain assumption Pixel-wise MSE is a valid training objective and exactness proxy for ARC solutions
- domain assumption The public evaluation set is comparable to the private leaderboard set for the ChatGPT 4.5 comparison
- domain assumption A fresh NCA trained from scratch on 2-3 examples can infer the intended ARC transformation
- domain assumption Small convolutional NCA update rules have sufficient expressive capacity for ARC transformations
Cite this review
Pith. "Pith review of ARC-NCA: Towards Developmental Solutions to the Abstraction and Reasoning Corpus." pith.science (2026). https://pith.science/paper/KH6L5T5J
@misc{pith2026250508778,
author = {Pith},
title = {Pith review of: ARC-NCA: Towards Developmental Solutions to the Abstraction and Reasoning Corpus},
year = {2026},
howpublished = {\url{https://pith.science/paper/KH6L5T5J}},
note = {Machine review of arXiv:2505.08778}
}
read the original abstract
The Abstraction and Reasoning Corpus (ARC), later renamed ARC-AGI, poses a fundamental challenge in artificial general intelligence (AGI), requiring solutions that exhibit robust abstraction and reasoning capabilities across diverse tasks, while only few (with median count of three) correct examples are presented. While ARC-AGI remains very challenging for artificial intelligence systems, it is rather easy for humans. This paper introduces ARC-NCA, a developmental approach leveraging standard Neural Cellular Automata (NCA) and NCA enhanced with hidden memories (EngramNCA) to tackle the ARC-AGI benchmark. NCAs are employed for their inherent ability to simulate complex dynamics and emergent patterns, mimicking developmental processes observed in biological systems. Developmental solutions may offer a promising avenue for enhancing AI's problem-solving capabilities beyond mere training data extrapolation. ARC-NCA demonstrates how integrating developmental principles into computational models can foster adaptive reasoning and abstraction. We show that our ARC-NCA proof-of-concept results may be comparable to, and sometimes surpass, that of ChatGPT 4.5, at a fraction of the cost.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
ARC Team, P. (2025). Arc prize leaderboard. https://arcprize.org/leaderboard
work page 2025
-
[2]
M., Niv, Y., and Barto, A
Botvinick, M. M., Niv, Y., and Barto, A. G. (2009). Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective. cognition , 113(3):262--280
2009
-
[3]
Burtsev, M. (2024). Learning elementary cellular automata with transformers. arXiv preprint arXiv:2412.01417
work page Pith review arXiv 2024
-
[4]
Chollet, F. (2019). On the measure of intelligence. arXiv preprint arXiv:1911.01547
arXiv 2019
-
[5]
Chollet, F. (2025). Openai o3 breakthrough high score on arc-agi-pub. https://arcprize.org/blog/oai-o3-pub-breakthrough
work page 2025
-
[6]
Chollet, F., Knoop, M., Kamradt, G., and Landers, B. (2024). Arc prize 2024: Technical report. arXiv preprint arXiv:2412.04604
arXiv 2024
-
[7]
Chollet, F., Knopp, M., and Kamradt, G. (2025). Arc-agi-2. https://arcprize.org/arc-agi \# arc-agi-2
work page 2025
-
[8]
Faldor, M. and Cully, A. (2024). Cax: Cellular automata accelerated in jax. arXiv preprint arXiv:2410.02651
arXiv 2024
Show all 36 references
-
[9]
Fischer, R., Jakobs, M., M \"u cke, S., and Morik, K. (2020). Solving abstract reasoning tasks with grammatical evolution. In LWDA , pages 6--10
2020
-
[10]
Friston, K. (2003). Learning and inference in the brain. Neural Networks , 16(9):1325--1352
2003
-
[11]
Gilpin, W. (2019). Cellular automata as convolutional neural networks. Physical Review E , 100(3):032402
2019
-
[12]
Guichard, E. (2024). Critically pre-trained neural cellular automata as robot controllers
2024
-
[13]
Guichard, E., Reimers, F., Kvalsund, M., Lepper d, M., and Nichele, S. (2025). Engramnca: a neural cellular automaton model of memory transfer. arXiv preprint arXiv:2504.11855
2025 arXiv
-
[14]
Kamradt, G. (2025). Analyzing o3 and o4-mini with arc-agi. https://arcprize.org/blog/analyzing-o3-with-arc-agi
2025
-
[15]
O., Glette, K., Pontes-Filho, S., and Lepper d, M
Kvalsund, M.-K., Ellefsen, K. O., Glette, K., Pontes-Filho, S., and Lepper d, M. E. (2024). Sensor movement drives emergent attention and scalability in active neural cellular automata. bioRxiv , pages 2024--12
2024
-
[16]
Langton, C. G. (1990). Computation at the edge of chaos: Phase transitions and emergent computation. Physica D: nonlinear phenomena , 42(1-3):12--37
1990
-
[17]
B., Dunsmore, K., and Gavelek, J
McVee, M. B., Dunsmore, K., and Gavelek, J. R. (2005). Schema theory revisited. Review of educational research , 75(4):531--566
2005
-
[18]
Menta, A., Archetti, A., and Matteucci, M. (2024). Latent neural cellular automata for resource-efficient image restoration. In ALIFE 2024: Proceedings of the 2024 Artificial Life Conference . MIT Press
2024
-
[19]
Meunier, D., Lambiotte, R., Fornito, A., Ersche, K., and Bullmore, E. T. (2009). Hierarchical modularity in human brain functional networks. Frontiers in neuroinformatics , 3:571
2009
-
[20]
Millidge, B., Seth, A., and Buckley, C. L. (2021). Predictive coding: a theoretical and experimental review. arXiv preprint arXiv:2107.12979
2021 arXiv
-
[21]
Mordvintsev, A., Randazzo, E., Niklasson, E., and Levin, M. (2020). Growing neural cellular automata. Distill , 5(2):e23
2020
-
[22]
Neumann, K. L. and Kopcha, T. J. (2018). The use of schema theory in learning, design, and technology. TechTrends , 62:429--431
2018
-
[23]
B., Risi, S., and Tufte, G
Nichele, S., Ose, M. B., Risi, S., and Tufte, G. (2017). Ca-neat: evolved compositional pattern producing networks for cellular automata morphogenesis and replication. IEEE Transactions on Cognitive and Developmental Systems , 10(3):687--700
2017
-
[24]
Pontes-Filho, S., Nichele, S., and Lepper d, M. (2023). Critical neural cellular automata
2023
-
[25]
Pontes-Filho, S., Walker, K., Najarro, E., Nichele, S., and Risi, S. (2022). A single neural cellular automaton for body-brain co-evolution. In Proceedings of the Genetic and Evolutionary Computation Conference Companion , pages 148--151
2022
-
[26]
and Mordvintsev, A
Randazzo, E. and Mordvintsev, A. (2023). Biomaker ca: a biome maker project using cellular automata. arXiv preprint arXiv:2307.09320
2023 arXiv
-
[27]
Reimers, F., Jain, S., Shrestha, A., and Nichele, S. (2023). Pathfinding neural cellular automata with local self-attention. Zenoob
2023
-
[28]
Seth, A. K. (2014). The cybernetic bayesian brain. In Open mind . Open MIND. Frankfurt am Main: MIND Group
2014
-
[29]
and Levin, M
Shomrat, T. and Levin, M. (2013). An automated training paradigm reveals long-term memory in planarians and its persistence through head regeneration. Journal of Experimental Biology , 216(20):3799--3810
2013
-
[30]
Stovold, J. (2023). Neural cellular automata can respond to signals. In ALIFE 2023: Ghost in the Machine: Proceedings of the 2023 Artificial Life Conference . MIT Press
2023
-
[31]
Sudhakaran, S., Grbic, D., Li, S., Katona, A., Najarro, E., Glanois, C., and Risi, S. (2021). Growing 3d artefacts and functional machines with neural cellular automata. In Artificial Life Conference Proceedings 33 , volume 2021, page 108. MIT Press One Rogers Street, Cambridg...
2021
-
[32]
Tesfaldet, M., Nowrouzezahrai, D., and Pal, C. (2022). Attention-based neural cellular automata. Advances in Neural Information Processing Systems , 35:8174--8186
2022
-
[33]
Variengien, A., Nichele, S., Glover, T., and Pontes-Filho, S. (2021). Towards self-organized control: Using neural cellular automata to robustly control a cart-pole agent. arXiv preprint arXiv:2106.15240
2021 arXiv
-
[34]
N., Kaiser, ., and Polosukhin, I
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems , 30
2017
-
[35]
Wolfram, S. (1997). New kind of science
1997
-
[36]
Xu, Y., Li, W., Vaezipoor, P., Sanner, S., and Khalil, E. B. (2023). Llms and the abstraction and reasoning corpus: Successes, failures, and the importance of object-based representations. arXiv preprint arXiv:2305.18354
2023 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.