{"id":"6d774aa8-4b01-4f25-b7c0-e19476f01c96","arxiv_id":"2505.08778","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"ARC-NCA shows that per-task test-time training of Neural Cellular Automata can solve about 13 percent of a 262-task ARC-AGI subset, but the claimed parity with ChatGPT 4.5 relies on results from different benchmark splits.","lead":"This paper trains Neural Cellular Automata from scratch on each ARC-AGI puzzle and reports solve rates around 10 to 13 percent on a 262-task subset, which it compares to ChatGPT 4.5 at roughly 1000 times lower cost. The comparison is weakened by the fact that the two systems were evaluated on different ARC evaluation sets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Parity claim is not apples-to-apples: ARC-NCA is scored only on 262 public tasks while ChatGPT 4.5's 10.3% is from the private set, as the paper's own Table 2 footnote admits.","rationale":"I read the paper in good faith: the proof-of-concept is real and the qualitative developmental solutions are the strongest part. The most defensible claim is that NCAs can solve a nontrivial set of ARC tasks via per-task test-time training. However, the paper's own Table 2 footnote explicitly flags that the ChatGPT 4.5 comparison uses the private evaluation set while all ARC-NCA numbers are public-set only, and the methods text says 'All results will be reported on the 262 problems that do not require resizing.' That is a self-admitted limitation, and it directly affects the abstract's headline. The MSE-threshold concern is also valid and worth verifying, but it is secondary to the set mismatch. My single concrete check, same tasks, same solution checker, and same submission rule, would settle both issues in one pass. I therefore keep the reader's conditional verdict: the paper is a useful baseline contribution once the comparison is rerun on an apples-to-apples basis.","tokens_in":11853,"tokens_out":5498,"duration_ms":55671,"concrete_test":"Run a same-set comparison: (1) evaluate ARC-NCA on all 400 public ARC-AGI tasks (including resizing tasks, using the maximal-padding protocol already described) and score outputs with the official ARC-AGI exact-match checker rather than log(MSE)<=-7; (2) run ChatGPT 4.5 on the same 262-task public subset used in the paper, keeping the same two-answer submission rule. If ChatGPT 4.5's solve rate on the identical 262-task subset is at or above ARC-NCA's, the parity claim fails; if ARC-NCA remains higher on the same subset and the official-checker rate matches the threshold-based rate, the claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim--that ARC-NCA is comparable to or better than ChatGPT 4.5 at 1000x lower cost--is not supported by the reported evaluation protocol. Table 2's footnote states that the ChatGPT 4.5 result (10.3%) was obtained on the ARC-AGI private evaluation set, whereas all ARC-NCA experiments were run on the public evaluation set and only on the 262 tasks that do not require resizing. The abstract and summary then restate 'comparable to, and sometimes surpassing' without this caveat. Because the paper provides no evidence that the 262-task public subset is comparable in difficulty to the full private set, the cross-model comparison is invalid, and the cost comparison inherits that invalidity. A secondary but real concern is the solve-rate threshold: log(MSE)<=-7 is asserted to be 'experimentally determined' to correspond to exact solutions, but no comparison against the official ARC checker is shown. This threshold affects the internal validity of the reported solve rates; the public/private mismatch would remain even if the threshold were exact. Both issues are concrete and fixable, but the evaluation mismatch is the more load-bearing because it directly undermines the headline comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ARC-NCA, a developmental test-time training approach in which a neural cellular automaton (standard NCA or one of several EngramNCA variants) is trained from scratch on the few training examples of each ARC-AGI task and then run to produce an output grid. On the 262 public ARC-AGI tasks that do not require resizing, individual models solve between 6.5% and 12.9% of tasks at an MSE-based threshold, with union rates up to 17.6%; a larger EngramNCA v3 model reaches 16.1%, and a maximally padded version reaches 27% at a loosened threshold. The paper reports a cost of roughly 3e-4 to 5e-4 USD per task, compared with 0.29 USD per task for ChatGPT 4.5, and claims its results are comparable to or better than ChatGPT 4.5 (10.3%) at a fraction of the cost. Qualitative examples and project code and videos are provided.","tokens_in":12044,"tokens_out":6318,"duration_ms":56007,"significance":"If the central comparative claim were supported, this would be a noteworthy result: a simple per-task developmental model achieving ChatGPT-4.5-level performance on ARC-style tasks at roughly a thousandth of the cost. The use of NCAs for 2D ARC tasks is, to my knowledge, novel, and the released code and videos are valuable for reproducibility and for the artificial-life community. The paper is best read as a proof of concept for a developmental baseline. However, the headline comparison to ChatGPT 4.5 is not supported by the reported evaluation protocol, because the baseline score comes from a different (private) evaluation set than the one used for ARC-NCA. The contribution is interesting but needs a fair comparison or a substantially softened claim.","major_comments":[{"comment":"The central comparative claim is not supported as stated. The footnote to Table 2 states that the ChatGPT 4.5 result (10.3%) was obtained on the ARC-AGI private evaluation set, whereas all ARC-NCA results are from the public evaluation set and, in the main experiments, only from the 262 tasks that do not require resizing. No evidence is offered that the 262-task public subset is comparable in difficulty to the full private set, and the cost comparison in Table 3 inherits this mismatch. The abstract and summary restate the claim that ARC-NCA is 'comparable to, and sometimes surpass[es]' ChatGPT 4.5 without this caveat. Please either evaluate both methods on the same task set (for example, by evaluating ChatGPT on the same 262 public tasks or submitting ARC-NCA outputs to the private leaderboard), or remove the cross-model comparison and reframe the results as a public-subset benchmark.","section":"Results, Table 2 (and Table 3, Summary and Discussion)"},{"comment":"The solve-rate metric relies on the assertion that log(MSE) <= -7 corresponds to an exact solution and that this threshold was 'experimentally determined,' but no comparison against the official ARC checker or ground-truth labels is shown. All reported solve rates, including the loosened-threshold results in Tables 5, 6, and 8, depend on this unvalidated mapping from MSE to correctness. A calibration analysis is needed: for a sample of threshold-passing outputs, verify exact equality with the official evaluation, and report the false-positive rate. Without this, the internal validity of the reported percentages is not established.","section":"Training, 'Determining the Quality of Solutions'"},{"comment":"The union solve rates count a task as solved if any model in the union produces a correct output, and the text states that taking the union yields a valid submission because 'two answers may be submitted.' This reasoning is incorrect for tasks with more than one test input: with four models producing one output per test input, only two of those outputs can be submitted per input, and without a selection mechanism the union is an upper bound on achievable performance, not a valid submission score. Please report per-model rates as the primary results and either propose a concrete selection procedure for the union or label the union as an oracle/upper-bound metric.","section":"Results, 'CA Union Results', Tables 4 and 6"}],"minor_comments":[{"comment":"The caption for Figure 7 says 'EngramNCA v3,' but the prose and the panel label refer to 'EngramNCA v1'; please correct the mismatch.","section":"Figure 7"},{"comment":"The header contains the typo 'Modle' instead of 'Model,' and the paper uses both 'Chat GPT' and 'ChatGPT'; please standardize.","section":"Table 3"},{"comment":"The abstract contains a typo ('abilites') and the phrase 'while only few (with median count of three) correct examples are presented' is awkward; please rephrase for clarity.","section":"Abstract"},{"comment":"The text says 'All results will be reported on the 262 problems that do not require resizing,' but the maximally padded experiment in Section 8 appears to address all tasks; please state explicitly which task set was used for Table 8 and how many tasks it covers.","section":"Further Experiments, Table 8"},{"comment":"The paper acknowledges that results are documented on single trials. Given that each NCA is trained from scratch, a brief discussion of run-to-run variability (or a small repeated-run study) would strengthen the robustness of the reported solve rates.","section":"Future Works"}],"recommendation":"major_revision","confidential_remarks":"The main obstacle to publication is the invalid cross-evaluation-set comparison with ChatGPT 4.5, which the authors themselves acknowledge in the Table 2 footnote. If the authors can supply a fair comparison on a shared task set, or remove the comparative claim and reposition the paper as an exploratory proof of concept on the public subset, the work could be publishable after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real first — NCA applied to ARC-AGI, with per-task test-time training, solving 10-13% of the 262 non-resizing public tasks. That is a useful data point. But the paper's headline claim that ARC-NCA is 'comparable to, and sometimes surpassing' ChatGPT 4.5 is not supported as stated, because the ChatGPT number comes from the private leaderboard while the NCA results are on a filtered public subset. The authors put a footnote in Table 2 admitting this, but the abstract and discussion repeat the parity claim without the caveat. That is the central fix the paper needs.\n\nWhat's genuinely good: the augmentations (learnable sensing, toroidal split, patch training) are thoughtful and specific to ARC's structure. The qualitative videos show real developmental behavior — cells growing, overshooting, then correcting. The cost estimate, about a thousandfold cheaper per task than the ChatGPT cited, is plausible and worth noting if the solve rates hold. And the paper is candid about single trials and the need for official evaluation submission.\n\nThe soft spots beyond the eval-set mismatch: the log(MSE) ≤ −7 threshold is called 'experimentally determined' but there is no comparison to the official ARC checker, so the absolute solve rates could be off. Multiple seeds are not reported. Everything is on 262 tasks; resizing tasks are excluded. These are all addressable with more work. None of them by itself kills the contribution, but together they mean the paper should be read as a proof-of-concept, not a benchmark entry.\n\nI'd send this to review. It's not a desk-reject: the idea is new, the execution is honest enough, and the flaws are concrete and fixable. A serious referee would ask for the comparison to be redone (or at least clearly reframed), the threshold to be validated, and at least a couple of seeds. But the core — NCAs can do a non-trivial slice of ARC — will likely survive those fixes. The paper is mainly for the artificial life and ARC research communities; it gives them a cheap baseline and a new direction rather than a SOTA result.","headline":"Real first: NCAs on ARC-AGI, but the ChatGPT parity claim rests on unmatched evaluation sets and needs a fix before it holds.","tokens_in":12672,"tokens_out":3503,"would_cite":true,"duration_ms":32333,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Neural cellular automata match ChatGPT 4.5 on ARC tasks at 1/1000 the cost.","keywords":["Abstraction and Reasoning Corpus","Neural Cellular Automata","EngramNCA","developmental computation","test-time training","ARC-AGI","few-shot visual reasoning","cellular automata"],"falsifier":"Run the official ARC-AGI checker on the outputs of the best EngramNCA and of the union of models for the 262 public tasks, replacing the $\\log(\\mathrm{MSE}) \\le -7$ rule with the checker's accept/reject. If the official solve rate is materially below 12.9% (or below 17.6% for the union), the threshold-based exact-solution claim is overoptimistic; evaluating the NCA and ChatGPT 4.5 on the same identical tasks would settle the parity claim.","tokens_in":11605,"feed_emoji":"🧫","tokens_out":8394,"duration_ms":71705,"temperature":0.7,"pith_summary":"The paper asks whether a developmental model—a neural cellular automaton grown anew for each puzzle—can handle the few-shot visual reasoning of the ARC-AGI benchmark. It reports that a standard NCA solves 10.7% of the 262 non-resizing public tasks, the best EngramNCA variant solves 12.9%, and the union of variants reaches 17.6%, while ChatGPT 4.5 scores 10.3% on the private evaluation set. The same runs cost roughly $0.0003–$0.0005 per task versus about $0.29 for the LLM. A sympathetic reader should take this as a proof of concept that developmental local-update rules, not just large pretrained transformers, can produce exact solutions to some ARC tasks from a handful of examples.","feed_headline":"Cellular automata match ChatGPT 4.5 on ARC at 1/1000 the cost","feed_subtitle":"Per-task training on 2-3 examples solves up to 17.6 percent of public ARC-AGI tasks for about $0.0005 each.","key_machinery":"The central object is the Neural Cellular Automaton (NCA): a grid of cells, each holding a continuous state vector updated by a shared convolutional neural network that reads local neighborhoods, so global patterns emerge from identical local rules. EngramNCA extends this with dual public and private cell states and two cooperating networks, GeneCA and GenePropCA, to encode and propagate hidden memory; the ARC-specific versions add learnable sensing kernels, a toroidal/non-toroidal split, and local patch training. The machinery operates by test-time training: for every task a new CA is trained from scratch on the example pairs by backpropagating pixel-wise MSE through the developmental steps, and a task counts as solved when $\\log(\\mathrm{MSE}) \\le -7$.","core_discovery":"ARC-NCA's central claim is that test-time training of small neural cellular automata—one freshly initialized and trained per ARC problem on its 2–3 training pairs—can produce exact outputs for 10–13% of the 262 non-resizing public tasks, and that combining several NCA variants reaches 17.6%. The best single variant, EngramNCA v3, adds learnable sensing filters and splits toroidal and non-toroidal propagation, solving 12.9% at a threshold of $\\log(\\mathrm{MSE}) \\le -7$. Because the cost per task is about $0.0004, the authors argue this is comparable to, and sometimes better than, ChatGPT 4.5's 10.3% solve rate at roughly 1000 times lower cost.","pith_inferences":["We infer that the parity claim is the least secure part of the paper: the ChatGPT 4.5 score comes from the private evaluation set while the NCA scores come from the public set, and the exact-solution threshold is not checked against the official ARC checker. A like-for-like evaluation on identical tasks would settle how much of the apparent parity is real.","If the developmental mechanism itself is what carries these solutions, then pretraining NCA on primitive transformations or initializing near criticality could raise solve rates without changing the test-time-training paradigm; this is a direct next experiment the paper leaves open.","The same per-task developmental training could be applied to ARC-AGI-2, where current AI systems score in single digits; a nonzero solve rate there would show the mechanism transfers beyond the original corpus.","A hybrid in which an LLM corrects near-solutions might preserve much of the cost advantage, since the NCA would supply most of the structure and only a few LLM calls per task would be needed."],"forward_implications":["A cheap per-task developmental learner can match a frontier LLM on a nontrivial slice of ARC-AGI, so solving ARC-style tasks does not necessarily require massive pretraining for every task type.","Combining diverse NCA variants in unions raises the solve rate from 12.9% to 17.6%, so ensembling developmental programs is a direct and validated lever.","Loosening the exactness threshold from $-7$ to $-6$ increases individual solve rates by 2–6 percentage points, suggesting that small architecture or post-processing changes may convert near-solutions into exact solutions.","Larger hidden states and maximal grid padding preserve or improve solve rates (16.1% and up to 27% at the looser threshold), indicating that grid-size-changing ARC tasks are tractable within the NCA paradigm.","The roughly three-orders-of-magnitude cost advantage makes per-task developmental training practical to run at scale and to combine with more expensive correction mechanisms."],"supporting_citations":[{"why":"Defines the ARC/ARC-AGI benchmark, its few-example structure, and the evaluation setup the paper targets.","marker":"Chollet, 2019"},{"why":"Supplies the Growing NCA architecture, update rule, and pixel-wise MSE training procedure that ARC-NCA adapts.","marker":"Mordvintsev et al., 2020"},{"why":"Introduces EngramNCA with dual public/private cell states and the GeneCA/GenePropCA training that ARC-NCA co-optimizes.","marker":"Guichard et al., 2025"},{"why":"Provides the ChatGPT 4.5 solve rate (10.3%) and per-task cost ($0.29) used as the comparison baseline.","marker":"ARC Team, 2025"}],"fun_headline_variants":["Tiny automata solve ARC for $0.0004, rivaling ChatGPT 4.5","Per-task NCA training hits 17.6% ARC solve rate at $0.0004","ARC-AGI: cellular automata match ChatGPT 4.5 for $0.0004 per task","Developmental NCAs solve ARC at 1000x less cost than ChatGPT"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on treating a pixel-error threshold as equivalent to an exact ARC solution and on treating the public and private ARC evaluation sets as comparable; if either assumption fails, the reported solve rates and the parity with ChatGPT 4.5 would need revision.","fun_headline_variants_meta":{"raw":{"variants":["Tiny automata solve ARC for $0.0004, rivaling ChatGPT 4.5","Per-task NCA training hits 17.6% ARC solve rate at $0.0004","ARC-AGI: cellular automata match ChatGPT 4.5 for $0.0004 per task","Developmental NCAs solve ARC at 1000x less cost than ChatGPT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001562,"raw_usage":{"total_tokens":6223,"prompt_tokens":913,"completion_tokens":5310,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":5210}},"tokens_in":529,"tokens_out":5310,"duration_ms":39984,"temperature":1.0,"reasoning_tokens":5210,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:45:57.504572+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the official ARC-AGI checker on the outputs of the best EngramNCA and of the union of models for the 262 public tasks, replacing the $\\log(\\mathrm{MSE}) \\le -7$ rule with the checker's accept/reject. If the official solve rate is materially below 12.9% (or below 17.6% for the union), the threshold-based exact-solution claim is overoptimistic; evaluating the NCA and ChatGPT 4.5 on the same identical tasks would settle the parity claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ChatGPT 4.5 solve rate (10.3%) and per-task cost ($0.29) used as the comparison baseline."}],"review_version":1}