{"id":"fc5ab8e5-7a6a-4bfb-b98e-d52728b754f9","arxiv_id":"2505.13391","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"PoNG, an architecture with group-convolution pathways, achieves state-of-the-art accuracy on several abstract visual reasoning benchmarks, though its edge on i.i.d. tasks largely comes from rule-label supervision.","lead":"PoNG, a new neural network for abstract visual reasoning, uses parallel pathways with group convolutions to beat prior models on several out-of-distribution benchmarks, including real-world visual analogies. Its gains on synthetic tasks depend heavily on auxiliary rule-label supervision, a dependency the paper does not fully account for in its baseline comparisons.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PoNG's reported gains may reflect ground-truth rule supervision rather than the proposed architecture; the paper never states whether baselines received rule labels, and removing both rule heads drops I-RAVEN to 79.7%, below RelBase's 89.6%.","rationale":"I agree with the reader's identification of the weakest assumption. The central claim is that PoNG's architecture generalizes across diverse abstract visual reasoning tasks, and the most striking evidence is the A-I-RAVEN and I-RAVEN results. However, the model's loss explicitly uses ground-truth rule labels with large coefficients, and the ablation removing both rule heads reverses the ranking against baselines. Because the paper does not disclose whether the baselines had any form of rule supervision, the comparison may overstate the architectural contribution. This is a correctness-risk concern, not a disagreement with consensus: the issue is that the independent variable (architecture) is entangled with the supervision signal. Other potential issues, such as reporting best-of-3 on VASR or finishing second on PGM average, are minor and do not affect the verdict. The paper should clarify baseline supervision and add matched-supervision comparisons; therefore, the reader's CONDITIONAL verdict remains appropriate.","tokens_in":17194,"tokens_out":7131,"duration_ms":67111,"concrete_test":"Inspect the released training code and configuration files of every baseline (SCL, RelBase, DRNet, PredRNet, etc.) for any use of ground-truth rule labels r, such as a BCE or contrastive loss on a dr-dimensional rule vector. If no baseline uses r, then the Tables 1–3 comparisons are unmatched, and the β=γ=0 row of Table 6 (79.7% vs. RelBase 89.6% on I-RAVEN) already demonstrates that the architecture alone is not state of the art.","verdict_should_be":"UNCHANGED","load_bearing_attack":"PoNG is trained with the joint loss L = CE(σ(ŷ), y) + β BCE(ζ(b_r1), r) + γ BCE(ζ(b_r2), r) (Section 3), where r is the ground-truth multi-hot rule vector and β=25, γ=5. The auxiliary rule heads P_r1 and P_r2 are therefore part of the model's supervision. Table 6 shows that removing both heads (β=γ=0) drops I-RAVEN accuracy from 95.9% to 79.7%, below RelBase (89.6%) and SCL (83.4%); on I-RAVEN-Mesh the drop is from 89.3% to 32.7%, far below RelBase (84.9%). The paper nowhere states whether the 13 baselines were given rule-label supervision, and standard AVR baselines such as WReN, RelBase, SCL, DRNet, and PredRNet are ordinarily trained only on answer labels. If those baselines lacked r, the headline comparisons in Tables 1–3 confound the proposed architecture with unequal access to the exact abstract rules. The authors' own ablation text acknowledges the 'high relevance' of the auxiliary signal, so the architecture-only contribution is not established by the reported experiments.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PoNG, a neural architecture for abstract visual reasoning that combines group convolution, normalization, and parallel pathways. The model is evaluated on four RPM-style benchmarks (I-RAVEN, I-RAVEN-Mesh, A-I-RAVEN, PGM) and two visual analogy datasets (VAP, VASR), reporting accuracy improvements over 13 baselines in several out-of-distribution regimes. The training loss includes two auxiliary heads that predict ground-truth rule vectors r, with weights β=25 and γ=5. An ablation removing both heads shows substantial performance drops. The paper provides public code and fixed random seeds.","tokens_in":17523,"tokens_out":5322,"duration_ms":46267,"significance":"If the reported gains were attributable to the architecture alone, PoNG would be a strong contribution to visual reasoning, especially for out-of-distribution generalization across synthetic and real-world tasks. Strengths include reproducible code, transparent ablations, and evaluation on multiple benchmarks. However, the significance is substantially weakened by the unequal supervision between PoNG and baselines: the rule-label auxiliary loss is not accounted for in the comparisons, and the paper's own ablation shows that without it PoNG underperforms several baselines. The claims need to be re-framed or re-validated with a controlled comparison.","major_comments":[{"comment":"The joint loss includes β·BCE(ζ(b_{r1}), r) + γ·BCE(ζ(b_{r2}), r) with β=25, γ=5, where r is the ground-truth multi-hot rule vector. The paper never states whether the 13 baselines received rule-label supervision, and the standard implementations of WReN, RelBase, SCL, DRNet, and PredRNet are trained only on answer labels. This confounds the architecture's contribution with access to exact abstract rules. Table 6 shows the magnitude: removing both rule heads (β=γ=0) lowers I-RAVEN accuracy from 95.9% to 79.7% and I-RAVEN-Mesh from 89.3% to 32.7%. The authors should disclose this supervision difference and provide a controlled comparison, e.g., by training baselines with the same rule-head loss or by comparing PoNG without rule heads on the same terms.","section":"Section 3, Eq. (1); Tables 1 and 6"},{"comment":"The answer-only variant (β=γ=0) is the appropriate architecture-only baseline. Under that ablation, PoNG achieves 79.7% on I-RAVEN and 32.7% on I-RAVEN-Mesh, which are below RelBase (89.6%, 84.9%) and SCL (83.4%, 80.9%). Consequently, the claim in the abstract and Section 4.3 that PoNG outperforms the existing literature methods is not supported for the architecture alone. The central claim needs to be qualified to the full model with auxiliary rule supervision, or the experiments need to establish that the architecture is competitive without that supervision.","section":"Section 4.3, Tables 1 and 6"}],"minor_comments":[{"comment":"The dimensionality dr is specified for I-RAVEN, A-I-RAVEN, I-RAVEN-Mesh, PGM, and VAP, but not for VASR; please clarify whether the rule heads are used for VASR and, if so, how the rule vector r is defined for real-world images.","section":"Section 3, last paragraph"},{"comment":"PoNG results on VASR are reported as best-of-3 and compared to baselines from the original paper, which may be single runs; please report mean±std as in other tables and state whether the baselines used multiple seeds.","section":"Section 4.3, Table 5"},{"comment":"The column I-RAVEN† reports results by the original model authors with model-specific setups, whereas other columns use the authors' own experimental setup; please clarify whether the comparison is apples-to-apples in terms of data splits and hyperparameter tuning.","section":"Section 4.2 and Table 1"},{"comment":"I-RAVEN-Mesh was originally introduced for transfer learning; using it for standard supervised learning is a protocol choice and should be explicitly justified or noted in the experimental setup.","section":"Section 4.1"},{"comment":"The error analysis is informative, but the table reports aggregated calibration metrics without confidence intervals; providing standard deviations or significance tests would strengthen the claim that TCN helps rule generalization.","section":"Appendix B, Table 11"}],"recommendation":"major_revision","confidential_remarks":"The supervision confound is the central issue: the authors' own ablation shows that without rule labels PoNG falls below several baselines, yet the paper does not disclose whether baselines had access to the same rule supervision. This should be resolved before publication, either with a controlled comparison or with appropriately qualified claims. I would encourage the editor to request such a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a well-executed empirical paper on abstract visual reasoning, with a new architecture (PoNG) that combines group convolution and TCN pathways plus a target-conditioned rule head. It reports strong results across several benchmarks, including real-world VASR. But the headline comparisons are clouded by unequal supervision: PoNG is trained with ground-truth rule vectors r through two auxiliary heads with balancing coefficients β=25 and γ=5, and the paper never says whether the 13 baselines received similar rule labels. The ablation makes the cost explicit: removing both rule heads drops I-RAVEN from 95.9% to 79.7%, below RelBase (89.6%) and SCL (83.4%); on I-RAVEN-Mesh it falls to 32.7% versus RelBase's 84.9%. That is a load-bearing confound for the i.i.d. comparisons. The authors acknowledge the 'high relevance' of the auxiliary signal but do not confront its implication: the architecture-only contribution is not established by those tables.\n\nWhat is good: the model is clearly described, the code is public, the ablation is thorough, and the VASR results are genuinely interesting—PoNG beats Supervised Concat by 5.8 points on random distractors and by 14.6 points on difficult distractors. The group-pair convolution (P4) and TCN are a reasonable new combination, and the target-conditioned rule head is a small but nice twist. The error analysis with concrete failure cases adds value. This is competent, honest work.\n\nSoft spots: beyond the baseline mismatch, the paper never states whether VASR even uses rule labels. If VASR lacks rule vectors, those gains are not confounded, but the paper should say so. Also, two benchmarks (A-I-RAVEN and I-RAVEN-Mesh) are self-authored; that is not circular, but it is a reason to lean on PGM and VAP for independent confirmation, and those results are more mixed—PoNG is second to DRNet on PGM average and roughly tied with PredRNet on VAP.\n\nBottom line: the architecture has merit, but the central generalization claim is not proven with the given comparisons. A rigorous revision with baselines trained under matched supervision, or an ablation that cleanly separates the rule-head contribution, would settle it. I would send this to peer review: the code, ablations, and multi-benchmark evaluation make it a useful empirical contribution, and the supervision confound is fixable in revision.","headline":"Solid AVR architecture paper whose headline gains are partly explained by ground-truth rule supervision; a matched-supervision comparison is needed before claiming the architecture generalizes.","tokens_in":18012,"tokens_out":3852,"would_cite":true,"duration_ms":31703,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single architecture, the Pathways of Normalized Group Convolution model (PoNG), claims to generalize across Raven's Progressive Matrices and visual analogy benchmarks, synthetic and real, and reports top results in several…","keywords":["abstract visual reasoning","Raven's Progressive Matrices","out-of-distribution generalization","group convolution","visual analogy","temporal context normalization","auxiliary rule supervision"],"falsifier":"Train RelBase or SCL under the identical loss, including the two rule heads with $\\beta=25$ and $\\gamma=5$, on A-I-RAVEN; if A/Color accuracy rises from about 67% to about 80%, then the auxiliary rule supervision, not the group-convolution pathways, explains the gain. If the gap persists, the pathways carry the generalization.","tokens_in":17040,"feed_emoji":"🧩","tokens_out":7942,"duration_ms":69234,"temperature":0.7,"pith_summary":"The paper introduces PoNG, a neural network for abstract visual reasoning puzzles such as Raven's Progressive Matrices, where the solver must pick the panel that completes a pattern. PoNG's claim is that its design, built around normalized group convolutions that process rows and row-pairs of the puzzle grid, transfers across many such tasks and generalizes to test puzzles whose rules were not seen in training. The reported numbers support the claim on several benchmarks: 80.3% on A/Color versus 70.0% for the best baseline, and 91.8% on real-world visual analogies versus 70.5% for Supervised Concat. A fair reader should take away that a single parameter-efficient architecture can top most baselines on both synthetic and real-world analogy reasoning, with the caveat that the training loss uses the puzzle's ground-truth rules as auxiliary supervision.","feed_headline":"Model hits 91.8% on real-world visual analogies","feed_subtitle":"PoNG also out-generalizes prior AI on Raven-matrix tests, beating the best baseline by up to 15.6 points.","key_machinery":"The load-bearing component is the Pathways block inside each reasoning block, with four parallel pathways P1 through P4. P3 and P4 use the named group convolution and group-pair convolution operators: they split the stacked panel embeddings into groups corresponding to matrix rows or row pairs, apply a shared-weight 1D convolution to each group, then temporal context normalization (TCN), and sum the groups. TCN, taken from the paper's reference Webb et al. (2020), preserves relations between representations within a group while discarding absolute magnitude, which the ablation study ties to extrapolating rules to held-out attributes. The two rule heads, one aggregating all answers and one weighting answers by the predicted answer distribution, inject the known rule structure into training through binary cross-entropy terms.","core_discovery":"PoNG is a two-stage model: a panel encoder turns each image into a vector, and a reasoner scores each candidate answer. The reasoner stacks panel embeddings with a learned position embedding and passes them through three blocks, each containing four parallel pathways: pointwise mixing (P1), local 1D convolutions (P2), group convolution (P3), and group-pair convolution (P4), with the last two augmented by temporal context normalization (TCN). Grouping is chosen so that P3 operates on rows of the matrix and P4 on pairs of rows. Three prediction heads produce the answer score and two rule predictions; the loss combines cross-entropy for the answer with binary cross-entropy for the two rule heads, with coefficients $\\beta=25$ and $\\gamma=5$. The central claim is that this combination generalizes across benchmark families: on A/Color, PoNG scores 80.3% against a 70.0% best baseline; on A/Color-D3 it leads by 15.6 percentage points; on VASR difficult distractors it reaches 70.5% best-of-three versus 54.9% for Supervised Concat; and on PGM it averages 57.3%, second to DRNet's 58.3%. The paper presents this as evidence of strong generalization from a single architecture rather than task-specific design.","pith_inferences":["The paper does not report whether the baseline models were trained with ground-truth rule supervision. If they were not, a fair test of PoNG's architectural contribution is to give the same rule-supervision loss to the baselines; the ablation where both rule heads are removed drops I-RAVEN accuracy from 95.9% to 79.7%, suggesting part of the reported gap could be supervision rather than convolutio","Because the ablation without both rule heads falls below RelBase and SCL, the comparison that isolates the architecture alone would be PoNG without rule heads against those baselines; on I-RAVEN that comparison would not favor PoNG.","The row- and row-pair grouping is a strong inductive bias for grid puzzles. A testable extension is to run PoNG on a non-grid set-reasoning task, such as text or graph analogies, where grouping into rows is meaningless; if performance degrades sharply, the claimed generality of the pathways block would need qualification."],"forward_implications":["On Raven-style benchmarks, PoNG's strongest gains appear exactly where the test distribution swaps in rules unseen at training, so the row- and row-pair grouping is doing generalization work rather than memorization.","On real-world VASR, the frozen perception backbone is the same across models, so the 70.5% versus 54.9% difficult-distractor gap isolates the reasoner's contribution on top of shared perception.","Because the rule-head losses are removable and removing both drops I-RAVEN accuracy from 95.9% to 79.7%, below RelBase and SCL, deploying PoNG on new benchmarks should preserve the auxiliary rule labels or expect the generalization gain to shrink.","The parameter count of 3.1M is far below several baselines (for example DRNet at 24.7M and SRAN at 45.7M), so the reported gains are not bought by scale.","The paper states that the pathways block is a generic module over sets of vector representations, implying the same reasoner could be lifted from image puzzles to other set-structured inputs without architectural change."],"supporting_citations":[{"why":"Supplies the PGM dataset, its eight generalization regimes, and the WReN baseline that anchors the field's comparisons.","marker":"[Barrett et al., 2018]"},{"why":"Provides the I-RAVEN benchmark used for the i.i.d. RPM evaluation and the SRAN baseline.","marker":"[Hu et al., 2021]"},{"why":"Defines A-I-RAVEN and I-RAVEN-Mesh, the out-of-distribution RPM benchmarks where PoNG shows its largest gains.","marker":"[Małkiński and Mańdziuk, 2025a]"},{"why":"Provides the VAP visual analogy benchmark with its five generalization regimes.","marker":"[Hill et al., 2019]"},{"why":"Introduces the real-world VASR benchmark and the zero-shot and supervised baselines PoNG is compared against.","marker":"[Bitton et al., 2023]"},{"why":"Supplies the TCN normalization operator that the ablations show is needed for held-out attribute generalization.","marker":"[Webb et al., 2020]"},{"why":"Provides the RelBase panel encoder that PoNG extends, and the RelBase baseline result.","marker":"[Spratley et al., 2020]"},{"why":"Supplies the frozen Vision Transformer backbone PoNG uses as its perception module on VASR.","marker":"[Dosovitskiy et al., 2021]"},{"why":"Provides the PredRNet baseline and the VAP result table PoNG is compared against.","marker":"[Yang et al., 2023a]"}],"fun_headline_variants":["PoNG beats baselines on abstract visual reasoning tasks","PoNG generalizes across synthetic and real-world analogies","PoNG leads by up to 15.6 points on visual reasoning benchmarks","New architecture tops baselines on 4 visual reasoning benchmarks","70.5% on hard distractors: PoNG generalizes in AVR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported advantage over baselines assumes that the only meaningful difference is the architecture, not the extra ground-truth rule labels PoNG uses in its auxiliary loss.","fun_headline_variants_meta":{"raw":{"variants":["PoNG beats baselines on abstract visual reasoning tasks","PoNG generalizes across synthetic and real-world analogies","PoNG leads by up to 15.6 points on visual reasoning benchmarks","New architecture tops baselines on 4 visual reasoning benchmarks","70.5% on hard distractors: PoNG generalizes in AVR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000554,"raw_usage":{"total_tokens":2651,"prompt_tokens":967,"completion_tokens":1684,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":1595}},"tokens_in":583,"tokens_out":1684,"duration_ms":13333,"temperature":1.0,"reasoning_tokens":1595,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:14:18.955951+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train RelBase or SCL under the identical loss, including the two rule heads with $\\beta=25$ and $\\gamma=5$, on A-I-RAVEN; if A/Color accuracy rises from about 67% to about 80%, then the auxiliary rule supervision, not the group-convolution pathways, explains the gain. If the gap persists, the pathways carry the generalization.","supporting_citations":[{"cited_title":"Learning to make analogies by contrasting abstract relational structure","cited_arxiv_id":null,"evidence_quote":"Provides the VAP visual analogy benchmark with its five generalization regimes."},{"cited_title":"Measuring abstract reasoning in neural networks","cited_arxiv_id":null,"evidence_quote":"Supplies the PGM dataset, its eight generalization regimes, and the WReN baseline that anchors the field's comparisons."},{"cited_title":"VASR : Visual analogies of situation recognition","cited_arxiv_id":null,"evidence_quote":"Introduces the real-world VASR benchmark and the zero-shot and supervised baselines PoNG is compared against."},{"cited_title":"Learning representations that support extrapolation","cited_arxiv_id":null,"evidence_quote":"Supplies the TCN normalization operator that the ablations show is needed for held-out attribute generalization."},{"cited_title":"A closer look at generalisation in RAVEN","cited_arxiv_id":null,"evidence_quote":"Provides the RelBase panel encoder that PoNG extends, and the RelBase baseline result."},{"cited_title":"An image is worth 16x16 words: Transformers for image recognition at scale","cited_arxiv_id":null,"evidence_quote":"Supplies the frozen Vision Transformer backbone PoNG uses as its perception module on VASR."}],"review_version":1}