Pith. sign in

REVIEW 4 major objections 4 minor 40 references

GIFARC: Synthetic Dataset for Leveraging Human-Intuitive Analogies to Elevate AI Reasoning

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GIFARC turns GIFs into 10,000 ARC-style puzzles with explicit analogies, and the paper reports that supplying such analogy labels in context shifts LLM reasoning on unseen ARC-AGI-2 tasks toward human-style everyday descriptions.

desk verdict A genuinely useful synthetic ARC dataset with a careful pipeline, but the human-intuitive analogy claim rests on an untested premise and an overclaim in the conclusion. read the letter →

arxiv 2505.20672 v1 pith:D5LNWWH2 submitted 2025-05-27 cs.AI

Woochang Sim , Hyunseok Ryu , Kyungmin Choi , Sungwon Han , Sundong Kim This is my paper
classification cs.AI
keywords AbstractionandReasoningCorpusanalogy-guidedsyntheticdatasetvision-languagemodelslargelanguageGIF-derivedtasksARC-stylegridtransformationin-contextlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GIFARC claims that one measurable reason AI systems lag behind humans on ARC-style visual reasoning is that they search for grid transformations without first finding an everyday analogy. To supply that missing prior, the paper builds a 10,000-task dataset by turning GIFs into ARC-style puzzles: each task comes with input-output grids, a short natural-language analogy such as “blocked water flow”, and an executable Python solution. The three-stage pipeline uses a vision-language model to read a GIF's scene and dynamics, a language model to distill that into a task sketch, and retrieval-based in-context generation to compile the sketch into deterministic grid-transformation code. The reported experiments show that when an LLM is given GIFARC examples in context before solving an unseen ARC-AGI-2 puzzle, it identifies the puzzle's analogy in everyday terms rather than raw grid descriptions. The paper presents GIFARC as a reusable resource for teaching models to “think analogically” before brute-force search, which if true would make ARC-style solutions more concise and human-understandable.

What carries the argument

The load-bearing machinery is the GIFARC task triple T = (E, α, ϕ): a set E of input-output grid pairs, a short natural-language analogy α (for example “blocked water flow”), and a Python program ϕ that implements the deterministic transformation F such that y = F(x). This triple is produced in three stages: a VLM (GPT o1) reads a GIF and returns a structured JSON record of scenario, objects, static and dynamic patterns, core principles, and interactions; a text LLM (GPT o3-mini) compresses that record into a task sketch of concepts and description; and a final LLM compiles the sketch into executable code using retrieval-augmented in-context learning from human-written ARC example tasks (the BARC seed set). The analogy label is copied from the sketch into the final task, so the everyday concept is wired directly to the code that makes the grids. In the application experiments, 15 such full triples are used as in-context examples to guide GPT 4.1-mini on unseen ARC-AGI-2 tasks.

What would settle it

Score the human analogy descriptions collected in Appendix D against GIFARC's ground-truth analogy labels using the same LLM evaluator and embedding cosine the paper applies to the models; if human scores do not meet or exceed the full-description model's scores, the dataset's claim to contain human-intuitive analogies is not supported. A separate fine-tuning run on the original ARC benchmark would settle the paper's closing claim that GIFARC training boosts solver accuracy rather than only changing in-context phrasing.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that analogies embedded in GIFs can be mined, compiled into executable ARC-style tasks, and then transferred to an LLM's problem-solving approach. The authors report that GPT 4.1-mini, given fifteen full GIFARC examples in context, described the underlying analogy of an ARC-AGI-2 task with phrases like “tidying up” and “organizing”, whereas a version given analogy-flattened examples stuck to grid-level terms such as “symmetrical horizontally” and “left cluster moving right”. Quantitatively, the full-description model's generated analogy scored 0.137 by an LLM evaluator and 0.829 by embedding cosine against the task's ground-truth analogy, versus 0.050 and 0.814 for the analogy-removed model. The authors interpret this as showing that GIFARC's analogy labels, not the grid examples alone, are what shift the model toward the human analogic approach.

Load-bearing premise

Everything depends on the unmeasured premise that the analogies a vision-language model extracts from GIFs are the same analogies a human would naturally see, and that such an analogy can be faithfully turned into a deterministic grid transformation; the paper collects human analogy descriptions for 12 tasks but never scores them against its own labels.

Editorial extensions

If this is right

  • If GIFARC works as claimed, analogy labels become a transferable prior: showing an LLM a handful of analogy-labeled grid tasks before an unseen ARC puzzle pushes it to describe the puzzle in everyday conceptual terms rather than grid-level mechanics.
  • The dataset itself is reusable: 10,000 tasks with ground-truth analogy α and executable solution ϕ can serve as training or prompting data for any ARC-style solver, not just the specific LLM tested.
  • Because each analogy is compiled into deterministic code, GIFARC could support checks of whether a model's stated analogy actually matches the transformation it computes, making reasoning steps more auditable.
  • The graded ablation (full description > without analogy > without analogy and without solution) implies that the analogy phrase is not decoration: removing it measurably reduces how well a model names the analogy of an unseen task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the current experiments measure analogy-naming in context, not benchmark solve rates; the paper's conclusion credits fine-tuning and accuracy gains on the original ARC benchmark, but those runs are not reported, so a fair reader should treat them as open hypotheses.
  • Editorial inference: since the pipeline compiles analogies into Python, one testable extension is to use GIFARC to explicitly train a verifier that checks whether a candidate solution's stated analogy is consistent with the executed transformation, which would make analogical reasoning auditable beyond phrasing.
  • Editorial inference: the GIF-based analogy inventory is limited by what a single short loop can show; adapting the pipeline to longer video clips would likely require disentangling multiple entangled analogies, which the paper itself flags in Section 6.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces GIFARC, a synthetic dataset of 10,000 ARC-style tasks generated from GIF images via a three-stage VLM/LLM pipeline: visual abstraction extraction (GPT o1), task sketching (GPT o3-mini), and executable ARC task generation (GPT o3-mini / GPT 4.1), with each task paired with a natural-language analogy label and Python solution code. The authors claim that embedding human-intuitive analogies into ARC-style tasks guides LLMs to evaluate tasks analogically before brute-force search, and they report two application experiments (Sections 5.1 and 5.2) comparing GPT-4.1-mini with full-description, analogy-removed, and solution-removed in-context contexts on ARC-AGI-2 tasks, as well as similarity scores between generated analogies and ground truth. The manuscript also includes dataset statistics, generation fidelity pass rates, code complexity metrics, and examples.

Significance. If the central claim were established, GIFARC would be a useful resource: it is released on Hugging Face, includes executable code and filtering checks, and attempts to scale synthetic ARC-style task generation with analogy labels in a reproducible pipeline. The pipeline's stage-wise pass-rate checks and the public visualization site are concrete strengths. However, the paper's headline claim about human-intuitive analogies currently rests on LLM-generated ground truth and a hand-curated in-context subset; the collected human evaluations are not scored. The resource itself may still be valuable for training and prompting, but the alignment-with-human-analogies claim needs direct evidence.

major comments (4)
  1. [Section 5.2 and Appendix D] The human evaluator analogies collected for 12 tasks (Appendix D) are never scored against the GIFARC ground-truth analogies; Figures 5a and 5b report only the two model conditions against the same LLM-generated ground truth. Because the ground-truth analogies were produced by GPT o1/o3-mini in the same pipeline, the reported alignment scores cannot distinguish 'GIFARC guides models toward human intuition' from 'GIFARC models agree with the LLM distribution that generated the labels.' This is the load-bearing gap for the paper's central claim.
  2. [Section 6] The conclusion states that 'Empirical results on the original ARC benchmark confirm that both fine-tuning on GIFARC and leveraging analogy cues in the reasoning step boost solver accuracy,' but no accuracy experiment on ARC (original, AGI-1, or AGI-2) is reported anywhere in Section 5 or the appendices. This claim is unsupported by the manuscript's evidence and should be removed or substantiated.
  3. [Section 5.1 and Appendix C.3] The full-description condition was produced by an LLM and then manually refined by the researchers (Appendix C.3), so the experiment does not evaluate the released GIFARC dataset as-is; it evaluates a hand-curated subset. In addition, the evaluation uses 15 in-context examples and 12 tasks with no error bars, confidence intervals, or significance tests, so the reported differences (0.137 vs. 0.050 LLM similarity; 0.829 vs. 0.814 embedding cosine) are not established as reliable.
  4. [Section 3.2 and Section 5.2] The premise that the analogies extracted by GPT o1 from GIFs are 'human-intuitive' is not validated independently: Step 1 (Section 3.2) instructs the VLM to extract 'core reasoning principles,' and the same LLM family later judges similarity to those labels. A concrete test would be to score the collected human analogies from Appendix D against the ground-truth labels and to report agreement statistics; without this, the dataset's main conceptual contribution remains a claim about LLM self-consistency rather than human-aligned reasoning.
minor comments (4)
  1. [Appendix D] The human-evaluator guideline instructs participants to describe analogies for '13 tasks,' while Section 5.2 and the surrounding text state that 12 tasks were used; the discrepancy should be corrected.
  2. [Appendix B.2] The user prompt contains the typo 'Insturction' instead of 'Instruction'; the sentence 'input_bitmap_generation_code must be executable Python code, when it generate a bitmap...' is also ungrammatical and should be rewritten.
  3. [Section 4, Table 3] The success rates are reported as percentages but the pass criteria are described only in prose; the table would benefit from a definition column clarifying what each stage's pass/fail judgment is based on.
  4. [Abstract and Section 1] The phrase 'build a more concise and human-understandable solution' should be 'builds a more concise and human-understandable solution' for grammatical consistency.

Circularity Check

1 steps flagged · score 6.0 of 10

The reported 'human-level analogic approach' evidence is an LLM-to-LLM comparison: Section 5.2 scores model outputs against GIFARC's own LLM-generated ground-truth analogies, while the collected human analogies (Appendix D) are never scored.

  1. other [Section 5.2, Figure 5 (with Section 3.2-3.4 for the provenance of the ground-truth analogy labels)]
    "The analogy α is copied from the task sketch created in Step 2, forming a ARC-style task T = (E, α, ϕ). ... The outputs of two models and three humans were then compared with the ground-truth analogies of the task generated with GIFARC pipeline. ... It resulted that the output of GPT 4.1-mini with full description showed 0.137 similarity, while the output of GPT 4.1 mini with analogy-removed description only measured 0.050."

    The reference 'ground truth' in the alignment experiment is not human judgment but an artifact of the GIFARC pipeline: α originates from GPT o1's visual abstraction (Step 1), is condensed by GPT o3-mini into a task sketch (Step 2), and is then copied into the task triple. The similarity judge is also GPT o3-mini. The full-description condition is built from exactly these (E, α, ϕ) triples, so the measured 0.137 vs 0.050 difference reflects agreement with the pipeline's own label distribution, not with human intuition. The three human analogies collected for the same 12 tasks (Appendix D) are never scored in the reported figures.

full rationale

GIFARC's data-generation pipeline is not itself circular: the GIF sources are external, the generated tasks are filtered by compilation and well-formedness checks, and the dataset is publicly released with executable solutions. The circularity is concentrated in the empirical support for the paper's headline claim that GIFARC aligns LLMs with human analogic reasoning. In Section 5.2, the 'ground-truth analogies' used as the evaluation reference are produced by the same GPT o1 / GPT o3-mini pipeline that created the GIFARC tasks, and the similarity judge is another GPT o3-mini. The three human analogies collected for the same 12 tasks (Appendix D) are not scored in the reported figures, so the reported differences (0.137 vs 0.050 for LLM similarity; 0.829 vs 0.814 for cosine similarity) measure agreement with the pipeline's own labels, not human similarity. This makes the human-alignment conclusion reduce by construction to an LLM-to-LLM consistency result. Separately, Section 6 asserts that 'Empirical results on the original ARC benchmark confirm that both fine-tuning on GIFARC and leveraging analogy cues in the reasoning step boost solver accuracy,' but no accuracy experiment is reported in the paper; Appendix C.3 also discloses manual refinement of the full-description context, further weakening the controlled comparison. These are omitted-support and validity concerns rather than additional circularity. No load-bearing self-citation chain or imported uniqueness theorem is present, so the score is 6 rather than higher.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the assumption that LLM-extracted GIF analogies are human-intuitive and that in-context usage of 15 curated samples represents the dataset effect. No fitted constants appear; the paper's evidence is based on hand-picked small samples and LLM-as-judge similarity. See red flags for selection bias and circular evaluation.

free parameters (3)
  • full_description sample count = 15
    Number of GIFARC samples selected by repeated multi-turn refinement prompts for the in-context context used in Section 5.1. Hand-selected, not a random sample.
  • evaluation task count = 12
    Number of ARC-style tasks used in Section 5.2 to measure analogy alignment; a small convenience sample.
  • human evaluator count = 3
    Three ARC-familiar humans produced analogy descriptions, but their scores are not reported in the paper.
assumptions (3)
  • domain assumption Analogies extracted by GPT o1 from GIFs are human-intuitive and semantically aligned with the GIF.
    The entire dataset premise relies on LLM extraction faithfully capturing the analogies a human would perceive. Only 3 humans were involved for 12 tasks, and their outputs are never scored in the paper (Appendix D, Section 5.2).
  • domain assumption ARC-style grids can faithfully encode the visual transformations from GIFs.
    The paper itself notes Version 1 failed to express objects from GIFs, requiring Version 2 with object bitmap seeds (Appendix B). The encoding fidelity is therefore not guaranteed.
  • domain assumption In-context learning with 15 curated GIFARC samples is representative of training on the full GIFARC dataset.
    Section 5.1 uses 15 hand-picked samples, yet the conclusion extrapolates to fine-tuning on GIFARC and accuracy gains on ARC benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GIFARC: Synthetic Dataset for Leveraging Human-Intuitive Analogies to Elevate AI Reasoning." pith.science (2026). https://pith.science/paper/D5LNWWH2

@misc{pith2026250520672,
  author       = {Pith},
  title        = {Pith review of: GIFARC: Synthetic Dataset for Leveraging Human-Intuitive Analogies to Elevate AI Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D5LNWWH2}},
  note         = {Machine review of arXiv:2505.20672}
}
read the original abstract

The Abstraction and Reasoning Corpus (ARC) poses a stringent test of general AI capabilities, requiring solvers to infer abstract patterns from only a handful of examples. Despite substantial progress in deep learning, state-of-the-art models still achieve accuracy rates of merely 40-55% on 2024 ARC Competition, indicative of a significant gap between their performance and human-level reasoning. In this work, we seek to bridge that gap by introducing an analogy-inspired ARC dataset, GIFARC. Leveraging large language models (LLMs) and vision-language models (VLMs), we synthesize new ARC-style tasks from a variety of GIF images that include analogies. Each new task is paired with ground-truth analogy, providing an explicit mapping between visual transformations and everyday concepts. By embedding robust human-intuitive analogies into ARC-style tasks, GIFARC guides AI agents to evaluate the task analogically before engaging in brute-force pattern search, thus efficiently reducing problem complexity and build a more concise and human-understandable solution. We empirically validate that guiding LLM with analogic approach with GIFARC affects task-solving approaches of LLMs to align with analogic approach of human.

Figures

Figures reproduced from arXiv: 2505.20672 by the authors.

Figure 1
Figure 1. Illustration of how differently an agent solves ARC-style task when it is guided with or [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of GIFARC data synthesis pipeline that transforms a single GIF into a cor [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Histogram of task types occurrence in GIFARC. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (24 more)
Figure 4
Figure 4. Figure 4: Experiment 1 case study: the LLMs learned in-context with richer analogy context result [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Similarity analysis between task-implied analogy found by GIFARC-trained LLMs and [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Landing Page of GIFARC Visualization website. It is publicly released at [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]
Figure 7
Figure 7. Figure 7: Snapshot of GIF used for generating task 962. [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]
Figure 8
Figure 8. Figure 8: GIFARC-generated task 962. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]
Figure 9
Figure 9. Figure 9: GIFARC-generated task 7442. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_9.png]
Figure 10
Figure 10. Figure 10: GIFARC-generated task 1037. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_10.png]
Figure 11
Figure 11. Figure 11: GIFARC-generated task 2767. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_11.png]
Figure 12
Figure 12. Figure 12: GIFARC-generated task 556. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_12.png]
Figure 13
Figure 13. Figure 13: GIFARC-generated task 3690. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_13.png]
Figure 14
Figure 14. Figure 14: GIFARC-generated task 385. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_14.png]
Figure 15
Figure 15. Figure 15: GIFARC-generated task 117. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_15.png]
Figure 16
Figure 16. Figure 16: GIFARC-generated task 410. 33 [PITH_FULL_IMAGE:figures/full_fig_p033_16.png]
Figure 17
Figure 17. Figure 17: GIFARC-generated task 202. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_17.png]
Figure 18
Figure 18. Figure 18: GIFARC-generated task 1413. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_18.png]
Figure 19
Figure 19. Figure 19: GIFARC-generated task 2500. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_19.png]
Figure 20
Figure 20. Figure 20: GIFARC-generated task 651. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_20.png]
Figure 21
Figure 21. Figure 21: GIFARC-generated task 7128. 38 [PITH_FULL_IMAGE:figures/full_fig_p038_21.png]
Figure 22
Figure 22. Figure 22: GIFARC-generated task 214. 39 [PITH_FULL_IMAGE:figures/full_fig_p039_22.png]
Figure 23
Figure 23. Figure 23: GIFARC-generated task 2222. 40 [PITH_FULL_IMAGE:figures/full_fig_p040_23.png]
Figure 24
Figure 24. Figure 24: GIFARC-generated task 7262. 41 [PITH_FULL_IMAGE:figures/full_fig_p041_24.png]
Figure 25
Figure 25. Figure 25: GIFARC-generated task 413. 42 [PITH_FULL_IMAGE:figures/full_fig_p042_25.png]
Figure 26
Figure 26. Figure 26: GIFARC-generated task 155. 43 [PITH_FULL_IMAGE:figures/full_fig_p043_26.png]
Figure 27
Figure 27. Figure 27: GIFARC-generated task 2061. 44 [PITH_FULL_IMAGE:figures/full_fig_p044_27.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 34 canonical work pages

  1. [1]

    On the Measure of Intelligence

    François Chollet. On the Measure of Intelligence. arXiv:1911.01547, 2019

    arXiv 1911

  2. [2]

    ARC Prize, 2024

    Lab42. ARC Prize, 2024

    work page 2024

  3. [3]

    Faith and fate: Limits of transformers on compositionality

    Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, et al. Faith and fate: Limits of transformers on compositionality. In NeurIPS, 2023

    work page 2023

  4. [4]

    Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus

    Seungpil Lee, Woochang Sim, Donghyeon Shin, Sanha Hwang, Wongyu Seo, Jiwon Park, Seokki Lee, Sejin Kim, and Sundong Kim. Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus. arXiv:2403.11793, 2024

    arXiv 2024

  5. [5]

    Abstraction and analogy-making in artificial intelligence

    Melanie Mitchell. Abstraction and analogy-making in artificial intelligence. Annals of the New York Academy of Sciences, 1505(1):79–101, 2021

    work page 2021

  6. [6]

    Neural networks for abstraction and reasoning

    Mikel Bober-Irizar and Soumya Banerjee. Neural networks for abstraction and reasoning. Scientific Reports, 14(1):27823, 2024

    work page 2024

  7. [7]

    Understanding and Benchmarking Artificial Intelligence: Ope- nAI’s o3 Is Not AGI

    Rolf Pfister and Hansueli Jud. Understanding and Benchmarking Artificial Intelligence: Ope- nAI’s o3 Is Not AGI. arXiv preprint arXiv:2501.07458, 2025

    arXiv 2025

  8. [8]

    Abstraction and Reasoning Corpus for Artificial General Intelligence v2

    Francois Chollet, Mike Knoop, and Greg Kamradt. Abstraction and Reasoning Corpus for Artificial General Intelligence v2. https://github.com/arcprize/ARC-AGI-2 , 2025

    work page 2025

Show all 40 references
  1. [9]

    Object- Centric Compositional Imagination for Visual Abstract Reasoning

    Rim Assouel, Pau Rodriguez, Perouz Taslakian, David Vazquez, and Yoshua Bengio. Object- Centric Compositional Imagination for Visual Abstract Reasoning. In ICLR Workshop on the Elements of Reasoning: Objects, Structure, and Causality, 2022

    2022

  2. [10]

    H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmark

    Solim LeGris, Wai Keen V ong, Brenden M Lake, and Todd M Gureckis. H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmark. arXiv preprint arXiv:2409.01374, 2024

    2024 arXiv

  3. [11]

    Do Large Language Models Solve ARC Visual Analogies Like People Do? In CogSci, 2024

    Gustaw Opiełka, Hannes Rosenbusch, Veerle Vijverberg, and Claire E Stevenson. Do Large Language Models Solve ARC Visual Analogies Like People Do? In CogSci, 2024

    2024

  4. [12]

    PQA: Perceptual Question Answering

    Yonggang Qi, Kai Zhang, Aneeshan Sain, and Yi-Zhe Song. PQA: Perceptual Question Answering. In CVPR, 2021

    2021

  5. [13]

    Addressing the Abstraction and Reasoning Corpus via Procedural Example Generation

    Michael Hodel. Addressing the Abstraction and Reasoning Corpus via Procedural Example Generation. arXiv:2404.07353, 2024

    2024 arXiv

  6. [14]

    Tenenbaum

    Samuel Acquaviva, Yewen Pu, Marta Kryven, Theodoros Sechopoulos, Catherine Wong, Gabrielle Ecanow, Maxwell Nye, Michael Tessler, and Joshua B. Tenenbaum. Communicating Natural Programs to Humans and Machines. In NeurIPS, 2022

    2022

  7. [15]

    Playgrounds for Abstrac- tion and Reasoning

    Subin Kim, Prin Phunyaphibarn, Donghyun Ahn, and Sundong Kim. Playgrounds for Abstrac- tion and Reasoning. In NeurIPS Workshop on Neuro Causal and Symbolic AI, 2022

    2022

  8. [16]

    The ConceptARC Bench- mark: Evaluating Understanding and Generalization in the ARC Domain

    Arseny Moskvichev, Victor Vikram Odouard, and Melanie Mitchell. The ConceptARC Bench- mark: Evaluating Understanding and Generalization in the ARC Domain. Transactions on Machine Learning Research, 2023

    2023

  9. [17]

    LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-Based Representations

    Yudong Xu, Wenhao Li, Pashootan Vaezipoor, Scott Sanner, and Elias B Khalil. LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-Based Representations. Transactions on Machine Learning Research, 2024

    2024

  10. [18]

    Combining induction and transduction for abstract reasoning

    Wen-Ding Li, Keya Hu, Carter Larsen, Yuqing Wu, Simon Alford, Caleb Woo, Spencer M Dunn, Hao Tang, Michelangelo Naim, Dat Nguyen, et al. Combining induction and transduction for abstract reasoning. arXiv preprint arXiv:2411.02272, 2024

    2024 arXiv

  11. [19]

    From Generation to Selection Findings of Converting Analogical Problem-Solving into Multiple-Choice Questions

    Donghyeon Shin, Seungpil Lee, Klea Lena Kovacec, and Sundong Kim. From Generation to Selection Findings of Converting Analogical Problem-Solving into Multiple-Choice Questions. In EMNLP, 2024

    2024

  12. [20]

    name": "object name

    Max Peeperkorn, Tom Kouwenhoven, Dan Brown, and Anna Jordanous. Is Temperature the Creativity Parameter of Large Language Models?, 2024. 10 Appendix A Examples of Prompts A.1 Full Prompt Example for Extracting Visual Abstractions The extraction of visual abstractions constitut...

    2024

  13. [21]

    Inspect the example puzzle implementations, making note of the functions used and the physical/geometric/topological/logical details

  14. [22]

    Inspect the new puzzle’s description

  15. [23]

    Brainstorm a possible implementation for the new puzzle

  16. [24]

    library”: “<Write only the libraries used in the code. Ex. from common import* \n import numpy as np \n ....>

    Generate a code block formatted like the earlier examples with a comment starting # concepts: listing the concepts and# description: describing the inputs and transforma- tion from the given description. When implementing code, please avoid using float type variables, numbers ...

  17. [25]

    At least 2 pairs of example problems (input/output)

  18. [26]

    Use of predictable transformation laws or analogies

  19. [27]

    Clear definition of each object’s role, interaction, and rule

  20. [28]

    Emphasis on fun/creativity and adjustable difficulty

  21. [29]

    Complex and creative Puzzle

  22. [30]

    input_bitmap_generation_code must be executable Python code, when it generate a bitmap, bitmap should be contained in the parameter which has formatted name, (short descrition, in 10 char)_input_bitmap_(idx) is the format you should keep, However the name of bitmap should star...

  23. [31]

    input_bitmap_generation_code

    solution_code must be executable Python # Creating an ARC-Style Puzzle Problem → At least 2 pairs of example input–output → There is a pattern/rule → The human/AI infers the rule and solves the problem # Detailed Objectives You must use the provided functions to create objects...

  24. [32]

    Either the permuted grid or the input grid is not well-formed

  25. [33]

    Verification Changes in Reasoning Steps

    The results differ even when only the colors are changed. Identity The input and output are completely identical. Non-Well Formed Output The transformation result is not well-formed (i.e., not a 2D list with equal row lengths and integer values between 0-9). Black Output The o...

  26. [34]

    What the provided input-output pairs are visualizing

  27. [35]

    What kind of analogy the pattern changing from input to output can be related to or expressed as

  28. [36]

    What the colors or object shapes shown in the input/output symbolize or represent

  29. [37]

    Ex-1) - It appears to be spreading from the center

    Please write the expected rules if you don’t know the task’s analogy. Ex-1) - It appears to be spreading from the center. Ex-2) - The movement seems to change randomly and the colors are changing. [Working Example] {Example task} Analogy: In this task the grid represents a sce...

  30. [38]

    empty screen

    Ensure the anchoring of the fractals at the bottom edge remains the same. 2) Preserve the radial symmetry of the central circular region and its common center point. 3) For each pulsation step in the input, magnify or contract the rings, lines, and fractals accordingly. 4) Con...

  31. [39]

    2) Preserve the radial symmetry of the central circular region and its common center point

    Ensure the anchoring of the fractals at the bottom edge remains the same. 2) Preserve the radial symmetry of the central circular region and its common center point. 3) For each pulsation step in the input, magnify or contract the rings, lines, and fractals accordingly. 4) Con...

  32. [40]

    2) Preserve the radial symmetry of the central circular region and its common center point

    Ensure the anchoring of the fractals at the bottom edge remains the same. 2) Preserve the radial symmetry of the central circular region and its common center point. 3) For each pulsation step in the input, magnify or contract the rings, lines, and fractals accordingly. 4) Con...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.