{"id":"b4445853-f0b8-460c-9fcb-a97a9aa71be8","arxiv_id":"2412.07322","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ConceptSearch uses LLM-generated programs with concept-based scoring to solve 29/50 ARC training tasks and speed up search by up to 30% versus pixel-distance scoring.","lead":"This paper introduces ConceptSearch, a search algorithm that uses large language models to write programs for ARC puzzles, guided by scoring functions that try to capture the transformation concept rather than pixel similarity. It reports solving 29 out of 50 selected ARC tasks, up from 13 with direct prompting, and up to 30% fewer search iterations with concept-based scores.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed improvement over direct GPT-4 prompting is confounded: ConceptSearch uses Gemini 1.5 Pro for generation while the 13/50 baseline is GPT-4, so the gain may reflect LLM capability rather than the search algorithm.","rationale":"The reader's weakest assumption correctly identifies the most load-bearing flaw: the headline comparison to direct prompting is confounded by the LLM choice. The paper's internal scoring-function comparison is controlled and supports the value of concept-based guidance, so the work should not be rejected outright. However, the abstract's 'significant performance improvement over direct prompting with GPT-4' is not supported without a same-LLM baseline. The efficiency claim is also weaker than stated because it is computed on small pairwise intersections and lacks error bars, but this is secondary. A simple controlled experiment (swapping in GPT-4 as the generator) would settle the attribution and is a reasonable condition for accepting the paper's quantitative claims.","tokens_in":9868,"tokens_out":9079,"duration_ms":86617,"concrete_test":"Run the identical ConceptSearch pipeline (same islands, iterations, ARC-DSL, database initialization, prompts, and all three scoring functions) with GPT-4 substituted as the program generator on the same 50 tasks. If the solved count drops to approximately 13/50 (or well below 29/50), the reported improvement over direct prompting is due to Gemini 1.5 Pro, not to the search algorithm. This single controlled experiment directly settles the attribution of the headline gain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline result—function-search solving 25/50 and 29/50 versus 13/50 for direct prompting—confounds two variables: the search algorithm and the underlying LLM. ConceptSearch generates programs with Gemini 1.5 Pro (and uses Gemini 1.5 Flash for hypothesis descriptions), whereas the 13/50 baseline is taken from Xu et al. (2024), which used GPT-4. No same-LLM direct-prompting baseline is reported, and no GPT-4-based ConceptSearch run is reported. Consequently, the 12-task improvement over direct prompting cannot be attributed to the search algorithm or the concept-based scoring; a stronger program synthesizer could account for all of it. The internal comparison among scoring functions (Hamming vs CNN vs LLM) is controlled and remains informative, but the abstract's 'significant performance improvement over direct prompting with GPT-4' is not established. The efficiency numbers in Table 2 are also computed on small pairwise intersections without variance estimates, so the '~30% efficiency' claim is secondary and less reliable than the paper implies.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ConceptSearch, a FunSearch-style evolutionary program search algorithm for the Abstraction and Reasoning Corpus (ARC). Starting from a database of ARC-DSL programs initialized with ARC training-task solvers, the algorithm uses Gemini 1.5 Pro to generate new programs from pairs of in-context programs selected by a scoring function. Three scoring functions are compared: normalized Hamming distance, a CNN-based embedding trained on ConceptARC, and an LLM-based natural-language scoring function that embeds hypotheses with a fine-tuned SentenceTransformer. Experiments on a 50-task subset of the ARC training set report 25/50 tasks solved with Hamming scoring, 25/50 with CNN scoring, and 29/50 with LLM scoring, compared with 13/50 for direct GPT-4 prompting taken from Xu et al. (2024). The paper further claims that concept-based scoring improves search efficiency by up to 30% in program iterations on tasks solved by both compared methods. The main technical contributions are the search algorithm and the comparison of scoring functions.","tokens_in":10064,"tokens_out":16503,"duration_ms":174155,"significance":"If the results are robust, the paper provides a useful demonstration that LLM-driven program search with concept-based guidance can solve a nontrivial fraction of ARC-style tasks, and the released code supports replication. The controlled comparison among the three scoring functions, all using the same Gemini generator and verified against ground-truth output grids, is the strongest part of the paper. However, the headline comparison with direct GPT-4 prompting is confounded by the change of base LLM, the efficiency claims rest on small and ill-specified intersections, and the LLM-based scorer is not a pure feature-extraction comparison. The paper's contribution, as currently evidenced, is narrower and less causally established than the abstract and conclusion claim.","major_comments":[{"comment":"The headline comparison is confounded by the base model. Section 4 states that ConceptSearch uses Gemini 1.5 Pro for program generation and goal hypothesis generation, while the 13/50 baseline in Table 1 is taken from Xu et al. (2024), which used GPT-4. No direct-prompting baseline with Gemini 1.5 Pro and no ConceptSearch run with GPT-4 is reported, so the search algorithm and the underlying LLM vary together. The improvement from 13/50 to 25/50 or 29/50 could therefore be due entirely to a stronger program synthesizer, and the abstract's claim of \"a significant performance improvement over direct prompting with GPT-4\" is not supported by the evidence. Please add a same-LLM direct-prompting baseline (or a GPT-4 variant of ConceptSearch) and revise all claims to compare methods under the same generator; the database initialized with ARC training solvers is an additional prior that the direct-prompting baseline lacks, so this should also be acknowledged or controlled.","section":"§4, Table 1"},{"comment":"The efficiency and success differences are not quantified with confidence. Table 2 reports percentage improvements \"based on the number of program-iterations taken to find the solution for tasks solved by both methods,\" but it gives no intersection sizes, no per-task iteration distributions, and no variance or significance test. With at most 29 solved tasks, a 29.3% improvement could be driven by one or two outliers. Similarly, Table 1's \"Mean iters\" is computed over an unreported intersection of tasks solved by all methods, and the 25/50 versus 29/50 accuracy difference is not tested for significance; for a paired binary outcome, four additional successes can be within chance (exact McNemar p is at best about 0.125). Please report pairwise intersection task counts, per-task iteration counts, and a paired permutation or McNemar test, and scale the claims to what those tests support.","section":"§4, Table 2 and §5"},{"comment":"The LLM-based scoring function is not a pure feature-extraction comparison with the CNN scorer. Section 3.3 generates the goal hypothesis by presenting the LLM with the top-10 programs selected using the CNN-based scoring function and their natural-language descriptions, while the CNN scorer receives only the grids. The LLM scorer therefore has access to additional information (selected programs and their descriptions, generated by a different model, Gemini 1.5 Flash) in addition to a different embedding model. The conclusion that \"LLMs have better feature extraction capabilities and overall effectiveness in handling ARC tasks\" is not supported by this design. Please ablate the LLM scorer (for example, using random in-context programs or omitting program descriptions) or rephrase the conclusion as a comparison of the full LLM-based scoring pipeline.","section":"§3.3, Figure 7"},{"comment":"The exclusion of the 50 evaluation tasks from the initial program database is asserted but not documented. Section 4 says \"It is also made sure that these 50 tasks are not available in any way, including the program database,\" but since the evaluation tasks are drawn from the ARC training set and the database is initialized with ARC training-task solvers, this exclusion is load-bearing: if any evaluation task's solver or a near-equivalent program were present in the initial population, the search would solve it trivially and the comparison to direct prompting would lose its meaning. Please specify how the exclusion was verified (for example, task IDs and program-equivalence screening) and, ideally, include an ablation with an empty or randomly initialized database.","section":"§3.1 and §4"}],"minor_comments":[{"comment":"Equation (3) has an unbalanced right parenthesis: `SLLM(h0, hf ) = ∥F (h0) − F (hf ))∥` should be `SLLM(h0, hf) = ∥F(h0) − F(hf)∥`.","section":"§3.3"},{"comment":"The Hamming distance is defined only for same-size grids; for ARC tasks where the output grid has different dimensions than the input grid, please specify how the distance is computed (padding, resizing, or a penalty for size mismatch).","section":"§4"},{"comment":"Algorithm 1 generates a single program `f` per step in the pseudocode, but Section 4 says each API call generates five programs; please align the pseudocode with the implementation.","section":"§3.1 and §4"},{"comment":"The stated maximum of 100 LLM API calls per task does not include the Gemini 1.5 Flash calls used to describe database programs or the per-island goal-hypothesis calls; please report the total API call budget.","section":"§3.3 and §4"},{"comment":"The exact 50-task subset is not identified; please list task IDs (or a public hash) in an appendix to make the comparison with Xu et al. (2024) reproducible.","section":"§4"},{"comment":"The full prompts are only shown in \"compact\" versions in Figures 3, 6, and 7; include the exact prompts, model versions, and sampling temperatures in an appendix.","section":"§3.1 and §4"},{"comment":"The conclusion's phrase \"when evaluated on a set of 50 tasks using GPT-4\" is misleading because the proposed method uses Gemini 1.5 Pro; please reword to distinguish the baseline model from the method.","section":"§6"},{"comment":"The column header \"Accuracy\" should be \"Solved / total\" or \"Accuracy (%)\" to avoid the fraction/percent mismatch with the text.","section":"§4, Table 1"},{"comment":"Please state explicitly that the column scoring function has X% fewer iterations than the row function and give the number of tasks in each intersection.","section":"§4, Table 2"},{"comment":"The claim that there is no task solved by the CNN scorer that is not solved by the LLM scorer is made without supporting counts; please report the 2x2 task-level table.","section":"§4"},{"comment":"Some flowchart labels overlap in the preprint's rendering; please check the final camera-ready version.","section":"Figure 2"},{"comment":"The citation to Greenblatt (2024) is a blog post with no version or access date; if it remains a preprint or blog, provide a stable URL and access date.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline case. The controlled scoring-function comparison and the released code are valuable, and the paper can likely be made sound with additional experiments: a same-LLM direct-prompting baseline, statistical tests on the paired 50 tasks, explicit intersection sizes for the efficiency comparisons, and a documented exclusion check for the initial program database. I therefore recommend major revision rather than rejection. The main risk is that the final version does not fix the headline comparison and readers attribute the 13/50 to 29/50 improvement to the search algorithm alone when the base model and the database prior also changed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this paper adapts FunSearch to ARC and introduces two concept-based scoring functions: a CNN embedding trained on ConceptARC, and an LLM-generated natural-language hypothesis embedding scored with a SentenceTransformer. The internal comparison among Hamming, CNN, and LLM scoring is clean—same Gemini 1.5 Pro generator, same search budget, same 50 tasks—and it shows the LLM-based score solving 29/50 vs 25/50 for Hamming and CNN, with fewer iterations. That is a real, useful result and the main reason to read the paper.\n\nSecond, the abstract's claim of \"significant performance improvement over direct prompting with GPT-4\" is not supported as stated. The 13/50 baseline comes from Xu et al. using GPT-4; this paper's search uses Gemini 1.5 Pro. There is no same-LLM direct-prompting baseline, so the jump from 13/50 to 25/50 could largely reflect a stronger generator rather than the search algorithm. The efficiency improvements (24.7%, 29.3%, 10.3%) are computed on small pairwise intersections of solved tasks with no variance estimates, so treat those numbers as suggestive.\n\nThe paper is otherwise transparent and honest. It evaluates on 50 training tasks (not the official ARC eval), ships code, and discusses a failed problem-iteration variant. The CNN classifier's own ~40% ConceptARC accuracy is reported, which is a good sign. Training the scoring functions on ConceptARC is external to the 50 test tasks, so that is not leakage. The final solution programs are verified against ground-truth grids, which is the right standard.\n\nSoft spots beyond the confound: no repeated runs or error bars on the headline 25/50 vs 29/50 numbers—a four-task difference could be noise. The LLM-based scoring uses two different models (Gemini 1.5 Pro for goal hypotheses and program descriptions, Gemini 1.5 Flash for program hypothesis descriptions) without ablating their separate contributions. The efficiency metric counts iterations to first correct program, which is reasonable but should be reported per task rather than as aggregate percentages.\n\nOverall, this is a solid, honest systems paper for the ARC program-search subfield. The internal scoring-function comparison is the valuable part, and the method is worth building on. A referee should require a same-LLM direct-prompt baseline and more careful efficiency statistics, but the paper deserves review.","headline":"Solid FunSearch-for-ARC paper with a clean internal scoring-function comparison; the headline improvement over direct GPT-4 prompting is confounded by the switch to Gemini 1.5 Pro.","tokens_in":10608,"tokens_out":2030,"would_cite":true,"duration_ms":20974,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ConceptSearch, an LLM-driven program-search loop with concept-based scoring, solves 29 of 50 ARC tasks, versus 13 for direct prompting and 25 for pixel-distance scoring.","keywords":["Abstraction and Reasoning Corpus","program synthesis","LLM-guided search","concept-based scoring","natural language hypotheses","function search","few-shot generalization","domain-specific languages"],"falsifier":"A reader could check this by running the same LLM in direct-prompting mode on the same 50-task set; if direct prompting alone solves 25 or more tasks, the search loop is not the source of the improvement. A second check is to record iteration counts on all solved tasks rather than only on the subset solved by both compared methods.","tokens_in":9644,"feed_emoji":"🧩","tokens_out":7195,"duration_ms":137693,"temperature":0.7,"pith_summary":"This paper is trying to establish that an LLM can solve more Abstraction and Reasoning Corpus tasks when its program-generation ability is wrapped in an iterative search loop than when it is asked to produce an answer directly. The loop keeps a database of candidate transformation programs, scores each by how well it captures the concept behind the input-output examples, and feeds the top candidates back into the LLM as in-context guidance. On a 50-task evaluation subset, the proposed search with a natural-language concept scorer solves 29 tasks, compared with 25 using pixel-overlap scoring and 13 in the direct-prompting baseline. The significance is that concept-level feedback, not raw pixel similarity, is what makes LLM-driven program search efficient enough to be practical on ARC-like puzzles.","feed_headline":"LLM program search with concept scoring solves 29 of 50 ARC puzzles","feed_subtitle":"Wrapping LLM code generation in a scored search loop beats both direct prompting and pixel-distance scoring.","key_machinery":"The central object is the scoring function used to rank programs in the search database. The LLM-based version first generates a text description of what a candidate program does, then embeds that description with a fine-tuned text-embedding model and compares it to an embedded description of the target transformation. This replaces the trivial Hamming distance between predicted and true output grids, which the paper argues is misleading because visually similar grids can come from very different underlying programs. A CNN-based alternative embeds grid transformations directly, trained on a concept-labeled ARC dataset, but the LLM-based version solves more tasks and searches more efficiently.","core_discovery":"The central claim is that \"concept-based scoring\"—measuring the distance between a candidate program's transformation and the transformation implied by the task examples—is the key to making LLM-driven program search work on ARC. A candidate program is converted into a natural-language hypothesis of its transformation; a text-embedding model maps that hypothesis and the task's goal hypothesis to vectors; the Euclidean distance between the vectors is the search score. With this scoring, the search solves 29/50 tasks on the paper's evaluation subset, versus 25/50 with Hamming distance and 13/50 for direct prompting. The paper also claims that concept-based scorers reach solutions in roughly 25-30% fewer program-generation iterations than Hamming distance on tasks solved by both methods.","pith_inferences":["Editorial inference: the fair comparison for the 13-to-29 improvement would run the same LLM under direct prompting and under search; the paper's baseline uses a different LLM, so part of the gain could be model choice.","Editorial inference: the paper's claim that combining CNN and LLM scoring adds no new solved tasks is based only on the solved-task union; a combined scorer might still improve efficiency or robustness on tasks solved by neither.","Editorial inference: varying the number of generated goal hypotheses and measuring success per API call would test whether the gains come from hypothesis diversity rather than from the search algorithm itself."],"forward_implications":["Wrapping a capable code-writing LLM in an iterative search loop should improve ARC performance without task-specific program-synthesis heuristics.","Concept-based scoring can reduce LLM inference cost by reaching solutions in fewer iterations than pixel-level scoring on tasks where both eventually succeed.","Natural-language hypotheses can serve as transferable intermediate representations between ARC tasks that share a transformation concept.","The same loop should apply to other few-shot transformation problems where success is binary but intermediate concepts are expressible in language.","The gap between 25/50 and 29/50 suggests that richer feedback signals, not just more sampling, are what push program search past plateaus on ARC."],"supporting_citations":[{"why":"provides the iterative program-evolution loop that ConceptSearch adapts.","marker":"Romera-Paredes et al. 2023"},{"why":"supplies the domain-specific language of grid transformations used for candidate programs.","marker":"Hodel 2024"},{"why":"defines the 50-task evaluation subset and the direct-prompting baseline of 13/50.","marker":"Xu et al. 2024"},{"why":"provides the concept-labeled ARC tasks used to train the CNN and fine-tune the text embedding.","marker":"Moskvichev, Odouard, and Mitchell 2023"},{"why":"supplies natural-language transformation descriptions used as in-context examples for hypothesis generation.","marker":"Acquaviva et al. 2023"},{"why":"provides the text-embedding model that maps language hypotheses to score vectors.","marker":"Reimers and Gurevych 2019"},{"why":"demonstrates that sampling many LLM programs scales ARC performance, motivating search-based alternatives.","marker":"Greenblatt 2024"}],"fun_headline_variants":["Concept scoring beats pixel scoring in LLM program search for ARC","LLM search with concept scoring: 29/50 ARC tasks, 30% fewer iterations","Concept-guided LLM search solves 29/50 ARC puzzles with 30% fewer steps","Concept-based scoring outperforms Hamming in LLM search for ARC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gain over the direct-prompting baseline assumes that the two settings differ only in the search loop; the paper's baseline comes from a different LLM, so part of the improvement could be due to that model choice.","fun_headline_variants_meta":{"raw":{"variants":["Concept scoring beats pixel scoring in LLM program search for ARC","LLM search with concept scoring: 29/50 ARC tasks, 30% fewer iterations","Concept-guided LLM search solves 29/50 ARC puzzles with 30% fewer steps","Concept-based scoring outperforms Hamming in LLM search for ARC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001092,"raw_usage":{"total_tokens":4540,"prompt_tokens":906,"completion_tokens":3634,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":3549}},"tokens_in":522,"tokens_out":3634,"duration_ms":27444,"temperature":1.0,"reasoning_tokens":3549,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:54:57.434536+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could check this by running the same LLM in direct-prompting mode on the same 50-task set; if direct prompting alone solves 25 or more tasks, the search loop is not the source of the improvement. A second check is to record iteration counts on all solved tasks rather than only on the subset solved by both compared methods.","supporting_citations":[],"review_version":1}