{"id":"9ee7fa47-75a3-4bea-98c4-cb7141e62a50","arxiv_id":"2506.13820","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"With human-in-the-loop feedback and structured decomposition, a GPT-4-based system produces correct morphological programs for nearly all IPARC tasks, including newly generated instances.","lead":"An interactive system pairing a human engineer with GPT-4 succeeds on nearly all tasks in the IPARC program synthesis benchmark, a set of 600 synthetic image-transformation puzzles that previously resisted automated solutions. The paper draws lessons about human-LLM collaboration from these controlled tasks, though the headline result requires extra hints for one category.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Solve rates are attributed to the LLM, but human-supplied example solutions and structural directives ('4 Dilations followed by 3 Erosions'; 'erosion SEs from dilation SEs') in §4.4 and Supp. Table 5 may be doing the synthesis; Cat A Hard original is 0% without snapshots.","rationale":"The reader's weakest assumption is exactly the concern I find most load-bearing: the human corrective feedback in Section 4.4 and Supplementary Table 5 appears to encode the target program's structure. I agree with that diagnosis. The method is explicitly human-in-the-loop, so some human guidance is not itself an objection; the objection is that the reported numbers do not separate the LLM's contribution from human-supplied constraints. Table 1's Cat A Hard row is particularly telling: 0% solved without snapshots, and only 29/99 with manual pruning and randomisation, so the abstract's 'across all categories' is true only with additional supervision. The final code in Listing 1.1 is an exhaustive enumerator over a human-narrowed hypothesis space; that is a legitimate engineering result, but it does not demonstrate that the LLM generates the solutions. A clean ablation removing verbatim example solutions and structural hints would settle whether the LLM is doing the synthesis or merely implementing a human's algorithm. This concern does not move the verdict: the paper remains a plausible workflow report with disclosed limitations, but the central claim needs rewording and the artifacts need to be released. Hence UNCHANGED, preserving the reader's CONDITIONAL verdict.","tokens_in":12673,"tokens_out":8428,"duration_ms":103430,"concrete_test":"Run a controlled ablation on Category B Hard and Category A Hard (snapshots): keep the initial task description and background IPARC paper identical, but restrict the human to binary correctness feedback (e.g., 'incorrect, regenerate') and remove (a) the verbatim example solutions and (b) the structural directive 'try 4 Dilations followed by 3 Erosions' with 'erosion SEs selected from the dilation SEs'. Measure per-category solve rates and interaction counts. If rates collapse toward the 0% LLM-only baseline reported in §4.4, the central claim must be re-attributed to human-supplied structure rather than to iStrucInd+.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in §5—that iStrucInd+ 'can generate code that solves tasks across all categories'—is only supported under a reading in which the human contributes the target program's structure. Three observations make this the load-bearing concern. (1) Table 1 reports 0/100 solved for Cat A Hard (original); the 100% row for Cat A Hard uses intermediate snapshot images, supervision outside the standard IPARC task specification, and Table 3 shows only 29/100 with manual pruning and 99/100 with added randomisation. The abstract's unqualified 'across all IPARC categories' is therefore not true for the original task format. (2) Section 4.4 states that prompts include example solution-sequences (2 for simple variants, 4 for hard variants) and the IPARC paper as background. Supplementary Table 5 shows the human, after the LLM fails, pasting verbatim example solutions and instructing it to 'try 4 Dilations followed by 3 Erosions' with 'erosion SEs must be selected from the dilation SEs used'. This is a structural constraint on the program, not generic correctness feedback. (3) The final code in Listing 1.1 is an exhaustive enumerator over that human-narrowed hypothesis space; it tests all 4-dilation/3-erosion pipelines until one matches. Since the LLM-only baseline in §4.4 failed even on simple variants, the reported solve rates are consistent with the human doing the synthesis and the LLM implementing the search. The paper's contribution as a workflow report may survive, but the central claim about LLM-driven program synthesis is overstated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports on iStrucInd+, a human-in-the-loop structured inductive programming approach that uses LLMs to propose data-flow decompositions and then to generate morphological programs for tasks from the IPARC challenge. The authors claim that iStrucInd+ solves tasks across all IPARC categories, and they report insights about LLM-based structure identification, freezing correct code, code reuse, and human creativity. The evaluation covers all 600 tasks, with additional tests on 20 newly generated tasks per category. The paper's central claim is that this is the first report of IPARC tasks being tackled with any degree of success.","tokens_in":13013,"tokens_out":2985,"duration_ms":37244,"significance":"If the central claim were established, the paper would be a notable contribution to the program-synthesis and human-LLM collaboration literature, and its qualitative insights about structuring, freezing, and reuse could inform future LLM-based synthesis tools. The use of 20 newly generated tasks per category is a genuine strength in guarding against contamination. However, the claim as stated is not supported by the reported evidence: Table 1 shows 0/100 solved for Category A Hard in the original format, and the human-LLM interactions in Section 4.4 and the supplementary material show that example solution-sequences and explicit structural constraints are supplied by the human. The paper therefore currently reads as a workflow report on a fairly heavily assisted human-LLM collaboration rather than as a demonstration that the method 'can generate code that solves tasks across all categories' without such assistance. With a properly qualified claim and additional ablations, the contribution could be both honest and useful; in its present form, the overclaim and the attribution problem are load-bearing.","major_comments":[{"comment":"The abstract and Section 5 claim success 'across all IPARC categories', but Table 1 reports 0/100 for Category A Hard (original) with input-output images only. The 100% row for that category uses intermediate snapshot images, which are not part of the standard IPARC task specification, and Table 3 shows only 29/100 with manual pruning and 99/100 with pruning plus randomization. The central claim therefore needs to be rephrased to state explicitly which task format and which human interventions are required, or the abstract and conclusions must be revised to avoid an unqualified 'across all categories' assertion.","section":"Abstract, §4.4, Table 1"},{"comment":"The solve rates are not cleanly attributable to the LLM, because the human corrective feedback includes structural information that effectively specifies the target program shape. In the supplementary transcript, the human pastes verbatim example solutions and then instructs the LLM to 'try 4 Dilations followed by 3 Erosions' with 'erosion SEs must be selected from the dilation SEs used', and Listing 1.1's generate_band2_sequences implements exactly that 4-dilation/3-erosion search with erosion SEs drawn from the dilation SEs. This is not mere correctness feedback; it is a human-supplied program skeleton. To support the paper's central attribution, the authors should either (a) reframe the contribution as a human-LLM collaboration in which the human supplies structural constraints, or (b) provide an ablation in which the prompts do not include example solution-sequences and the human restricts feedback to pass/fail or correctness information, and then report the resulting success rates.","section":"§4.4, Supplementary Table 5, Listing 1.1"},{"comment":"The empirical evaluation rests on a single human engineer with no repeated runs, no confidence intervals, and no released code or interaction logs. The claim that iStrucInd+ solves 'all tasks' in a category therefore cannot be independently verified, and the 100% figures could reflect a particular human's skill or the specific task ordering. The Data and Code Availability section promises public release only upon publication; to make the central claim reproducible, the authors should release the code, the exact prompts used, the human-LLM interaction logs, and ideally multiple independent runs or at least a documented protocol for selecting the human engineer.","section":"§4.3, §4.4, Data and Code Availability"},{"comment":"The baseline comparison is too weak to support the claim that the structured decomposition is what makes the approach succeed. The text says only that a 'best-guess, 1-shot prompt' failed on the simple variants, without giving details about prompt construction, number of attempts, or temperature settings, and it was not tried on the hard variants. A stronger baseline would keep the same interactive protocol and the same human effort but remove the DFD staging, or would compare against an unstructured interactive LLM session with the same number of interactions. Without such a control, the paper cannot rule out that the success comes primarily from the human's problem-solving and the LLM's role as a code editor, rather than from the proposed structured-induction mechanism.","section":"§4.4"}],"minor_comments":[{"comment":"The heading 'The IP ARC Challenge' contains an unintended space; elsewhere the paper consistently uses 'IPARC'.","section":"Section 2.1"},{"comment":"The name 'Djikstra' should be 'Dijkstra' in the first sentence and in the reference list.","section":"Section 1"},{"comment":"The 'Tasks Solved' column would be clearer with explicit denominators, e.g., 100/100, 0/100, and 29/100, to avoid ambiguity about whether the entries are counts or percentages.","section":"Table 1"},{"comment":"There is a discrepancy in the number of example solution-sequences used: Section 4.4 says two tasks are used for simple variants and four for hard variants, while the supplementary prompt for Category B Hard says '2 example' tasks are attached. Please clarify the exact numbers for each category and phase.","section":"§4.4 and Supplementary Material Stage 1"},{"comment":"The claimed reduction of the search space from 'about 69 billion combinations to about 12 million' is stated without a derivation; please provide the counting formula or a reference for these numbers.","section":"§4.5, Code Jogs Creativity"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful workflow-level contribution, but the current framing overclaims LLM capability relative to the evidence. The authors should be asked to either substantially revise the central claim to describe the human-assisted nature of the method or provide strong ablations that separate the LLM's contribution from the human-supplied structure. I would also encourage the editors to insist on code/data release before publication, since the current evaluation is a single-human, no-error-bar study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read on arXiv:2506.13820. The genuinely new thing is the first reported success on IPARC tasks, and that is real if you treat the system as a human–LLM team. The workflow—LLM proposes a DFD, human ratifies and refines, sub-programs are built and frozen—is a sensible extension of the authors' earlier iStrucInd work, and the insights about freezing correct code, code reuse, and LLM output sparking human ideas are plausible and worth hearing. I also credit them for validating on 20 newly generated tasks per category, which is a good check against memorization. So the paper has value as a workflow report and as a benchmark data point.\n\nThe soft spots are substantial, though. The abstract says \"successfully solves tasks across all IPARC categories,\" but Table 1 shows Category A Hard (original) at 0%. The 100% rows for that category require intermediate snapshot images—supervision outside the original task spec—or manual pruning plus randomisation. That is a genuine overclaim, not a nuance. Second, the human interventions in Section 4.4 and Supplementary Table 5 go beyond simple correctness feedback. The human pastes verbatim example solutions and instructs the LLM to search for \"4 Dilations followed by 3 Erosions\" with erosion SEs drawn from the dilation SEs. That is a structural constraint on the target program. The final code in Listing 1.1 then enumerates over that narrowed space. So the solve rates are at least as much a measure of the human's ability to infer the program shape as of the LLM's synthesis ability. The paper does disclose these interventions in the body, which keeps it honest, but the abstract and conclusion do not carry the same caveats.\n\nThe evaluation also rests on a single human engineer with no repeated runs or error bars, and neither code nor full interaction logs are released. That makes the quantitative claims hard to assess.\n\nAll that said, the central idea is not fundamentally flawed. A well-scoped version of this paper—one that says \"a structured human-in-the-loop approach can solve most IPARC tasks, with Cat A Hard needing extra supervision, and the human contributes significant structural insight\"—would be a decent contribution. I'd send it to peer review, but with a request for major revision: fix the abstract, be explicit about what the human contributes relative to the LLM, and ideally release the artifacts. As it stands, I wouldn't cite it in my own work because the headline claim is misleading and the evidence base is too thin.","headline":"A worthwhile human-in-the-loop program synthesis workflow, but the abstract overclaims and the human's structural hints do much of the synthesis work.","tokens_in":13577,"tokens_out":2430,"would_cite":false,"duration_ms":28909,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A structured human-LLM collaboration can synthesize working morphological programs for all categories of the IPARC benchmark, a first for the challenge.","keywords":["program synthesis","large language models","human-in-the-loop","structured programming","IPARC challenge","mathematical morphology","data flow diagrams","inductive programming"],"falsifier":"Re-run iStrucInd+ on newly generated IPARC tasks with a human who is allowed only generic feedback ('look for patterns') and never shown example solutions or told operation counts; if most tasks go unsolved, the human's structural hints are the load-bearing component.","tokens_in":12454,"feed_emoji":"🧩","tokens_out":5274,"duration_ms":58552,"temperature":0.7,"pith_summary":"This paper claims that a structured human-LLM collaboration, called iStrucInd+, can synthesize working programs for every category of the IPARC benchmark, a 600-task program-synthesis challenge that had previously resisted automated solutions. The method has a software engineer build a data-flow diagram, partly proposed by the LLM, and then interactively construct sub-programs using a four-tag communication protocol. On all categories except the hardest, the pipeline solves 100% of tasks without special help; for Category A Hard it needs either intermediate snapshots or human-added pruning and randomization. The authors report this as the first successful tackling of IPARC tasks, and draw lessons about structuring, freezing code, reusing code, and how LLM output can trigger human creativity. If correct, the work shifts the practical question from 'can an LLM write the whole program' to 'how should a human and LLM divide the structuring work'.","feed_headline":"Structured human-LLM method solves all IPARC task categories","feed_subtitle":"First reported success on the 600-task benchmark; human structuring, not raw codegen, is the key.","key_machinery":"The engine of the method is iStrucInd+, an interactive procedure in which an LLM first proposes a data-flow diagram (a labelled directed acyclic graph of sub-tasks with pre/post-conditions), the human ratifies, refutes, revises, or rejects that proposal, and then a second LLM-driven loop generates Python code for each sub-task using the same four-tag protocol. Correctly verified sub-programs are 'frozen' and reused, preventing later edits from corrupting them. The underlying program space is defined by mathematical morphology operations—dilation, erosion, and hit-or-miss with structuring elements—so the generated code enumerates or searches over sequences of these operators, with iteration expressed as repeated applications. The DFD is the load-bearing abstraction: it lets the human see the computation as composition of functions, catch hallucinations such as invented constraints, and inject search-space restrictions in a localized way.","core_discovery":"The central discovery is that IPARC's six task categories become solvable when program synthesis is decomposed into two LLM-assisted activities: proposing a data-flow diagram and interactively filling each block with morphological-operation code. The authors show that iStrucInd+ solves 100% of tasks in Category A Simple, all Category B variants, and Category A Hard when supplied with one intermediate snapshot or when the human inserts a pruning rule and randomization. The one-shot baseline, in which the LLM is asked to produce the morphological program directly, fails even on simple tasks. The paper therefore claims that structured decomposition, rather than raw code-generation ability, is what unlocks LLM performance on unfamiliar synthetic domains, and that human ratification and correction of the LLM's proposed structure is an essential part of the mechanism.","pith_inferences":["If the pattern holds on other synthetic benchmarks, the practical bottleneck is not LLM code generation but the human's ability to propose and verify a decomposition; investing in DFD tooling may pay off more than investing in larger models.","The snapshot result suggests that intermediate supervision—providing one checkpoint image—is a cheap and transferable way to make combinatorial morphology tasks tractable, a technique that could generalize to other image-transformation synthesis problems.","A testable extension: run iStrucInd+ on ARC itself with a morphology-like DSL; the paper's claim that human intuition is limited on IPARC does not tell us whether the method would also beat vision-based ARC solvers on problems where humans are strong.","The paper's own data hint that the human's structural hints are doing much of the work; a natural next experiment is ablating those hints to measure the LLM's independent structuring contribution."],"forward_implications":["Program-synthesis benchmarks like IPARC are not out of reach for LLM-based systems when the LLM is asked to propose structure (a DFD) rather than final code directly.","A human who can read and correct a data-flow diagram is sufficient to solve unfamiliar synthetic tasks, provided at least one intermediate snapshot is available for the hardest search spaces.","Freezing verified sub-programs during interactive development prevents regression bugs that appear as interactions grow longer.","Reusing and adapting verified code from related tasks is faster and more reliable than resynthesis from scratch, especially where sequence, selection, and iteration patterns repeat.","LLM-generated code can suggest search-space reductions to the human, as when inspecting exhaustive-search code led to a pruning rule that cut the candidate space from around 69 billion to about 12 million combinations."],"supporting_citations":[{"why":"Defines the IPARC challenge, its six categories, and the morphological-program target that the paper aims to solve.","marker":"[4]"},{"why":"Introduces the original iStrucInd structured inductive programming system that this paper extends with LLM-assisted DFD creation.","marker":"[26]"},{"why":"Supplies the two-way intelligibility protocol (RATIFY, REFUTE, REVISE, REJECT) used for human-LLM interaction.","marker":"[2]"},{"why":"The structured programming theorem that underlies IPARC's sequence/selection/iteration design, grounding the program search space.","marker":"[3]"},{"why":"The ARC benchmark that inspired IPARC and provides the input-output image-task format and the reasoning-focused framing.","marker":"[6]"},{"why":"Identifies the specific LLM used for sub-program construction; the empirical results depend on this model.","marker":"[20]"}],"fun_headline_variants":["Decomposition unlocks LLM on IPARC","Human-structured LLM beats IPARC tasks","Two-step LLM method masters IPARC suite","LLM structure, not raw code, wins IPARC","IPARC solved by human-guided LLM pipeline"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The human's corrective feedback must guide the LLM without implicitly supplying the target program's structure; if hints like 'try 4 dilations followed by 3 erosions' encode the solution shape, the reported solve rates are really measuring human problem-solving.","fun_headline_variants_meta":{"raw":{"variants":["Decomposition unlocks LLM on IPARC","Human-structured LLM beats IPARC tasks","Two-step LLM method masters IPARC suite","LLM structure, not raw code, wins IPARC","IPARC solved by human-guided LLM pipeline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00019,"raw_usage":{"total_tokens":1273,"prompt_tokens":816,"completion_tokens":457,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":432,"completion_tokens_details":{"reasoning_tokens":386}},"tokens_in":432,"tokens_out":457,"duration_ms":6090,"temperature":1.0,"reasoning_tokens":386,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:44:00.593925+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run iStrucInd+ on newly generated IPARC tasks with a human who is allowed only generic feedback ('look for patterns') and never shown example solutions or told operation counts; if most tasks go unsolved, the human's structural hints are the load-bearing component.","supporting_citations":[{"cited_title":"Challa, A","cited_arxiv_id":null,"evidence_quote":"Defines the IPARC challenge, its six categories, and the morphological-program target that the paper aims to solve."},{"cited_title":"Surana and A","cited_arxiv_id":null,"evidence_quote":"Introduces the original iStrucInd structured inductive programming system that this paper extends with LLM-assisted DFD creation."},{"cited_title":"Baskar, A","cited_arxiv_id":null,"evidence_quote":"Supplies the two-way intelligibility protocol (RATIFY, REFUTE, REVISE, REJECT) used for human-LLM interaction."},{"cited_title":"B¨ ohm and G","cited_arxiv_id":null,"evidence_quote":"The structured programming theorem that underlies IPARC's sequence/selection/iteration design, grounding the program search space."}],"review_version":1}