{"id":"a2c765a6-5c83-43b9-b75b-42d46d3fbf97","arxiv_id":"2505.23881","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LLM-generated search heuristics run through the CPro1 protocol with the reasoning model o3-mini-high produced verified constructions resolving open instances in 7 Handbook design families and newer problems.","lead":"A protocol that asks reasoning AI models to write C search programs solved listed open instances in 7 combinatorial design families, with verified new constructions included. The result suggests AI-written heuristics can close small open problems that manual search left unresolved.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline existence claims rest on Python verifiers and open-status lists that are not audited in the manuscript; a definitional mismatch between a verifier and the cited design definition (e.g. BTD pair counting) would invalidate a claimed resolution.","rationale":"The reader's weakest assumption is essentially the same as mine: verifier correctness and open-status completeness. I agree these are the load-bearing points; the individual existence claims do not depend on reproducibility of the LLM run, because the produced designs and code are fixed artifacts. My stress-test sharpens the reader's concern: the risk is not only a coding bug but a semantic mismatch between the prompt-level definitions and the Handbook definitions used to declare an instance open. The BTD example is a concrete place where such a mismatch could hide. This does not amount to an objection to the method; it is a reason to keep the verdict CONDITIONAL rather than ACCEPT until an independent verification and open-status audit is performed. A single full run and heuristic problem selection affect broader claims about LLM capability, but the existence results stand or fall on the verifier and open-status audit. Therefore I recommend no change to the reader's CONDITIONAL verdict.","tokens_in":21476,"tokens_out":11422,"duration_ms":118219,"concrete_test":"Run an independent verification pass on the repository: write fresh verifiers for BTD, BRD, and SymmW directly from the 2006 Handbook definitions, resolving the BTD pair-count convention before running, and execute them on every posted solution for those families alongside the supplied verifiers. Then check Dinitz's Part V/VI update pages and the cited 2023 symmetric-weighing-matrix result to confirm the listed instances, especially SymmW(22,16) and BTD(16,22;9,1,11;8,5), were still open. Any rejected solution, or any prior construction found, invalidates the corresponding headline result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Central claim: CPro1 produced explicit constructions that satisfy the definitions of seven Handbook design types and resolve listed open instances. Since each construction is posted and machine-verifiable, the only load-bearing condition is that the supplied verifier implements exactly the mathematical definition used by the cited open-instance literature, and that the open-instance lists are complete. The verifier code is not reproduced in the manuscript; Appendix Tables 5-8 are prompt definitions and may differ from the Handbook. A concrete example is BTD: the paper defines pair occurrence as sum_b m_vb m_wb = L, so a block containing v twice and w once contributes two to the pair count. If the Handbook's BTD counts one occurrence per block, the certified (16,22;9,1,11;8,5) solution need not be the design whose existence was open. Similar convention risks exist for JCC (vertex vs edge cover) and Covering Sequence (length at most L vs exactly L). Separately, 'newly solved' depends on the completeness of citations [8,13,16,18] and a hand-filtered Feb. 2025 scan; a missed prior construction would affect novelty, not existence, of a posted design. The single nondeterministic run and selection effects, disclosed in Section 5, weaken generalization claims but not individual existence statements.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies the Constructive Protocol CPro1, an LLM-based pipeline that generates and tunes search heuristics in C, to open existence instances of combinatorial designs. Using the reasoning model o3-mini-high, it reports solving open instances for 7 of 16 design types selected from the 2006 Handbook of Combinatorial Designs, including three types (Bhaskar Rao Designs, Symmetric Weighing Matrices, Balanced Ternary Designs) that were not solved by an earlier CPro1 run with a non-reasoning model. It also reports new Covering Sequences, Johnson Clique Covers, Uniform Nested Steiner Quadruple Systems, and improved Deletion Codes relative to a recent FunSearch-based result. The existence claims are supported by explicit constructions listed in the appendix and repository, each checked by a Python verifier rather than by the LLM's own assertion. The paper includes ablations, scaled-down repeated runs, and a limitations section that discloses single-run, nondeterministic behavior and the non-existence-proof limitation.","tokens_in":21749,"tokens_out":8037,"duration_ms":76980,"significance":"If the listed constructions and their verifiers are correct, the paper provides a notable positive demonstration that LLM-generated heuristics can resolve open existence questions in combinatorics, with independently checkable artifacts. The contributions include concrete constructions for several previously open parameter sets, a clear protocol, ablations showing the role of each pipeline component, and repeated scaled-down runs that give some evidence against pure luck. The main risk is not circularity: the load-bearing outputs are explicit arrays/sequences/codes checked by an external verifier. The significance is somewhat tempered by the paper's own disclosure that these are single full-scale runs and that some successes come from the prototyping set used to develop the method.","major_comments":[{"comment":"The central load-bearing artifact is the Python verifier, but the manuscript does not reproduce it or provide a formal mapping from each verifier check to the cited mathematical definition; the appendix only gives prompt definitions. Because Algorithm 1 certifies outputs solely through this verifier, a definitional mismatch would invalidate the claimed existence of the listed designs. Please include the verifier source for each design type and state explicitly which definitional convention is used.","section":"Appendix Tables 5–8; Algorithm 1"},{"comment":"The appendix defines pair occurrence in a Balanced Ternary Design as sum_b m_{vb} m_{wb} = L, so a block containing v twice and w once contributes 2 to the pair count. If the Handbook or the open-instance literature counts each block at most once per pair, then the certified (16,22;9,1,11;8,5) array would not be the object whose existence was open. Please state the convention for BTD pair occurrence and confirm that the verifier matches the literature.","section":"Table 5 (BTD definition)"},{"comment":"Two other definitional conventions need confirmation. The Covering Sequence definition in Table 7 says 'length at most L', but Table 1(b) lists L as the solved parameter; if the open instances in [5] are stated for exact length L, a shorter sequence does not resolve them. For Johnson Clique Cover, the appendix defines a cover of all vertices, while some graph-theoretic sources define a clique cover of all edges; the manuscript should confirm which notion is used in [25] and that the verifier matches it.","section":"Table 7 (Covering Sequence; Johnson Clique Cover)"},{"comment":"The claim that 7 of 16 Handbook problems are 'newly solved' depends on the completeness of the open-instance lists taken from [7] and the updates [8, 13, 16, 18], plus the hand-filtered February 2025 scan. The manuscript does not audit these lists per problem. A missed prior construction would not affect the existence of the posted design, but it would affect the novelty claim; please supply an open-status audit or cite authoritative updated tables for each problem.","section":"Section 3.1; Table 1"}],"minor_comments":[{"comment":"The caption reports p<0.0001 from a Z-test on 25 scaled-down runs; please state whether the four comparisons are adjusted for multiple testing and clarify the null hypothesis.","section":"Section 4.2, Figure 1"},{"comment":"The Covering Sequence appendix states n<=16 and L<=1200, but Table 1(b) includes n up to 20 and L up to 207000; indicate whether these are the 'adjacent instance' extensions described in Section 4 and correct the stated range.","section":"Table 7 versus Table 1(b)"},{"comment":"In the Deletion Code definition, 'deleting two distinct bits' should read 'deleting s distinct bits'.","section":"Table 8"},{"comment":"The caption says 'for each LLM and total number of candidate programs' but no candidate-count column is shown; please correct the caption.","section":"Table 3 caption"},{"comment":"The open-status updates [8, 13, 16, 18] are cited as URLs and dates; provide version or access-date information so the open-status audit is reproducible.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The main risk is verifier/definition mismatch; I would recommend requesting the verifier source code and an open-status audit before acceptance. If those check out, the existence claims are strong and the paper would be publishable. The contribution is incremental but useful as evidence for LLM-generated heuristics in combinatorial search."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the reason to read this paper is not the LLM protocol—CPro1 is largely inherited from the author's earlier arXiv paper—but the verified constructions. The headline result is a 22x22 symmetric weighing matrix with w=16, an instance open since the 2006 Handbook, plus new BRDs, BTDs, and improved deletion codes. The constructions are posted, each is checked by an independent Python verifier, and the appendix shows at least one solution per family. That is real, checkable evidence.\n\nWhat the paper does well: it is honest about what is and isn't new, includes ablations and repeated scaled-down runs, and discloses the single-run, nondeterministic nature of the experiments. The failure on Cap Sets and the note about the later FPGA result on DTS are the kind of negative results that make the limitations section trustworthy.\n\nSoft spots, in proportion. The aggregate claims—'7 of 16 problems, 3 newly solved'—rest on one full-scale run per problem and a hand-filtered literature scan. That makes the rates fragile, but the individual existence results are not fragile in the same way: a posted construction either satisfies the definition or not. The real vulnerability is that the verifier code and the open-status audit are outside the manuscript. For peer review, the verifier source should be in the paper or a frozen repository, and the open-instance list should be checked against the cited sources. The stress-test note raises BTD pair counting as a possible convention mismatch. I don't think that one lands: the paper defines blocks as multisets and counts pairs as sum_b m_vb m_wb, which is the standard BTD convention. There is a more plausible convention worry for Covering Sequences, where the prompt says 'length at most L' while the open-instance table lists a target length L; a shorter cyclic sequence would not settle existence at exactly L. I'd want that clarified. Similarly, 'newly solved' depends on the completeness of citations, though a missed prior construction would only hurt novelty, not existence.\n\nBottom line: the paper is worth a serious referee. The method is incremental, but the verified constructions are concrete, the failures are reported, and the work is reproducible enough to audit. I'd send it to review with a request to make verifiers and open-status checks explicit.","headline":"Verified constructions, not the LLM protocol, are what make this paper worth a referee's time.","tokens_in":22325,"tokens_out":4047,"would_cite":true,"duration_ms":40821,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["05B30"],"pacs":[],"model":"deepseek-v4-flash","headline":"A code-generating automated protocol, CPro1, guided by a reasoning large language model, constructs verified solutions to long-open combinatorial design problems, resolving open cases for 7 of 16 problem types from the 2006 Handbook of…","keywords":["combinatorial design","open instances","large language models","reasoning models","code generation","search heuristics","existence problems","verified construction"],"falsifier":"Recompute each claimed solution independently: parse the arrays and code words from the paper's published outputs and check them against the design definitions without relying on the supplied verifier, then audit each instance's status against current authoritative tables. If any array fails the definitional constraints or an instance turns out to have been solved before, the central claim has an error.","tokens_in":21255,"feed_emoji":"🧩","tokens_out":8226,"duration_ms":64947,"temperature":0.7,"pith_summary":"This paper claims that an automated protocol, CPro1, can use a reasoning large language model to write and tune search programs that construct explicit solutions to long-open combinatorial design existence problems. The authors report verified solutions for 7 of 16 problem types selected from the 2006 Handbook of Combinatorial Designs, including cases in Bhaskar Rao Designs, Symmetric Weighing Matrices, and Balanced Ternary Designs that CPro1 with non-reasoning models had not solved. They also report new Covering Sequences, Johnson Clique Covers, Deletion Codes, and a Uniform Nested Steiner Quadruple System from recent literature. If the results hold, the listed instances are no longer open and the approach offers a way to attack small existence questions without writing bespoke mathematics proofs.","feed_headline":"Reasoning AI solves open cases for 7 of 16 design problems","feed_subtitle":"The CPro1 protocol with a reasoning model also solved three problems that non-reasoning models missed.","key_machinery":"The load-bearing object is the Constructive Protocol CPro1: an automated experiment loop that turns a textual design definition and a Python validity verifier into candidate search programs. It generates 1,000 candidate C implementations from LLM proposals, tunes each candidate's hyperparameters on development instances using the verifier as a score function, keeps the top five for code optimization, selects the top two after two-hour runs, and finally runs those for 48 hours on open instances, outputting only results that pass the verifier. The reasoning LLM supplies the heuristic ideas -- simulated annealing, tabu search, depth-first search, genetic algorithms, and two-phase decompositions -- while the verifier supplies correctness.","core_discovery":"The central discovery is that a protocol which repeatedly asks a reasoning LLM to propose diverse search strategies in C, then tunes hyperparameters, scores candidates against known development instances, optimizes the fastest candidates, and lets survivors run for 48 hours on open instances, can produce verified combinatorial designs where existence was previously unknown. The paper's specific claim is that CPro1 with the o3-mini-high reasoning model resolves open instances for 7 of the 16 Handbook problems, and that for three of those -- Bhaskar Rao Designs, Symmetric Weighing Matrices, and Balanced Ternary Designs -- the reasoning model succeeds where the same protocol with a non-reasoning model did not. The outputs are explicit arrays, matrices, or code words that pass a Python verifier implementing the design definition.","pith_inferences":["Extension: If the verifier-based protocol can be trusted, the bottleneck for automating combinatorial existence shifts from the search itself to producing correct verifiers and accurate open-instance lists; one could test this by applying the same pipeline to a fresh blind set of open instances.","Extension: The two-phase strategies that emerged (build the 0/1 incidence pattern before assigning signs, or build a Steiner system before nesting pairs) suggest that reasoning models can rediscover decomposition principles that are not explicitly prompted, which would be worth studying as a general pattern in LLM-guided search.","Extension: The paper's single full run per problem leaves stochasticity untested; a natural next experiment would be repeated runs on the same problems to measure run-to-run success rates and whether the solved instances are stable.","Extension: Because the successful programs are randomized heuristics with little visible structure, the method proves existence but does not automatically yield human-readable mathematical constructions; an independent mathematical verification of the published arrays could turn these computational discoveries into theorems."],"forward_implications":["For every solved instance listed in the paper's results, an explicit certified construction now exists, so the existence question for those parameter values is settled.","The three newly solved Handbook classes (Bhaskar Rao Designs, Symmetric Weighing Matrices, Balanced Ternary Designs) gain at least one new instance each, shrinking the open-instance lists for those problems.","The Deletion Code results improve on the previously reported LLM-search state of the art for small parameters, providing larger sets of codewords than earlier methods.","The protocol provides a template for attacking other small open existence questions: supply a definition, a verifier, and development instances, and let the protocol search for a heuristic.","For problem types where the community has already invested heavily in specialized search, such as Covering Arrays, the protocol does not necessarily compete; its successes come from less-explored design types."],"supporting_citations":[{"why":"Provides the 2006 Handbook problem selection and the open-instance lists for the 16 Handbook design types.","marker":"[7]"},{"why":"Defines CPro1 and reports the earlier non-reasoning model baseline results that this paper extends.","marker":"[40]"},{"why":"Supplies the reasoning model used for the main CPro1 runs in this paper.","marker":"[37]"},{"why":"Supplies a second reasoning model used in the prototyping-set comparison.","marker":"[19]"},{"why":"Updates the status of the symmetric weighing matrix with n=23 and w=16 as solved, identifying n=22 as a remaining open instance.","marker":"[13]"},{"why":"Lists the open Covering Sequence instances from February 2025 that CPro1 solves.","marker":"[5]"},{"why":"Lists the open Johnson Clique Cover instances that CPro1 solves.","marker":"[25]"},{"why":"Introduces the Uniform Nested Steiner Quadruple System open instance that CPro1 solves.","marker":"[4]"},{"why":"Provides the earlier LLM-search Deletion Code results that CPro1 replicates and improves.","marker":"[51]"},{"why":"Introduces the earlier LLM-search protocol and the Cap Set problem used as a comparison target.","marker":"[39]"}],"fun_headline_variants":["Reasoning LLM cracks 7 open combinatorial design problems","CPro1 reasoning model solves 7 open design instances","Reasoning models solve 3 designs missed by non-reasoning","7 of 16 open design problems solved by reasoning AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Python verifier encodes each design definition correctly and that the cited sources' open-instance lists are complete; if either fails, a listed 'solved' instance may not actually be solved or may not have been open.","fun_headline_variants_meta":{"raw":{"variants":["Reasoning LLM cracks 7 open combinatorial design problems","CPro1 reasoning model solves 7 open design instances","Reasoning models solve 3 designs missed by non-reasoning","7 of 16 open design problems solved by reasoning AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000443,"raw_usage":{"total_tokens":2245,"prompt_tokens":946,"completion_tokens":1299,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":1230}},"tokens_in":562,"tokens_out":1299,"duration_ms":10683,"temperature":1.0,"reasoning_tokens":1230,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:39:08.336133+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute each claimed solution independently: parse the arrays and code words from the paper's published outputs and check them against the design definitions without relying on the supplied verifier, then audit each instance's status against current authoritative tables. If any array fails the definitional constraints or an instance turns out to have been solved before, the central claim has an error.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the 2006 Handbook problem selection and the open-instance lists for the 16 Handbook design types."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines CPro1 and reports the earlier non-reasoning model baseline results that this paper extends."},{"cited_title":"OpenAI o3-mini system card, 2025","cited_arxiv_id":null,"evidence_quote":"Supplies the reasoning model used for the main CPro1 runs in this paper."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Updates the status of the symmetric weighing matrix with n=23 and w=16 as solved, identifying n=22 as a remaining open instance."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Lists the open Covering Sequence instances from February 2025 that CPro1 solves."},{"cited_title":"On the clique covering numbers of Johnson graphs","cited_arxiv_id":"2502.15019","evidence_quote":"Lists the open Johnson Clique Cover instances that CPro1 solves."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the Uniform Nested Steiner Quadruple System open instance that CPro1 solves."},{"cited_title":"LLM-Guided Search for Deletion-Correcting Codes","cited_arxiv_id":"2504.00613","evidence_quote":"Provides the earlier LLM-search Deletion Code results that CPro1 replicates and improves."},{"cited_title":"Romera-Paredes, M","cited_arxiv_id":null,"evidence_quote":"Introduces the earlier LLM-search protocol and the Cap Set problem used as a comparison target."}],"review_version":1}