{"id":"d51ed366-e3ac-47f5-968d-74392cb51826","arxiv_id":"2608.11394","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"GraphAlignCoder fine-tunes a code LLM with graph-aligned, teacher-generated Lean proof-flow supervision and reports improved Pass@1 on LiveCodeBench v6 and BigCodeBench, but the proof obligations are not derived from an independent formal specification.","lead":"This paper trains a code model on program graphs and on Lean proof traces generated from the correct solution, then distills that structure back into plain Python generation. It reports gains over a base model and over CodeRL on LiveCodeBench v6 and BigCodeBench Hard, but the formal obligations are produced by an LLM reading the answer, not by an independent specification.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Lean kernel checks only that the teacher's invented obligations are provable, not that they describe the Python program's actual behavior; the paper's central attribution rests on that unverified bridge.","rationale":"The reader's weakest assumption identifies the same load-bearing gap: the semantic bridge between kernel-checked Lean obligations and hidden-test Python behavior is never established. I agree that this is the central risk. The paper's headline improvement over code-only SFT and CodeRL is internally consistent, but its claimed mechanism—that formal proof structure is transferred into code generation—requires the obligations to be both true of the reference solution and discriminative against incorrect solutions. Without that, the formal verification is a certificate of provability, not of program correctness. Other concerns, such as the under-specified CodeRL baseline, missing artifacts, and the ablated consolidation definition, are real but secondary; they affect reproducibility and attribution detail, not the core semantic assumption. The CONDITIONAL verdict remains appropriate: the concern is serious but addressable with a targeted validation of obligation entailment, and the paper should not be rejected outright if that validation succeeds.","tokens_in":11076,"tokens_out":9135,"duration_ms":86119,"concrete_test":"Run an entailment probe on the 50 LiveCodeBench v6 and 23 BigCodeBench Hard tasks solved by GraphAlignCoder. Reconstruct the accepted obligation set O_i from the saved scaffolds and compile each obligation into an executable assertion at its corresponding program region (loop invariant, branch guard, return postcondition). First, instrument the reference solution y_i and require every assertion to pass. Then instrument the code-only SFT and CodeRL solutions that fail the hidden tests, and require at least one obligation to be violated on each failing program; also require each assertion to be violated by at least one syntactically valid mutation so that it is not vacuous. If obligations pass on y_i but rarely reject failing programs—or fail on y_i—the formal proof-flow graph is not carrying hidden-test correctness and the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that GraphAlignCoder transfers formal correctness structure from proof-flow graphs into Python generation—depends on the obligations produced in Eqs. (2)–(4) being true specifications of y_i and strong enough to rule out programs that fail hidden tests. The paper never checks this. The teacher LLM sees y_i and the region inventory derived from y_i, invents Lean hints from a template library, and Lean's kernel verifies only that the instantiated proposition φ_ij is a theorem in the resulting context. A provable obligation can be vacuous, tautological, or unrelated to the Python execution; kernel checking does not connect φ_ij to the benchmark's hidden-test behavior. FilterC/FilterO (Eqs. 11, 13) filter for consistency with y_i and the implementation graph, not for entailment of hidden tests. Since the alignment A_i = C_I ∩ C_F (Eq. 10) intersects two descriptions both derived from y_i, agreement is expected even if the formal statements carry no semantic content. The method could therefore be distilling arbitrary GPT-5.2-generated annotations, and the reported gains would not show that formal proof structure is the active ingredient. This is the load-bearing assumption because the paper's contribution over code-only SFT is precisely the proof-graph supervision. The kernel check is genuine but narrow: it establishes φ_ij follows in the Lean context, not that φ_ij is a correct specification of y_i or of the hidden tests.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"GraphAlignCoder is a training framework that aims to inject formal correctness structure into Python code generation. For each benchmark task, the paper parses the reference solution y_i into an implementation graph (Eq. 1), then uses GPT-5.2 to generate Lean proof hints (Eq. 2) that are instantiated as kernel-checked Lean proofs (Eqs. 3–5) and extracted into a proof-flow graph (Eq. 6). The two graphs are projected into roles and aligned (Eqs. 8–10), filtered (Eqs. 11–13), and converted into auxiliary concept and obligation prediction targets during an injection phase (Eqs. 15–19); a subsequent consolidation phase (Eqs. 20–21) removes the auxiliary outputs and trains the model on ordinary task-to-Python examples. The paper reports Pass@1 results on LiveCodeBench v6 (50/175), BigCodeBench Hard (23/148), and BigCodeBench Full (363/1140), along with ablations distinguishing structural injection, consolidation, and their combination.","tokens_in":11419,"tokens_out":4226,"duration_ms":38941,"significance":"The underlying idea is timely and, if validated, would be a meaningful step toward using mechanically checked proof artifacts as supervision for code generation. The paper has concrete strengths: the Lean pipeline uses kernel checking rather than relying only on LLM self-report; the implementation graph is built deterministically from Python's AST; and the evaluation targets public benchmarks with a standard Pass@1 contract. The reported gains over code-only SFT and CodeRL are substantial on LiveCodeBench v6 and BigCodeBench Hard. However, the central scientific claim—that proof-flow structure is the active ingredient producing these gains—is currently not established, because the formal obligations are generated from the same reference solution y_i and are filtered by consistency with y_i rather than by their ability to predict hidden-test outcomes. The ablations also contain an attribution inconsistency that must be resolved before the component analysis can be interpreted. With an additional validation study and a clarified ablation, the contribution would be significant for the code-generation and program-reasoning communities.","major_comments":[{"comment":"The semantic fidelity of the proof-flow graph is unvalidated, and this is load-bearing for the paper's central attribution. The Lean kernel checks that the LLM-invented proposition φ_ij is provable in its local context, but it does not check that φ_ij specifies the actual behavior of y_i or that satisfying the obligation entails passing the hidden tests. Since both G_I(y_i) and the proof hints are derived from the same reference solution y_i, the alignment A_i = C_I ∩ C_F (Eq. 10) is expected even if the formal statements carry little semantic content. FilterC and FilterO (Eqs. 11 and 13) filter for consistency with y_i and with the implementation graph, not for entailment of the benchmark's hidden-test behavior. I ask the authors to add a direct validation of the obligation bridge: for example, measure whether model-generated programs that satisfy a predicted obligation have a higher hidden-test pass rate than those that do not, or compare GraphAlignCoder against an equally expensive control that uses arbitrary LLM-generated region annotations (without Lean) under the same filtering. Without such evidence, the reported gains could in principle come entirely from GPT-5.2-generated annotations, not from formal proof structure.","section":"Methodology, Eqs. (2)–(4) and Eqs. (11)–(13); Eq. (10)"},{"comment":"The ablation attribution is internally inconsistent. The contribution bullet states that structural injection increases LiveCodeBench v6 from 33 to 44 and that consolidation 'further raises' the count to 49, but Table 4's 'Consolidation-Only' row is initialized directly from the base model, not from the structural-injection checkpoint. Therefore the 49-task result is not evidence about the effect of adding consolidation after injection. Moreover, consolidation-only alone (49 tasks) already outperforms structural injection alone (44 tasks), which undercuts the text's claim in RQ3 that 'verification-graph injection produces the initial reasoning gain.' The authors should either report a true staged ablation (injection checkpoint → consolidation) or revise the attribution language to match the actual design.","section":"Evaluation, RQ3 and Table 4; contribution list"},{"comment":"The CodeRL comparison is not sufficiently specified. The text says only that CodeRL 'denotes the reported execution-feedback baseline,' without stating whether the numbers were obtained by rerunning CodeRL under the same base model, training data, decoding protocol, and evaluation harness as GraphAlignCoder, or whether they were copied from the original CodeRL paper or another prior report. Because the headline contribution claims are relative improvements over CodeRL (31.6% on LiveCodeBench v6 and 43.8% on BigCodeBench Hard), the authors must clarify the provenance of these numbers and, if they were not rerun, provide a same-contract rerun or explicitly discuss the mismatch.","section":"Evaluation, 'Baselines and Ablations' and Table 1"},{"comment":"No error bars, variance estimates, or multiple-seed runs are reported, and several of the decisive margins are very small: BigCodeBench Full 363 vs. 359 for CodeRL and 363 vs. 352 for consolidation-only, LiveCodeBench v6 50 vs. 49 for consolidation-only, and BigCodeBench Hard 23 vs. 19 in Table 4. Since the training pipeline is stochastic, the claim that GraphAlignCoder 'consistently outperforms' these baselines is not established for margins of one to four tasks. The authors should report variance over at least three seeds for the main models and ablations, or provide a stability analysis that justifies the comparisons.","section":"Evaluation, Tables 1 and 4"},{"comment":"The curriculum in Eq. (21) depends on the injection steps S_I and consolidation steps S_C, but only S_C = 65 is reported. The number of injection steps S_I, the learning rate, batch size, warmup, and the number of examples used for injection are not given. This is a reproducibility gap for a method whose entire contribution is a two-phase training schedule. The authors should report these hyperparameters, and ideally release the training code and filtered instruction data.","section":"Methodology, Eq. (21) and 'Training and Hardware Setting'"}],"minor_comments":[{"comment":"The Sorting row appears to contain a typo: 'Sorting 52 (.400)' should likely read 'Sorting 5 2 (.400)' to match the column format, since N=5 and the rates are 2/5.","section":"Table 3, Sorting row"},{"comment":"The abstract says GraphAlignCoder 'consistently outperforms' baselines across all benchmarks, but Table 2 shows that on BigCodeBench Easy tasks GraphAlignCoder solves 160 tasks versus 164 for code-only SFT and 163 for CodeRL. The text in RQ2 acknowledges this, but the abstract and contribution bullets should be qualified to avoid an overbroad claim.","section":"Abstract and Table 2"},{"comment":"The reference for the USENIX Security paper uses '35nd' instead of '35th'; please correct the ordinal.","section":"References"},{"comment":"The description of 'Consolidation-only' says it is trained from tasks 'augmented with graph-derived verification information,' but the consolidation phase in Eqs. (20)–(21) is defined as code-only training. Please clarify whether consolidation-only receives auxiliary text as input during training or only auxiliary objectives, since the current wording conflates the two.","section":"Evaluation, 'Baselines and Ablations'"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to interest the software-engineering and LLM-for-code community, and the kernel-checked proof-flow idea is a fresh angle. The main risk is that the central attribution to formal proof structure is not yet supported: the obligations are LLM-generated from the reference solution and never connected to hidden-test outcomes. This is fixable with a focused validation study and a control ablation, so I would not reject outright, but the current manuscript does not yet establish the claimed mechanism. I would also ask the editor to ensure the CodeRL numbers are clarified as either rerun under identical conditions or explicitly imported with all caveats."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The honest take: this paper has a novel and plausible training pipeline, but its central claim—that formal proof structure is what drives the gains—is not established. The proof-flow graph is generated by GPT-5.2 from the correct solution, and Lean's kernel checks only that the invented obligations are provable, not that they entail passing the hidden tests. So the \"verification\" signal is weaker than advertised. That said, the framework itself is new: building implementation graphs and proof-flow graphs, aligning them, and training on three complementary targets is not present in the cited literature. The ablation separating structural injection, consolidation-only, and the full model is a good design, and the results are internally consistent. The motivating example is clear and the writing is direct.\n\nWhere it gets soft: the unvalidated bridge between the proved obligations and the hidden-test behavior is load-bearing. A provable obligation can be vacuous or unrelated to the Python execution, and FilterC/FilterO check consistency with the correct solution, not entailment of tests. Since both graphs derive from the same y_i, agreement is expected even if the formal statements carry no semantic content. The method could be distilling arbitrary LLM-generated annotations, and the reported gains would not show that formal structure is the active ingredient. That is not a fatal flaw—auxiliary supervision can still help—but the paper overstates what the kernel check certifies. Additional concerns: the CodeRL baseline is under-specified (was it rerun or taken from the original paper?), there are no error bars (23 vs 16 on BCB Hard is a small count), no code or data are released, and the contribution list has an inconsistency: it says consolidation \"further raises\" from 44 to 49, but consolidation-only is an independent ablation started from the base model, not a continuation of the injection checkpoint. The full model gets 50, so the narrative is muddled.\n\nWho is this for? Researchers in verification-guided code generation or multi-task training for code LLMs. It is a reasonable conference submission. With released artifacts, a clarified baseline, variance estimates, and a corrected interpretation of what the proof-flow graph actually certifies, it could become a solid contribution.\n\nRecommendation: I would send it to peer review. The framework is plausible, the empirical pattern is consistent across benchmarks, and the central weakness is addressable. Ask for a major revision that either demonstrates the obligations are non-vacuous and test-relevant, or re-frames the contribution as LLM-generated structural supervision rather than formal verification.","headline":"Promising graph-alignment training pipeline, but the claim that formal proof structure is the active ingredient is not backed by the evidence; worth peer review with revisions.","tokens_in":11903,"tokens_out":2807,"would_cite":false,"duration_ms":25759,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GraphAlignCoder trains a code model to organize programs around correctness obligations by aligning an implementation graph with a Lean proof-flow graph, then consolidating into Python-only generation.","keywords":["code generation","program graphs","formal proof","Lean","correctness obligations","LLM training","verification-graph injection","pass@1"],"falsifier":"One concrete test: collect a sample of tasks, generate the formal hint sets and proof-flow graphs exactly as in the paper, and check whether there exists a program that satisfies every extracted region obligation but fails the official hidden tests; the first such counterexample would show the obligations underdetermine correctness. A second test: replace the task-aligned Lean hints with kernel-checked proofs of arbitrary true statements, keeping the same graph topology but no semantic alignment, and retrain; if pass@1 stays at 50/175, the alignment, not the formal verification structure, is carrying the result.","tokens_in":10884,"feed_emoji":"🧩","tokens_out":7687,"duration_ms":89788,"temperature":0.7,"pith_summary":"This paper tries to establish that code large language models can be trained to organize programs around correctness obligations by aligning an implementation graph—nodes and edges for branches, loops, calls, and returns—with a formal proof-flow graph extracted from kernel-checked Lean proofs. The payoff would be a training signal richer than pass/fail execution feedback but more grounded than free-form natural-language critique. The paper reports that the method solves 50/175 LiveCodeBench v6, 23/148 BigCodeBench Hard, and 363/1140 BigCodeBench Full tasks, improving over the base model, code-only SFT, and CodeRL. The central insight is that formal proof structure and executable program structure share a latent organization: proof case splits mirror program branches, invariant obligations mirror loop bodies, and witness construction mirrors return assembly.","feed_headline":"Proof-flow graphs lift code model to 50 solved tasks","feed_subtitle":"GraphAlignCoder's aligned proof graphs outperform execution-feedback training across three code benchmarks.","key_machinery":"The carrying object is the aligned scaffold $S_i = (G^I_i, G^F_i, H^F_i)$, built from the implementation graph $G^I_i = (V^I_i, E^I_i, R_i)$ parsed from the Python AST and the proof-flow graph $G^F_i$ extracted from Lean tactic-goal trajectories. Region groups $R_i$ include guards, loops, calls, recursive sites, updates, and returns; proof edges carry labels for case analysis, induction, rewriting, invariant propagation, and witness construction. Their overlap $A_i$ is used to filter region-level conditions, and the two-phase curriculum $L_{\\text{inject}} = L_{\\text{code}} + L_{\\text{concept}} + L_{\\text{obligation}}$ followed by consolidation $L_{\\text{con}}$ transfers the structure into ordinary Python generation. This scaffold is what lets the model learn why individual regions are correct instead of only whether the whole program runs.","core_discovery":"GraphAlignCoder claims that correctness structure can be transferred into code generation through graph alignment rather than through execution outcomes. On a reference solution, the framework deterministically builds an implementation graph whose regions include guards, loops, updates, and returns; in parallel, a constrained teacher LLM produces Lean hints that Lean elaborates into kernel-checked proofs, from which a proof-flow graph is extracted by recording goal-state transitions such as case splits, induction, rewriting, and witness construction. The alignment $A_i = C^I_i \\cap C^F_i$ keeps only implementation roles supported by both graphs, and the model is trained on three objectives—executable code, shared verification concepts, and region-grounded obligations—before a consolidation phase retrains the same checkpoint on Python-only examples and removes auxiliary channels. On the paper's evaluation contract of greedy single-sample pass@1, this yields 50/175 LiveCodeBench v6, 23/148 BigCodeBench Hard, and 363/1140 BigCodeBench Full, with the largest gains on hard problems and on categories requiring sequenced dependent operations such as filesystem/OS and web/text processing.","pith_inferences":["Inference: the method may work partly as structured data augmentation rather than as genuine verification, since the teacher writes the obligations and the proof checks only that the teacher's invented statements are true; comparing against non-verification scaffolds with the same graph topology would isolate the formal-proof contribution.","Inference: because the consolidated model is trained on only 514 code-only examples, the results suggest a small amount of structure-aware curriculum can yield large pass@1 shifts; testing with fewer examples could estimate the sample efficiency of the approach.","Inference: the same alignment mechanism could be applied to program repair, where failing tests define region-level obligations and the proof-flow graph proposes which branch, guard, or update must change.","Inference: the paper does not show that satisfying all generated obligations entails passing the hidden tests, so a testable extension would be to measure how often a program satisfying the extracted obligations fails the official suite."],"forward_implications":["The reported results imply that execution-feedback reinforcement learning is not required to obtain these gains; proof-flow structure can serve as the semantic training signal.","Deployed inference needs only ordinary Python generation, because auxiliary graph outputs are removed during consolidation.","The ablation results imply that structural injection alone shifts the model away from broad Python capability, so consolidation is a required component, not an optional cleanup.","The improvements concentrate on hard and dependent-operation categories, suggesting the framework helps most where hidden constraints require preserving intermediate state and constructing a valid final object."],"supporting_citations":[{"why":"Supplies the CodeRL execution-feedback baseline that GraphAlignCoder must beat and does beat on all three benchmarks.","marker":"(Le et al. 2022)"},{"why":"Supplies Lean as the proof assistant whose kernel-checked tactic traces become the proof-flow graph.","marker":"(de Moura et al. 2015)"},{"why":"Defines the program dependence graph notion that underlies the implementation-graph regions.","marker":"(Ferrante, Ottenstein, and Warren 1987)"},{"why":"Supplies the correctness-obligation view that the paper's region-level conditions formalize.","marker":"(Floyd 1967; Hoare 1969)"},{"why":"Provides the classical compiler-analysis background for control flow, data flow, and program structure used in graph construction.","marker":"(Aho et al. 2006)"},{"why":"Supplies the LiveCodeBench v6 held-out evaluation benchmark and its contamination-free protocol.","marker":"(Jain et al. 2025)"},{"why":"Supplies the BigCodeBench evaluation benchmark, including the Hard subset used for the main comparison.","marker":"(Zhuo et al. 2025)"},{"why":"Motivates using Lean proof state and retrieval structure as a source of process-level supervision.","marker":"(Yang et al. 2023)"}],"fun_headline_variants":["Proof graphs push code LLM to 50 solved tasks","GraphAlignCoder: correctness transfer beats execution feedback","Aligned proof graphs add 43.8% on hard code tasks","Code generation learns from proof-flow, not just execution","GraphAlignCoder: proof-flow alignment lifts solved count by 31.6%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the teacher-generated, kernel-checked Lean obligations faithfully encode the hidden-test correctness constraints of each task; the paper never validates that a program satisfying those obligations would pass the hidden tests, so if the teacher's formal hints are semantically off-target, the whole scaffold teaches structure that may be unrelated to true correctness.","fun_headline_variants_meta":{"raw":{"variants":["Proof graphs push code LLM to 50 solved tasks","GraphAlignCoder: correctness transfer beats execution feedback","Aligned proof graphs add 43.8% on hard code tasks","Code generation learns from proof-flow, not just execution","GraphAlignCoder: proof-flow alignment lifts solved count by 31.6%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000893,"raw_usage":{"total_tokens":3876,"prompt_tokens":995,"completion_tokens":2881,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":2793}},"tokens_in":611,"tokens_out":2881,"duration_ms":19883,"temperature":1.0,"reasoning_tokens":2793,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:12:29.873602+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One concrete test: collect a sample of tasks, generate the formal hint sets and proof-flow graphs exactly as in the paper, and check whether there exists a program that satisfies every extracted region obligation but fails the official hidden tests; the first such counterexample would show the obligations underdetermine correctness. A second test: replace the task-aligned Lean hints with kernel-checked proofs of arbitrary true statements, keeping the same graph topology but no semantic alignment, and retrain; if pass@1 stays at 50/175, the alignment, not the formal verification structure, is carrying the result.","supporting_citations":[],"review_version":1}