{"id":"5c83b25e-ea1e-46eb-93b2-a7d9d2764aba","arxiv_id":"2412.00314","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"HuCoSC scores generated code by recursively decomposing code into sub-blocks, storing dependency semantics, and using an LLM to compare the semantic descriptions of reference and generated code.","lead":"This paper introduces HuCoSC, an LLM-based method that evaluates generated code by recursively decomposing code into small semantic pieces, storing their meanings, and comparing the reference and generated code semantically. In tests, its scores correlated more closely with human experts and with code execution than existing code-quality metrics.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Problem-statement injection, prompt design, and recursion threshold are tuned on the evaluation sets; the headline correlations are in-sample, and metric labels conflict (Spearman vs Pearson).","rationale":"The paper's contribution is an empirical one: a recursive semantic-comprehension pipeline that scores code, and evidence that its scores correlate better with human experts and execution than existing methods. What would have to be true is that Table I's correlations are unbiased estimates, not artifacts of choosing prompts and policies on the same data. The text shows they are not: the problem-statement policy PInitial was adopted after comparing PFull and PLack on Code-Pair (RQ3); the recursion threshold is fixed without a development set; and the baselines' prompts are described as designed to achieve their best performance (V.A.2) with access to the same test data. This is a correctness risk, not a disagreement with consensus. The stored-semantics concern raised by the reader is real and is partially supported by RQ2's depth degradation, but it is secondary: even an imperfect semantic store could yield good correlations. The first-order check is whether the claimed advantage survives without access to the test set. The metric-label discrepancy (Introduction says Spearman; Table I reports Pearson as rs) and duplicate rows in Table I make the reported numbers hard to rely on even before that check. I therefore keep the reader's CONDITIONAL verdict: the idea is coherent and the mechanism is plausible, but the evidence is not yet sufficient to accept the headline claim.","tokens_in":15071,"tokens_out":8249,"duration_ms":86292,"concrete_test":"Split Code-Pair by problem (85 CodeNet tasks) into three folds, and hold out HumanEval entirely. Use only fold 1 to fix the recursion threshold (1..5), the problem-statement injection mode (PInitial/PFull/PLack), the prompt and few-shot templates, and the choice among available GPT models. Lock the configuration, then compute Pearson and Kendall correlations on folds 2 and 3 and on HumanEval, clarifying exactly which code samples and execution labels are used (HumanEval has 164 problems, not 500). Report bootstrap 95% confidence intervals for the HuCoSC-versus-best-baseline gap. If the held-out gap is not significantly positive, the claim 'surpasses all existing methods' should be scaled back to 'can be competitive after tuning on a development set.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that HuCoSC outperforms all baselines depends on the correlation numbers in Table I being unbiased estimates of performance on new code-evaluation tasks. That condition is not met. The paper's own narrative shows the design was chosen using the evaluation data: Section IV.C fixes the recursion threshold at 3; Section III.C and RQ3 (Fig. 8) compare PFull, PLack, and PInitial on Code-Pair samples and then adopt PInitial as the method; the few-shot examples, evaluation criteria, and prompt templates are described as 'designed' without a separate development corpus. There is no train/validation/test split and no release of code, data, or prompts, so every number in Table I is a post-selection best case. This matters because the reported advantage of HuCoSC over its closest baseline (e.g., GPT-4 Turbo few-shot: .853 vs .746 on Code-Pair) is smaller than the swing caused by changing the problem-statement policy (RQ3 reports a .132-.18 drop for PFull). If the policy and threshold were chosen after seeing the outcome, the headline gap is not out-of-sample evidence. The label inconsistency (Introduction reports 0.769/0.853 as Spearman; Table I reports them as Pearson with Kendall tau values .662/.780) further prevents interpretation and suggests the reporting pipeline itself needs correction. The strongest claim is therefore CONDITIONAL at best pending a genuinely held-out evaluation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HuCoSC, an LLM-based framework for code quality evaluation that recursively decomposes code into sub-codes, stores and updates dependency semantics in a Semantic Dependency Decoupling Storage, and finally scores generated code by semantic comparison against a reference. The authors evaluate HuCoSC on a new Code-Pair dataset using human expert scores and on HumanEval using execution correctness, and report correlations that they claim surpass all existing match-based and LLM-based baselines. The central claim is that recursive semantic comprehension yields substantially higher agreement with human judgment and with code execution than direct LLM prompting or token-based matching.","tokens_in":15332,"tokens_out":4898,"duration_ms":45262,"significance":"The recursive semantic comprehension idea is plausible and, if the reported results hold under a properly controlled evaluation, would be a useful contribution to LLM-based code evaluation. The construction of the Code-Pair dataset with expert ratings is a valuable resource, and the paper includes illustrative examples where token-based metrics fail while HuCoSC succeeds. However, the current evidence does not establish the headline claim: the metric labels are inconsistent, the HumanEval sample size as stated is impossible, design choices appear to have been made after seeing the evaluation data, and no statistical uncertainty is reported. The paper is therefore not yet ready for publication, though the approach is sufficiently promising that the issues may be fixable with a substantially revised evaluation.","major_comments":[{"comment":"The correlation metric reported for the headline results is inconsistent. The Introduction reports 0.769 and 0.853 as Spearman correlations, while Table I labels the same values as Pearson rs and also reports Kendall tau values, and Section IV.E states that only Kendall-Tau and Pearson are used. Since every central comparison in the paper rests on these coefficients, the authors must state which metric was actually computed and recompute all tables under a single, clearly specified convention.","section":"Section I, Section IV.E, Table I"},{"comment":"The paper states that 'Due to the extensive size of the HumanEval dataset, we randomly selected 500 cases to serve as our validation set,' but the HumanEval dataset contains 164 problems. This makes the reported HumanEval correlations impossible as written. The authors must clarify what was sampled (for example, 500 generated programs drawn from the 164 problems) and rerun or relabel the analysis accordingly.","section":"Section IV.B.2"},{"comment":"The headline results appear to be in-sample after design choices were made on the evaluation data. Section IV.C fixes the recursion threshold at 3 without a sensitivity study, and Section V.C compares PFull, PLack, and PInitial on the Code-Pair dataset and then adopts PInitial, with Table I reporting results for that selected policy. Because there is no train/validation/test split and no separate development corpus, the reported advantage over baselines (e.g., GPT-4 Turbo few-shot .853 vs .746 on Code-Pair) can reflect selection rather than a general property. A genuinely held-out evaluation, or a nested validation procedure, is required to support the claim that HuCoSC surpasses existing methods.","section":"Section IV.C, Section V.C, Table I"},{"comment":"No statistical uncertainty is reported for any correlation in Table I. There are no confidence intervals, p-values, or tests of whether the difference between HuCoSC and the closest baseline is significant, and the stochasticity of the LLM outputs is not quantified despite the temperature being set to 0.2 rather than 0. The claim that HuCoSC 'surpasses all existing methods' is therefore not supported by significance testing.","section":"Table I, Section V.A"},{"comment":"The fidelity of the Semantic Dependency Decoupling Storage is load-bearing but unvalidated. Algorithm 1 (lines 5-18) assumes that every external dependency has already been analyzed and stored when it is needed, and the updating step rewrites dependency semantics after each sub-code; however, no experiment measures whether these stored descriptions preserve or corrupt state changes. RQ2 (Fig. 7) shows that semantic quality declines with code depth, which is consistent with storage errors accumulating. A concrete check, such as comparing stored dependency descriptions against ground-truth variable semantics on a small set of programs, would substantially reduce this correctness risk.","section":"Section III.B, Algorithm 1, Section V.B"}],"minor_comments":[{"comment":"The Introduction says the authors 'selected 200 problems' from CodeNet, while Section IV.B.1 says the Code-Pair dataset is built from 85 CP tasks. This discrepancy should be resolved.","section":"Section I vs. Section IV.B.1"},{"comment":"The row 'GPT-4 Turbo (CoT)' appears twice with identical values, once after 'GPT-4 Turbo (Zhuo w/R.)' and once after 'GPT-4 Turbo (few-shot)'; one of these rows is likely mislabeled.","section":"Table I"},{"comment":"The 'Expert' row reports values 0.97 and 0.91, but the text does not explain how these were computed or what quantity they represent; clarify this row or remove it.","section":"Table I"},{"comment":"The text says the Pearson correlations for CoT and few-shot prompts on HumanEval were 0.142 and 0.166, but Table I reports 0.142 and 0.186 for GPT-3.5 Turbo few-shot; please correct the mismatch.","section":"Section V.A.3"},{"comment":"The description of the Chain-of-Thought baseline says it 'Prompts the LLM to explain the reasoning or steps for a problem before coding,' which appears to describe code generation rather than code evaluation; clarify the actual evaluation-time CoT prompt.","section":"Section IV.D"},{"comment":"No code, prompts, or the Code-Pair dataset are released, which makes the reported numbers difficult to reproduce; releasing these artifacts would strengthen the paper.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is worth pursuing, but the evaluation is not yet adequate for publication. The most serious problems are the metric-label inconsistency, the impossible HumanEval sample size, and the apparent selection of design choices on the evaluation data. If the authors can supply a corrected, held-out evaluation with proper statistical reporting and release their artifacts, the paper could become a solid contribution. I would not recommend acceptance on the current version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is worth knowing, but the numbers are not. Recursive semantic comprehension is a genuine new combination: AST-based decomposition, a dependency dictionary to carry variable/function semantics across code fragments, and selective problem-statement injection. The motivation examples are clear, the method is plausible, and if the results held out it would be a practical advance for LLM-based code evaluation.\n\nThe empirical support is currently compromised. The stress test is correct: Section V-C selects PInitial over PFull/PLack after running on Code-Pair; the recursion threshold is fixed at 3 without sensitivity analysis; and the prompts and few-shot examples are 'designed' with no separate development set. So the headline correlations are post-selection best cases. The gap over baselines (e.g., 0.853 vs 0.746 for GPT-4 Turbo on Code-Pair) is smaller than the swing caused by changing the problem-statement policy (0.132–0.18), so we cannot tell how much of the improvement is real.\n\nThere are also reporting errors that block interpretation. The Introduction calls the numbers Spearman; Table I labels them Pearson. The HumanEval subset is said to be 500 cases, but HumanEval has 164 problems. A row for GPT-4 Turbo CoT is duplicated with different values. No significance tests or confidence intervals are reported. No code, data, or prompts are released.\n\nWhat is good: the method is coherent, the dependency-storage idea is thoughtful, and the Code-Pair dataset could be useful even if the ground-truth details are incomplete. RQ2 and RQ4 give some qualitative support that recursive decomposition helps with complex code.\n\nI would send this to peer review, because the idea deserves serious scrutiny and the empirical flaws are addressable. But I would require a held-out evaluation, release of artifacts, corrected tables, and error analysis before accepting any performance claim. The paper is mainly for researchers building LLM-based evaluation pipelines; they should treat the reported correlations as preliminary, not as a benchmark.","headline":"Interesting method, compromised evaluation.","tokens_in":15878,"tokens_out":3107,"would_cite":false,"duration_ms":27992,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that recursively decomposing code into dependency-aware sub-code chunks lets an LLM evaluate generated code quality with correlations up to 0.853 against human experts and 0.753 against code execution, outperforming…","keywords":["code quality evaluation","large language models","recursive semantic comprehension","semantic dependency decoupling","LLM as evaluator","code generation evaluation","HumanEval","Code-Pair"],"falsifier":"Run HuCoSC on a pair of programs that compute the same function but with different control flow (for example, an iterative loop versus recursion) and identical variable names; if the score differs by more than one point on the 0–4 scale, the semantic comparison is being influenced by surface structure rather than captured semantics. A more direct check: delete a state-changing line inside a deeply nested sub-code and see whether the dependency-storage description for the affected variable changes; if the description is unchanged, the storage has failed to track the state update.","tokens_in":14848,"feed_emoji":"🧠","tokens_out":6952,"duration_ms":55758,"temperature":0.7,"pith_summary":"The paper claims that code quality can be evaluated automatically by having a large language model recursively decompose code into small sub-code chunks, store and update the semantics of variables and functions in a dependency dictionary, and then compare the reconstructed semantics of the generated code against a reference code. On a new 428-pair dataset called Code-Pair, this method (HuCoSC) achieves Pearson correlations of 0.769 with GPT-3.5 Turbo and 0.853 with GPT-4 Turbo against human expert scores, and 0.594 and 0.753 against code execution on HumanEval, surpassing all tested token-matching and direct-prompt baselines. If the result holds, it means LLM-based evaluators can be made much more reliable for judging generated code without large test suites, and the recursive decomposition also produces explanations of the score.","feed_headline":"Recursive code reading lifts LLM evaluator agreement to 0.85","feed_subtitle":"Dependency-aware code chunks let GPT-4 Turbo judge code closer to human experts than token matching or direct prompts.","key_machinery":"The load-bearing mechanism is a recursive code-semantic-comprehension pipeline built from three parts: (1) Code Recursive Decomposition, which splits the AST into sub-codes under predefined node types (For, While, Assign, If, ClassDef, FunctionDef, Switch, Call) and recurses until sub-code nesting depth falls below a threshold; (2) Semantic Dependency Decoupling Storage, a dictionary mapping dependency names to textual semantic descriptions that is updated after each sub-code is analyzed, so each sub-code can be understood independently with the needed context; and (3) Code Semantic Summarization and Comparison, where the sub-code semantics are merged into a whole-code semantic and then compared with the reference code's semantic to produce a score and explanation.","core_discovery":"The central discovery is that recursively decomposing code into manageable sub-code units and maintaining a Semantic Dependency Decoupling Storage—a dictionary that stores textual semantic descriptions of each external dependency and updates those descriptions as analysis proceeds—lets an LLM capture code semantics accurately enough to score generated code against a reference. The semantic comparison unit then identifies differences and produces a score plus explanation, with the problem statement introduced only when input statements are analyzed to avoid hallucination. The paper reports that this pipeline outperforms all baselines and that the advantage grows with code complexity: at a code depth of 4, the simplified single-pass variant loses 0.7 points (with GPT-3.5 Turbo) or 0.19 points (with GPT-4 Turbo) in expert-rated semantic quality compared with the recursive version.","pith_inferences":["Editorial inference: The dependency-storage design suggests the method's ceiling is the fidelity of the stored semantic descriptions; if a variable's state change is missed, the final score inherits that error, so tracking mutable state more explicitly (e.g., by simulating updates) is a natural next step.","Editorial inference: Because the method already works without a problem statement, the same recursive comprehension and comparison could be applied to tasks such as code translation, code clone detection, or unit-test generation, where a reference semantic is available but no natural-language problem description is.","Editorial inference: The reported drop in correlation with code depth implies that for very long or deeply nested programs, further gains would require either a lower decomposition threshold or a second pass that verifies consistency between sub-code summaries.","Editorial inference: A direct test of semantic fidelity would be to feed HuCoSC a pair of programs that are semantically identical but written with different control flow (e.g., a loop versus recursion); high scores would confirm the storage captures meaning rather than surface structure."],"forward_implications":["On Code-Pair, HuCoSC reaches Spearman/Pearson correlations of 0.769/0.853 with human experts using GPT-3.5 Turbo/GPT-4 Turbo, higher than every baseline tested.","On HumanEval, it reaches 0.594/0.753 Pearson correlation with functional correctness, again above all baselines, showing the semantic description carries execution-relevant information.","Removing the problem statement entirely drops Code-Pair correlation by 0.03 (GPT-3.5) and 0.12 (GPT-4), so the method still works without problem statements, for tasks like code translation.","Recursive decomposition mitigates LLM hallucination on complex code: expert-rated semantic quality declines more slowly with code depth than in the simplified single-pass variant.","Adding problem statements at every comprehension step lowers correlation and inflates scores, so selective problem-statement injection is a design choice that matters."],"supporting_citations":[{"why":"Supplies the expert scoring criteria (0–4 scale) used to label the Code-Pair dataset and to motivate why token matches overrate semantically wrong code.","marker":"[9]"},{"why":"Baseline CodeBertScore representing pretrained-model semantic matching that HuCoSC must outperform.","marker":"[14]"},{"why":"Baseline LLM-based code evaluation framework (with and without reference code) that HuCoSC is compared against.","marker":"[17]"},{"why":"Source of competitive-programming problems and reference codes for the new Code-Pair dataset.","marker":"[23]"},{"why":"HumanEval dataset used as the validation set for correlation with code execution.","marker":"[25]"},{"why":"Provides the AI-chain idea of decomposing a complex task into multiple LLM interactions, which the recursive comprehension unit adapts.","marker":"[26]"},{"why":"Chain-of-thought prompting baseline used in the comparison, and the conceptual basis for the step-by-step semantic analysis.","marker":"[36]"}],"fun_headline_variants":["LLM judges code like humans via recursive semantic splits","Recursive code comprehension boosts LLM evaluator human correlation","Semantic decomposition makes LLM code scoring truly human-like","Recursive semantic analysis narrows LLM code evaluator gap with humans"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline assumes the textual semantic descriptions stored for each variable, function, and class preserve the actual state changes of the running code; if a stored description loses or distorts a state change, every downstream semantic comparison and score inherits that error.","fun_headline_variants_meta":{"raw":{"variants":["LLM judges code like humans via recursive semantic splits","Recursive code comprehension boosts LLM evaluator human correlation","Semantic decomposition makes LLM code scoring truly human-like","Recursive semantic analysis narrows LLM code evaluator gap with humans"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000542,"raw_usage":{"total_tokens":2609,"prompt_tokens":970,"completion_tokens":1639,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":1570}},"tokens_in":586,"tokens_out":1639,"duration_ms":11506,"temperature":1.0,"reasoning_tokens":1570,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:30:53.367472+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run HuCoSC on a pair of programs that compute the same function but with different control flow (for example, an iterative loop versus recursion) and identical variable names; if the score differs by more than one point on the 0–4 scale, the semantic comparison is being influenced by surface structure rather than captured semantics. A more direct check: delete a state-changing line inside a deeply nested sub-code and see whether the dependency-storage description for the affected variable changes; if the description is unchanged, the storage has failed to track the state update.","supporting_citations":[{"cited_title":"Out of the bleu: how should we assess quality of the code generation models?","cited_arxiv_id":null,"evidence_quote":"Supplies the expert scoring criteria (0–4 scale) used to label the Code-Pair dataset and to motivate why token matches overrate semantically wrong code."}],"review_version":1}