{"id":"3b30ec58-cc20-4c5a-af00-de6b52521a4d","arxiv_id":"2502.07835","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"SBC is a hybrid metric that reverse-generates requirements from LLM-written code and scores their semantic, lexical, and completeness match to the original requirement, but its validity is not established.","lead":"This paper proposes a way to score AI-generated code by asking an LLM to read the code back as a plain-language requirement, then comparing that description with the original requirement using similarity, BLEU, and completeness. The authors say this could help developers, especially junior ones, see missing features and hallucinations without reading all the code.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SBC's validity as a code-alignment metric rests on untested reverse-generation fidelity; same-LLM self-consistency and absence of any ground-truth correlation leave Eq. (1)'s developer-insight claims unsupported.","rationale":"The reader's weakest-assumption diagnosis aligns with the most load-bearing weakness I found: SBC's interpretation as a measure of requirement-code alignment requires that reverse generation faithfully recovers what the code actually does, and that similarity to the original requirement tracks correctness. The paper establishes neither. The same-LLM design creates a concrete self-consistency threat: the model's strong prior over the original requirement can lead it to 'reconstruct' that requirement even from defective code, inflating semantic similarity. The completeness component is also too shallow to catch semantically paraphrased hallucinations, since it only compares extracted keyword sets. Section 5 itself concedes that human feedback and correlation analyses are future work, which is a direct admission that the paper's central evaluative claim is currently unvalidated. Thresholds like 'interpretable above 0.55' in Sec. 4 are derived from the very outputs being scored, with no external anchor. An injected-bug experiment would provide a decisive test: if SBC cannot reliably distinguish correct code from code with a single removed requirement, then it cannot support the paper's promise of actionable developer insights. The reader's verdict of REJECT is therefore appropriate; my analysis does not move it, but it sharpens the specific technical condition that must be met before the approach could be accepted.","tokens_in":5965,"tokens_out":3061,"duration_ms":29027,"concrete_test":"Select 20-30 requirements from the dataset. For each, generate code C with one of the four models, then create a deliberately defective variant C_bug by removing a single required behavior (e.g., delete input validation, a loop boundary, or an API call) that is expressed in R. Run the full reverse-generation pipeline (Sec. 3.2) on both C and C_bug with temperature 0, and compare the SBC scores. If SBC does not show a consistent, statistically significant drop for C_bug across the majority of cases, or if the score distributions overlap substantially, then Eq. (1) does not track requirement-code alignment, and the central claim fails regardless of internal consistency.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the SBC score (Eq. 1) reveals missing functionality and hallucinations depends on two implicit assumptions: (i) the reverse-generated requirement R' is a faithful natural-language description of the generated code C, and (ii) similarity between the original requirement R and R' is monotonically related to C's correctness/completeness. Neither assumption is tested. Because the same LLM produces C from R and then R' from C, R' can be heavily influenced by the model's prior over R; high R-R' similarity may reflect regeneration consistency rather than code quality. The paper does not inspect R' for fidelity to C, does not include buggy code as a negative control, and reports no human ratings or correlation with pass@k/CodeBLEU; Sec. 3.5 and Sec. 5 explicitly defer correlation and human validation to future work. The 'missing' and 'extra' elements in Eqs. (2)-(8) are keyword-set differences, so a hallucinated feature phrased with synonyms can escape detection. Consequently, the threshold claims in Sec. 4 (interpretable above 0.55, semantically close above 0.65) are calibrated on the same scores they are meant to validate, with no external anchor. If R' fails to track C, then SBC measures paraphrastic similarity between two texts produced by the same model, and the paper's developer-facing conclusions do not follow.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a reverse-generation technique and a composite SBC score for evaluating LLM-generated code without a reference implementation. Given an original requirement R, an LLM generates code C, and the same LLM is then prompted to reverse-generate a natural-language requirement R' from C. The SBC score in Eq. (1) is 0.7 times the semantic similarity between R and R', 0.1 times BLEU, and 0.2 times a keyword-based completeness score. The authors apply this procedure to 90 curated requirements across UI, data, and business-logic layers using four open LLMs, running three iterations per model, and claim that missing and extra keyword sets can reveal omitted functionality and hallucinations. The paper also proposes interpretability thresholds of 0.55 and 0.65 for SBC scores in Section 4.2.","tokens_in":6208,"tokens_out":5235,"duration_ms":50854,"significance":"If the SBC score could be shown to track human judgments of requirement-code alignment, the approach would be a valuable reference-free diagnostic tool, particularly for junior developers. The strengths of the manuscript are its clearly specified formula, the use of open models, the release of code and data, and the explicit temperature-zero setting for reproducibility. However, the central validity claim is currently untested: there are no human labels, no correlation with execution-based metrics such as pass@k, no negative-control experiments, and no comparison with existing semantic metrics. The paper's significance is therefore prospective rather than established.","major_comments":[{"comment":"The paper explicitly defers human validation and correlation analysis to future work, but these are not optional additions; they are the evidence needed to support the claim that SBC measures requirement-code alignment. Section 4's statements that missing and extra elements provide 'valuable insight' and that SBC detects hallucinations are unsupported without such validation. Please add a study in which human raters judge the correctness and completeness of the generated code, and report correlations between SBC and human judgments. In addition, report SBC's correlation with at least one established metric (e.g., pass@k or CodeBLEU) on a dataset with known correct and incorrect code samples.","section":"Sec. 3.5 and Sec. 5"},{"comment":"Because the same LLM generates C from R and then reverse-generates R' from C, high semantic similarity between R and R' can reflect the model's prior over the original requirement rather than the fidelity of C. This self-consistency threat is not controlled. A minimal control would be to introduce deliberately defective code (e.g., removing a required function or inserting a hallucinated API call) and show that SBC drops accordingly; another would be to have a different LLM perform the reverse generation, or to have human annotators check whether R' is actually a faithful description of C. Without such controls, Eq. (1) is not shown to measure correctness or completeness, only paraphrastic consistency between two LLM outputs.","section":"Sec. 3.4, Eq. (1)"},{"comment":"The completeness component is computed as the difference between two extracted keyword sets. This is purely lexical: a hallucinated feature that is phrased with synonyms, or a missing requirement that is paraphrased in R', will not appear as an extra or missing keyword. The claim in Section 4 that missing and extra keyword sets provide a strong indicator of hallucinations is therefore too strong. Please evaluate the precision and recall of the missing/extra detection against human-annotated requirement coverage, and consider a semantic matching step or normalization (e.g., stemming, synonym expansion).","section":"Sec. 3.4, Eqs. (2)-(8)"},{"comment":"The interpretability thresholds of 0.55 and 0.65 are presented as if they have intrinsic meaning, but they are derived from visual inspection of the same SBC outputs that they are meant to interpret. This is circular. The authors should specify the exact procedure by which these thresholds were selected, and validate them on held-out data or against human judgments. Without an external anchor, a claim such as 'easy to interpret above 0.55' is not a reproducible finding.","section":"Sec. 4.2"}],"minor_comments":[{"comment":"The phrase 'This paper introduces a novel scoring mechanism' appears effectively twice, once in the abstract and once in the introduction; the duplication should be removed.","section":"Abstract and Sec. 1"},{"comment":"The reverse-generation prompt is not quoted or described in detail. For reproducibility, please include the exact prompt template used to obtain R' from C, perhaps in an appendix.","section":"Sec. 3.2"},{"comment":"The dataset is said to cover 90 requirements across multiple application layers, but the per-layer distribution is not reported. A small table or figure would help assess coverage and potential bias.","section":"Sec. 3.1"},{"comment":"Only three iterations per model are reported, with no variance or confidence intervals. Given that the paper compares models, please report per-question distributions or error bars, at least for the final SBC scores.","section":"Sec. 4.1"},{"comment":"The use of Google Sheets for visualization is not a reproducible analysis workflow; please provide the scripts or notebooks that generate the figures.","section":"Sec. 3.6"},{"comment":"There is a typo, 'entrprises', which should be 'enterprises'.","section":"Sec. 4.3"},{"comment":"Several figures are referenced in the text but the included manuscript does not show them. Please ensure that all figures are present, with axis labels and legends.","section":"Figures"},{"comment":"The reference formatting is inconsistent (e.g., [14] and [15] have different styles). Please unify the bibliography style.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a clear proposal for a reference-free evaluation metric, but its central claim is currently unsupported by external evidence. I chose major revision rather than rejection because the required validation steps are well-defined and could be supplied in a revision: human-rated correctness/completeness data, negative controls, and correlation with existing metrics. If the authors cannot provide such evidence, the claims in Sections 4 and 5 should be substantially scaled back, and the paper would then be a much weaker contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take on arXiv:2502.07835. The paper is a clearly written proposal for a reverse-generation SBC score, and the formula itself is new relative to the cited work. But the central claim—that SBC measures requirement-code alignment and gives developers actionable insight—is not supported by evidence in the manuscript. The main threat is the same-LLM self-consistency loop: the same model generates C from R and then R' from C. If the model has a strong prior over R, R' can come back close to R even when C is wrong. The paper does not check reverse-generation fidelity, does not include buggy-code negative controls, reports no human ratings, and does not correlate SBC with pass@k or CodeBLEU. Sections 3.5 and 5 explicitly defer those checks to future work. That is honest, but it means Eqs. (1)–(8) remain a candidate metric, not a validated one.\n\nCredit where due: the contribution is concrete: 90 curated requirements across UI/data/business layers, four open-weight models with temperature 0, the SBC formula is explicit, and the missing/extra keyword output is a practical artifact. The authors also avoid the trap of using closed hosted models, which supports reproducibility in principle.\n\nSoft spots in proportion: the thresholds 0.55/0.65 are calibrated on the same SBC scores they are meant to explain, so they carry no external anchor. The completeness term is keyword-set difference; synonyms for hallucinated features evade it. Weights 0.7/0.1/0.2 are asserted, not derived. The GitHub link is a placeholder, so the code and data are not currently assessable. And the Related Work claim of \"no prior work\" on reverse generation is too strong—metamorphic prompt testing is close in spirit, and the paper itself describes it.\n\nNet: if you want a metric to compare with human judgment or to debug in a study, SBC is a reasonable starting point. As a standalone result claiming developer benefits, it is too early. I'd cite it as related work only if I needed a recent example of LLM-as-evaluator for code; otherwise I'd wait for validation. For a venue with real referees, I would send it out with instructions to focus on validation, not desk-reject it: the idea is testable and clearly specified. Good reading-group material on evaluation validity.","headline":"A concrete but unvalidated proposal: same-LLM self-consistency and deferred human/correlation checks leave the SBC score's central validity claim unsupported.","tokens_in":6779,"tokens_out":2946,"would_cite":false,"duration_ms":28681,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reverse-generated requirements reveal missing code and hallucinations","keywords":["LLM code generation","reverse generation","SBC score","semantic similarity","BLEU","completeness","code evaluation","AI-assisted software development"],"falsifier":"Give a set of requirements with known buggy code (for example, code that omits an error branch or a required endpoint), run the reverse-generation pipeline, and check whether SBC systematically ranks those flawed outputs below correct outputs and whether the missing-keyword list names the actual omission; a single counterexample where wrong code restates the requirement perfectly at a score above 0.9 would refute the proxy.","tokens_in":5722,"feed_emoji":"🧩","tokens_out":5659,"duration_ms":44886,"temperature":0.7,"pith_summary":"The paper proposes that an LLM can audit its own generated code by a reverse step: given the code, the LLM restates what it thinks the code does, and this restatement is compared with the original natural-language requirement. The comparison yields a weighted SBC score (0.7 semantic similarity, 0.1 BLEU, 0.2 completeness) together with lists of missing and extra keywords. The claim is that this score quantifies requirement-code alignment without any reference implementation, and that the missing and extra lists give developers direct, interpretable signals of omitted functionality and hallucinations. The authors test the idea on a new set of 90 multi-layer requirements with four open LLMs and report that all models perform similarly, with reconstructed requirements readable at SBC scores above 0.55. If true, this would give junior developers a lightweight validation tool that does not require inspecting code or writing tests.","feed_headline":"Reverse-generated requirements reveal missing code and hallucinations","feed_subtitle":"It re-asks the LLM what its code does, then weighs meaning, BLEU, and completeness against the original requirement.","key_machinery":"The reverse-generation pipeline is the load-bearing mechanism: requirement to LLM-generated code, then a prompt asking the LLM to reconstruct the requirement from that code, then a comparison of original and reconstructed requirements. Equation (1) combines cosine semantic similarity from a sentence-embedding model with weight 0.7, BLEU n-gram overlap with weight 0.1, and a completeness score over extracted keywords with weight 0.2, as defined in Eqs. (2) through (8). The weights encode the belief that meaning-level alignment matters more than lexical overlap, and the missing and extra keyword lists are what turn the numeric score into developer-facing insights about what the code forgot or invented.","core_discovery":"The central claim is that reverse generation can turn a code-generation LLM into a feedback channel: instead of measuring whether the output matches a reference program, you measure whether the output, when translated back to natural language, still matches the input requirement. The SBC score formalizes this as $$f_s = 0.7 \\times s_{\\text{sem}} + 0.1 \\times \\text{BLEU} + 0.2 \\times \\text{comp}$$ where the semantic term is cosine similarity between encoded requirements, BLEU captures lexical overlap, and the completeness component extracts keywords from both requirement texts and penalizes both missing and extra elements, which the paper interprets as a hallucination signal. The experiments across 90 requirements, four open models, and three runs show closely aligned score trends, and the paper claims that the reconstructed requirements are easy to interpret when the SBC score exceeds 0.55.","pith_inferences":["The decisive test the paper leaves undone is whether SBC tracks human judgments of code correctness; a natural experiment is to have engineers rate the 90 code snippets and compare their ratings with SBC, missing-keyword count, and pass@k on hand-written tests.","If the proxy holds, reverse generation could also measure requirement drift across code edits, turning SBC into a regression signal during refactoring.","The same pipeline could compare two candidate implementations against one requirement by reconstructing requirements from each and measuring their distance to the original, providing a reference-free ranking alternative to CodeBLEU-style metrics.","Without validation, high SBC is just consistency: an LLM that confidently restates its own wrong code will score high, so the method's practical value depends entirely on the untested premise that restatement fidelity tracks requirement satisfaction."],"forward_implications":["Without any reference code, a developer or CI pipeline can obtain a numeric alignment score plus a human-readable restatement of what the generated code actually implements.","The missing-keyword list acts as an omission detector and the extra-keyword list as a hallucination detector, pointing at specific requirement elements rather than giving only a score.","Because the method is metric-only and uses open models, it can be embedded into AI coding assistant workflows so that every generated snippet is returned with its reverse requirement and SBC score.","Across the 90-requirement test set, the four open LLMs track each other closely, suggesting the reverse-generation signal depends more on the task than on the model choice.","The approach applies across UI, data-layer, and business-logic requirements, so it is not limited to a single programming language or application tier."],"supporting_citations":[{"why":"Supplies the BLEU component used in the SBC score for lexical overlap.","marker":"[5]"},{"why":"Establishes the LLM-as-evaluator baseline with human-alignment evidence that the paper extends with reverse generation.","marker":"[13]"},{"why":"Provides a recent LLM-based code evaluation metric whose reliance on instruction-based judgment the paper repositions toward requirement reconstruction.","marker":"[14]"},{"why":"Supplies a prior reference-free validation method (metamorphic prompt testing) and its detection rates, which the paper's approach implicitly competes with.","marker":"[15]"},{"why":"Defines pass@k, the execution-based evaluation the paper argues against because it requires manually written tests.","marker":"[4]"},{"why":"Provides evidence that token-based metrics such as BLEU correlate weakly with human judgment on code, motivating the SBC hybrid.","marker":"[1]"},{"why":"Represents neural code-similarity evaluation that still depends on high-quality reference solutions, which the paper positions SBC as avoiding.","marker":"[8]"},{"why":"Supplies the rationale for choosing open models over closed hosted ones for transparency and reproducibility.","marker":"[17]"}],"fun_headline_variants":["Reverse generation turns LLM code into requirement checks","SBC score catches hallucinations by re-asking the LLM","New metric: re-generate requirements from code to find gaps","How to make LLM code review itself via reverse generation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the similarity between the original requirement and the LLM's restatement of its own generated code indicates whether the code actually fulfills that requirement.","fun_headline_variants_meta":{"raw":{"variants":["Reverse generation turns LLM code into requirement checks","SBC score catches hallucinations by re-asking the LLM","New metric: re-generate requirements from code to find gaps","How to make LLM code review itself via reverse generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000183,"raw_usage":{"total_tokens":1337,"prompt_tokens":992,"completion_tokens":345,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":278}},"tokens_in":608,"tokens_out":345,"duration_ms":3716,"temperature":1.0,"reasoning_tokens":278,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T13:35:06.984947+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give a set of requirements with known buggy code (for example, code that omits an error branch or a required endpoint), run the reverse-generation pipeline, and check whether SBC systematically ranks those flawed outputs below correct outputs and whether the missing-keyword list names the actual omission; a single counterexample where wrong code restates the requirement perfectly at a score above 0.9 would refute the proxy.","supporting_citations":[{"cited_title":"Bleu: a method for automatic evaluation of machine translation,","cited_arxiv_id":null,"evidence_quote":"Supplies the BLEU component used in the SBC score for lexical overlap."},{"cited_title":"ICE-Score: Instructing Large Language Models to Evaluate Code,","cited_arxiv_id":null,"evidence_quote":"Provides a recent LLM-based code evaluation metric whose reliance on instruction-based judgment the paper repositions toward requirement reconstruction."},{"cited_title":"Out of the bleu: How should we assess quality of the code generation models?,","cited_arxiv_id":null,"evidence_quote":"Provides evidence that token-based metrics such as BLEU correlate weakly with human judgment on code, motivating the SBC hybrid."}],"review_version":1}