{"id":"dea97452-99cc-4ea5-ac93-80679fe11e80","arxiv_id":"2411.16927","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A prompt-engineered LLM pipeline can generate production code assertions with up to 83.5% compile accuracy and 0.526 ROUGE-L similarity to developer-written assertions.","lead":"Assertify uses large language models to automatically write assertion checks inside production Java code. It reports that the generated checks are often structurally similar to those written by human developers, though semantic accuracy is lower.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Same-repository few-shot examples leak project style into prompt, so ROUGE-L 0.526 may measure in-corpus mimicry rather than generalizable resemblance; held-out evaluation needed.","rationale":"The reader identified that few-shot examples and evaluation come from the same 22 repositories, and I agree this is a real validity threat. My analysis sharpens the mechanism: the Similar Method Extractor explicitly retrieves the most similar methods from the same pool, making same-project leakage likely for style features such as variable naming and assertion phrasing. This concern directly targets ROUGE-L, the primary evidence for structural similarity, rather than the more peripheral limitations of missing runtime validation or baselines. However, I do not think the concern invalidates the paper entirely; it justifies a conditional acceptance with the requirement of a repository-level held-out evaluation. The paper does provide a new dataset and a reproducible pipeline, which are useful contributions. The reader's verdict of CONDITIONAL remains appropriate, so I recommend no change to the verdict.","tokens_in":16238,"tokens_out":2767,"duration_ms":28563,"concrete_test":"Perform a repository-level held-out evaluation: split the 22 repositories into five folds, select few-shot examples only from the training folds, and evaluate on the disjoint test folds using prompt E with GPT-4o. Recompute SNE, SME, and average ROUGE-L. If the ROUGE-L drops materially (e.g., below 0.45) or the few-shot advantage over prompt D disappears, the reported 0.526 is project-specific rather than generalizable. Report the per-fold variance as well, since the current paper reports no error bars.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of structural similarity rests on ROUGE-L scores comparing generated assertions to developer-written assertions. The evaluation design, described in Section 6.1, partitions 2,810 methods from the same 22 repositories into an evaluation set (983 methods) and a few-shot learning set (1,827 methods). The Similar Method Extractor (Section 5) then selects the top three most similar methods from the FSL set by cosine similarity and includes them in prompt E. Because the FSL set is drawn from the same repositories as the evaluation set, the most similar methods can come from the same project as the candidate method. These in-project examples can share local naming conventions, project-specific idioms, and assertion style, allowing the LLM to mimic the house style rather than reason about semantics. Consequently, the reported ROUGE-L improvement from few-shot learning (e.g., +0.09 for GPT-4, +0.07 for GPT-3 and GPT-4o) and the headline similarity of 0.526 may be inflated by style leakage. The strongest claim, 'producing assertions that resemble the original assertions,' depends on this metric, so it is the load-bearing point. The absence of repository-level held-out evaluation makes it impossible to distinguish true generalization from project-specific pattern copying.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Assertify, a pipeline that uses large language models (GPT-3, GPT-4, GPT-4o) with context-rich prompts and few-shot learning to generate production-code assertions in Java. The approach extracts method metadata, generates code summaries, includes input/output descriptions and summaries of invoked methods, and retrieves the top three similar methods as few-shot examples. The evaluation is performed on 983 methods from 22 GitHub repositories and measures syntactic parse success, static semantic success via repository compilation, and ROUGE-L similarity to developer-written assertions. The authors report syntactic accuracy up to 97.4%, static semantic accuracy up to 83.5%, and an average ROUGE-L score of 0.526, concluding that few-shot learning improves structural similarity to original assertions.","tokens_in":16523,"tokens_out":4842,"duration_ms":44949,"significance":"If the reported results are robust, Assertify would address a genuine gap: most prior work generates assertions for unit tests, whereas production assertions serve documentation and runtime-checking purposes. The paper contributes a reusable dataset of 2,810 methods with developer-written assertions, a modular end-to-end tool, and prompt-ablation experiments across three GPT models, with artifacts released for reproducibility. The syntactic and static-semantic accuracy results, even with the study's limitations, provide a useful data point for applying LLMs to production-code instrumentation. However, the central RQ3 claim of structural resemblance to developer assertions rests on a single metric and an evaluation design that may leak project-specific style into the few-shot prompts; the manuscript needs stronger validation before the headline similarity claim is accepted.","major_comments":[{"comment":"The few-shot learning dataset and the evaluation set are both drawn from the same 22 repositories. Because the Similar Method Extractor selects the top three cosine-similar methods from the FSL set without excluding examples from the candidate's own repository, the model can be prompted with methods that share project-specific naming conventions, idioms, and assertion style. Since RQ3 is answered exclusively with ROUGE-L, the reported 0.526 score may reflect in-corpus mimicry rather than generalizable resemblance to developer-written assertions. The paper should evaluate on a repository-held-out split, or at least report the fraction of selected few-shot examples coming from the same repository as the candidate and analyze ROUGE-L conditioned on that factor.","section":"§6.1, §5 (Similar Method Extractor)"},{"comment":"The results section is internally inconsistent about which model achieves the best ROUGE-L score. The text first states that scores range \"from 0.374 for GPT-3 up to 0.526 for GPT-4,\" then says Assertify \"performs best (overall) when using the GPT-4o model,\" and finally gives \"the best ROUGE-L similarity of 0.52 and 0.51 with the GPT-4 and GPT-4o models, respectively.\" The abstract reports 0.526 without model attribution. These statements cannot all be correct, and the discrepancy directly affects the paper's central claim. Please report the exact per-model, per-prompt ROUGE-L values and reconcile the abstract with Section 7.","section":"§7 (RQ3)"},{"comment":"The static semantic evaluation only checks whether the repository compiles after inserting the generated assertions; it does not execute the code or validate that the asserted boolean conditions actually hold at runtime. The authors acknowledge this limitation, but the phrasing of RQ2 and several results paragraphs (e.g., \"assertions free of syntactic and static semantic errors\") may lead readers to infer semantic correctness. Because an assertion that compiles but is always false is worse than useless in production, the claims should be strictly limited to \"syntactically valid and compilable\" unless a dynamic evaluation is added, even on a subset of the dataset.","section":"§6.2 (SME), §7 (RQ2)"},{"comment":"All reported figures are point estimates from a single inference run per configuration, with no error bars, confidence intervals, or repeated runs. Given that LLM APIs are stochastic, statements such as \"few-shot learning increases the ROUGE-L score by 0.09 with GPT-4\" are not statistically supported. The paper should either run each configuration multiple times and report variance, or explicitly frame the results as a single-run exploratory observation rather than a comparative evaluation.","section":"§7 (all RQs)"}],"minor_comments":[{"comment":"The phrase \"as it receives more information about the model\" appears to be a typo; the intended meaning is likely \"more information about the method\" or \"about the code.\"","section":"§7 (RQ3)"},{"comment":"References [7] and [8] are the same paper (TOGA), and references [36] and [37] are also the same paper; please deduplicate.","section":"References"},{"comment":"The text uses \"SourceGraph\" while the reference uses \"Sourcegraph\"; unify the spelling.","section":"§2, Reference [6]"},{"comment":"The SNE rate is defined, but the results report \"syntactic accuracy\" and \"static semantic accuracy\" without explicitly stating that accuracy equals one minus the corresponding error rate; stating this relationship at the metric definitions would improve readability.","section":"§6.2"},{"comment":"The caption and axis labels of Figure 7 are not described in the text; please ensure the figure clearly distinguishes per-model trend lines and indicates whether the reported values are ROUGE-L F1 scores.","section":"§7 (RQ3), Figure 7"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid early-stage systems paper, but the evaluation is currently not rigorous enough for the strength of the claims. The same-repository few-shot leakage and the single-run, no-baseline design are the main barriers. I would encourage the editor to treat the paper as a candidate for major revision rather than rejection because the underlying idea and dataset are useful and the fixes (held-out evaluation, repeated runs, dynamic checks) are feasible within the scope of a revised manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: worth a look if you work on LLM-based code generation or program assertions. It targets a genuinely under-served task—production assertions, not unit test assertions—and ships a new corpus of 2,810 methods from 22 mature Java repos, with 983 eval methods. That alone is a contribution. The pipeline is straightforward prompt engineering with five ablated prompt variants and three GPT models, and the paper honestly reports that ROUGE-L measures structural similarity, not correctness.\n\nThe strengths: clear problem definition, sensible context extraction (method signature, code summary, I/O descriptions, summaries of invoked methods), ablation shows few-shot helps, and the artifact is public. The syntactic (87.5–97.4%) and static semantic (62.3–83.5%) accuracies are measured by parsing and compiling the modified repo, so those numbers are objective. The classification of common errors (undefined symbol, unreachable statement, bad operand types) is useful.\n\nWhere it's soft: First, no runtime or dynamic validation. An assertion can parse, compile, and lexically resemble the developer's assertion while being false or vacuous at runtime. The paper acknowledges this in Section 8, but the abstract's \"resemble the original assertions\" is the ceiling of the claim. Second, the few-shot examples and evaluation set come from the same 22 repositories (Section 6.1). Since the Similar Method Extractor picks top-3 similar methods by cosine similarity, a candidate method can easily be paired with methods from its own project, leaking house style. So the +0.07–0.09 ROUGE-L gain from few-shot, and the headline 0.526, may be partly in-corpus mimicry rather than generalizable resemblance. A held-out-repo split would be needed to know. Third, there are no error bars, no baseline comparison (e.g., static analysis or fine-tuned CodeBERT), and the hand-chosen thresholds (0.5 similarity, 50 assertions, 500 stars) are not sensitivity-analyzed. These are fixable in revision.\n\nThe central argument—that LLMs can produce plausible, compilable production assertions given context—holds up. The claim that they are as good as developers' assertions does not, because the evaluation never checks semantics at runtime. The paper is a solid empirical starting point, not an overclaiming mess.\n\nWho it's for: anyone building assertion generators or evaluating LLM code tools; it also demos a reasonable evaluation harness for insertion-location errors. I'd send it to review with a request for held-out evaluation and, ideally, a dynamic check or human study. The dataset alone is worth citing.","headline":"A useful new dataset and task framing for LLM-based production assertion generation, but the evaluation stops at compilability and lexical similarity—no runtime validity—and same-repo few-shot examples may inflate the similarity numbers.","tokens_in":16991,"tokens_out":2274,"would_cite":true,"duration_ms":21711,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Assertify claims that LLMs, guided by context-rich prompts with few-shot examples, can generate production assertions for Java code that are syntactically accurate up to 97.4%, statically meaningful up to 83.5%, and structurally similar…","keywords":["production assertions","large language models","prompt engineering","few-shot learning","Java","static semantics","ROUGE-L","assertion generation"],"falsifier":"Run Assertify on methods from Java repositories that are not in the original 22-repository corpus, using the same prompt template and few-shot examples drawn only from the original corpus, and compare the average ROUGE-L to the reported 0.526; a drop toward the no-few-shot baseline would show the similarity is corpus-specific.","tokens_in":16080,"feed_emoji":"🛠️","tokens_out":4586,"duration_ms":41471,"temperature":0.7,"pith_summary":"This paper claims that large language models, given carefully assembled context, can generate production assertions directly inside Java source code, a task distinct from generating unit-test assertions. It introduces Assertify, a pipeline that extracts method context, builds prompts with code summaries, input/output details, and few-shot examples, then inserts model-generated assertions at predicted line numbers. On 983 methods from 22 mature Java repositories, the best configuration achieves syntactic accuracy up to 97.4%, static semantic accuracy up to 83.5%, and ROUGE-L structural similarity of 0.526 against developer-written assertions. If these numbers hold beyond the corpus, automation could relieve developers of a manual, error-prone task and improve code reliability.","feed_headline":"LLM tool writes production assertions close to developer style","feed_subtitle":"Context-rich prompts with few-shot examples push syntactic accuracy to 97.4% on Java production code.","key_machinery":"The carrying mechanism is a context-enrichment pipeline rather than a new model or loss function. Assertify strips assertions and comments from a candidate method, then builds a prompt that includes the method name and signature, a generated code summary, input/output descriptions, summaries of internally invoked methods, and the top three similar methods selected by cosine similarity over method vectors for few-shot learning. The LLM returns assertion-plus-line-number pairs, which a postprocessor filters by checking that the line numbers fall within the method's braces; the repository is compiled after insertion to detect static semantic errors, and ROUGE-L measures structural similarity.","core_discovery":"On its own terms, the paper establishes that prompt engineering with few-shot learning is what makes LLM-generated production assertions resemble developer-written ones. It shows that each added context layer—code summary, input/output description, invoked-method summaries, and similar methods—raises structural similarity and reduces syntactic and static-semantic errors, with the fullest prompt giving the best results. The paper also reports that generated assertions are typically equal to or stronger than the original assertions, and that few-shot learning specifically increases the frequency of exact assertion-set matches.","pith_inferences":["The reported ROUGE-L similarity likely overstates generalization because the few-shot examples and the evaluation methods come from the same 22 repositories; a held-out repository test would be a stricter measure.","The static semantic check verifies only that the code compiles, not that the assertion is true at runtime, so the 83.5% figure is an upper bound on the share of assertions that are dynamically meaningful.","The same pipeline should transfer to other statically typed languages, but few-shot examples would need to be drawn from the target language; that transfer is a testable extension the paper leaves open.","If few-shot examples were taken from a separate corpus, the measured gain from few-shot learning would cleanly separate genuine generalization from in-corpus style mimicry."],"forward_implications":["Production assertions can be generated automatically in Java codebases without running tests or needing a test prefix, which existing unit-test assertion generators require.","The best configuration uses the richest prompt plus few-shot examples; removing few-shot examples costs the most accuracy in the ablation study.","Generated assertions tend to be equal to or stronger than developer-written assertions, not merely weaker approximations.","Model choice trades cost and latency against accuracy, with the strongest model giving the top scores while cheaper models remain viable for developers with tighter budgets."],"supporting_citations":[{"why":"Supplies the ROUGE-L metric used to measure structural similarity to developer-written assertions.","marker":"[19]"},{"why":"Supplies the few-shot learning concept that the prompt engineering technique relies on.","marker":"[26]"},{"why":"Provides additional background on few-shot learning that motivates selecting similar methods for prompts.","marker":"[35]"},{"why":"Provides the repository search and extraction source used to collect the 22-repository corpus.","marker":"[6]"},{"why":"Represents the unit-test assertion generation approach the paper argues does not transfer to production assertions.","marker":"[7]"},{"why":"Represents the dynamic invariant detection baseline that the paper contrasts with its execution-free generation approach.","marker":"[11]"},{"why":"Supports the reliability of LLM-generated code summaries, which the prompt construction uses as context.","marker":"[21]"}],"fun_headline_variants":["LLM few-shot prompts write production assertions like developers","Few-shot LLM prompting yields developer-style production assertions","Assertify: LLMs generate production assertions matching developer style","Context-rich LLM prompts mimic developer assertions in production","LLM few-shot prompt hits 97.4% syntactic accuracy on assertions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The few-shot examples and the evaluation methods are drawn from the same 22 repositories, so the high structural similarity may reflect the model imitating in-corpus assertion style rather than generalizing to unseen projects.","fun_headline_variants_meta":{"raw":{"variants":["LLM few-shot prompts write production assertions like developers","Few-shot LLM prompting yields developer-style production assertions","Assertify: LLMs generate production assertions matching developer style","Context-rich LLM prompts mimic developer assertions in production","LLM few-shot prompt hits 97.4% syntactic accuracy on assertions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000468,"raw_usage":{"total_tokens":2282,"prompt_tokens":842,"completion_tokens":1440,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":458,"completion_tokens_details":{"reasoning_tokens":1358}},"tokens_in":458,"tokens_out":1440,"duration_ms":10049,"temperature":1.0,"reasoning_tokens":1358,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:43:32.710773+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Assertify on methods from Java repositories that are not in the original 22-repository corpus, using the same prompt template and few-shot examples drawn only from the original corpus, and compare the average ROUGE-L to the reported 0.526; a drop toward the no-few-shot baseline would show the similarity is corpus-specific.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the repository search and extraction source used to collect the 22-repository corpus."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents the dynamic invariant detection baseline that the paper contrasts with its execution-free generation approach."}],"review_version":1}