{"id":"bdb9ed83-eee2-44ba-b0ee-a37dfaf53edf","arxiv_id":"2608.04195","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SONAR evaluates code summaries without reference summaries by regenerating code from the summary, and shows that only correctness and abstraction strongly predict LLM performance on downstream software engineering tasks.","lead":"This paper introduces SONAR, a reference-free method that judges code summaries by using them to regenerate the code and checking whether the regeneration succeeds. It finds that correct and abstract summaries matter most for AI assistants on software tasks, while conciseness and fluency matter little to those AI consumers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SONAR's Correctness score may measure benchmark function-name leakage rather than summary quality; no control shows regeneration depends on summary content.","rationale":"The reader identified the differential fuzzer's test set T as the weakest assumption. I agree that Eq. 1 depends on an underspecified fuzzing oracle, but I see an upstream, more load-bearing assumption: the regeneration step G(s,c) must actually use the summary's content. The paper's pool of benchmark functions, combined with a context that includes the function name, creates a plausible leakage path: for highly descriptive names or widely memorized functions, G can produce x without relying on s. This would sever the link between the measured Correctness score and the summary's quality, which is exactly what RQ1 and RQ2 need. The fuzzer concern matters only after that link is established; if the regeneration signal is insensitive to s, no fuzzer refinement can fix the construct validity of Correctness. I still recommend the reader's CONDITIONAL verdict rather than a stronger rejection, because the paper includes independent supporting evidence (RQ1 agreement, RQ4 sensitivity) and the leakage hypothesis is testable. The concrete control experiment would settle whether the concern lands; until then, the central claim is plausible but not fully secured.","tokens_in":19376,"tokens_out":9069,"duration_ms":82026,"concrete_test":"Conduct a leakage control on a stratified sample of 100 functions from the evaluation pool. For each function, compute the Correctness score under (a) the original summary, (b) a summary of a different function from the same pool, and (c) a no-summary prompt containing only the signature and context c, using the same generator G. If the mean FuncSim for (b) and (c) is comparable to (a) (e.g., greater than 0.8 or within 0.1), the score is not driven by summary content. Additionally, repeat (a) with function names anonymized to identifiers like func_1234 while preserving parameter types and the code body; if scores drop substantially, the descriptive benchmark name is leaking behavioral information into Correctness. Report per-condition score distributions and paired significance tests.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that 'a summary is used to regenerate code' and that the round trip 'empirically grounds' summary evaluation rests on an untested assumption: that the code regenerated by G actually depends on the summary text s, over and above the minimal context c. In Section III-A, c explicitly includes the function name, return type, and parameter names. The evaluation pool (Section IV-A1) is drawn from HumanEval, MBPP, BigCodeBench, and The Vault — benchmarks whose functions are public and widely memorized, and whose HumanEval and MBPP names are highly descriptive (e.g., greatest_common_divisor). If G can reconstruct x from c alone, or from memorized benchmark solutions, then Eq. 1's FuncSim(x, G(s,c)) will be high regardless of s, so Correctness no longer measures summary quality. This would directly invalidate the RQ2 correlations: Correctness would rank summaries by how well they align with a canonical implementation or a descriptive name, not by how useful the summary is to a downstream LLM. The paper's threats section concedes that scores 'may be affected by ... the influence of their pretrained knowledge,' but no experiment quantifies this. RQ1's 96% agreement with human preferences is not decisive, because agreement is computed only on pairs where both authors unanimously preferred one summary; if leaked context dominates for many functions, the remaining non-leaky functions could drive the reported agreement. The reader's fuzzer concern is real but downstream; even a perfect functional-equivalence oracle cannot rescue a regeneration signal that does not respond to the summary.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes SONAR, a reference-free framework for evaluating LLM-generated source code summaries along four dimensions: Correctness, Abstraction, Conciseness, and Fluency. The framework follows a 'regenerate, then evaluate' principle: a summary, together with minimal code context (function name, return type, parameter names, imports), is given to a code generator, and the functional similarity between the original and regenerated code, computed via differential fuzzing, serves as the quality signal. Correctness is this functional similarity directly; Abstraction is the diversity of a pool of regenerated implementations; Conciseness is the complement of the largest behavior-preserving compression ratio; Fluency is inverse perplexity. The authors validate the three regeneration-based dimensions against human preference labels (90% overall agreement), correlate SONAR scores with LLM performance across four downstream software engineering tasks (retrieval, translation, optimization, test oracle generation) with six metrics and three task LLMs, evaluate 11 summarizer LLMs under three prompting strategies, and run a sensitivity analysis across scorer LLMs.","tokens_in":19632,"tokens_out":12913,"duration_ms":112584,"significance":"If the findings hold, this is a useful contribution to code summarization evaluation: it offers an execution-grounded, reference-free alternative to BLEU/ROUGE/BERTScore-style metrics, and it gives quantitative support to the increasingly common intuition that what makes a summary useful to an LLM consumer differs from what human developers value. The experimental breadth is a genuine strength: 11 summarizer LLMs, four downstream tasks, three task LLMs, six metrics, sensitivity analysis, and partial correlations. The differential-fuzzing oracle and the use of downstream task performance as an external test are the right kind of evidence for a reference-free metric, and the Abstraction-as-diversity construct is novel and interesting, though it rests on the a priori axiom that implementation diversity measures level of abstraction.","major_comments":[{"comment":"The paper's central mechanism, namely that regenerated code 'empirically grounds' summary evaluation, is not tested against the possibility that regeneration depends on the minimal context c rather than on the summary s. Because c includes the function name, return type, and parameter names (Section III-A), and because the evaluation pool is drawn from publicly available benchmarks (HumanEval, MBPP, BigCodeBench, The Vault) whose function names are often highly descriptive (e.g., greatest_common_divisor in Fig. 1), the generator G may reconstruct a canonical implementation from c alone or from memorized benchmark solutions, making FuncSim(x, G(s,c)) high regardless of s. The threats section (§V-A) concedes 'the influence of their pretrained knowledge' as a factor, but no experiment quantifies it. I ask for a control that computes FuncSim(x, G(c)) with the summary omitted and with a deliberately incorrect summary, reporting how often and by how much the summary changes the regenerated code, along with the distribution of Correctness scores; without this, the claim that Correctness measures summary quality rather than the function's name and benchmark notoriety is unsubstantiated.","section":"III-A, Eq. (1); IV-A"},{"comment":"The RQ2 correlations pool multiple summaries per source function (roughly 14–21 summaries per function across the approximately 100 sampled HumanEval/MBPP functions), so the reported Spearman correlations can be driven by between-function difficulty rather than by summary quality: functions that are inherently easier also tend to receive higher SONAR Correctness and Abstraction scores and higher downstream success rates. This confound directly affects the conclusion in §IV-D4 that 'the summary itself drives downstream performance.' I ask the authors to recompute the significant correlations in Table III within functions (e.g., function-stratified or with function and summarizer as covariates) and to report whether the signals survive; if they do not, the task-dependence claim would need to be restricted to between-function comparisons.","section":"IV-D3, Table III"},{"comment":"FuncSim in Eq. (1) is the cornerstone of both the Correctness and Conciseness scorers, but the test input set T is not characterized: the paper does not report |T|, how T is generated per function, what coverage is achieved, or any calibration of the fuzzer's ability to detect behavioral differences (for example, mutation-based false-negative rates). The fuzzer's validity is inherited from two self-cited papers [31], [32] rather than demonstrated here, and the stated fallback to reference-based code evaluation metrics for code that cannot be run in isolation introduces a second oracle whose share in the reported scores is unspecified. I ask that the authors report the distribution of |T| and of FuncSim across the function pool, validate the oracle on functions with known behavioral variants, and state explicitly how many summaries in each experiment (RQ1, RQ2, RQ4) were scored with the reference-based proxy instead of execution.","section":"III-A1, Eq. (1)"},{"comment":"The RQ1 validation protocol starts with 100 summary pairs per dimension, discards any pair on which the two annotators do not unanimously agree (retaining 84, 86, and 89 pairs for Correctness, Abstraction, and Conciseness, respectively), and reports agreement only on the retained pairs. This makes the reported 87–96% agreement rates conditional on the annotators having agreed, which inflates the rates relative to an unbiased estimate, and it makes disagreements and 'Not Sure' responses invisible. I ask the authors to report agreement on the full 100 pairs with disagreement and 'Not Sure' counted as non-agreement, to report inter-annotator agreement (e.g., Cohen's kappa on the full set), and, since both annotators are authors of the paper and designers of SONAR, to have the annotation audit performed or repeated by annotators not involved in designing the framework.","section":"IV-C1, Table I"}],"minor_comments":[{"comment":"The abstract and §IV-D4 state that correlations are 'up to 14X higher' than the best baseline, but the ratios implied by Table III are mostly between roughly 1.2 and 4 (for example, 0.59/0.15 for Pass@1 and 0.41/0.20 for Success Rate); please specify exactly which task metric and which baseline produce the 14× ratio and how that ratio is computed.","section":"Abstract; IV-D4"},{"comment":"Many correlations are tested across six task metrics and multiple dimensions without any multiplicity control; please add a note on how many of the starred entries survive, for instance, an FDR correction at 5%.","section":"Tables III and IV"},{"comment":"Fig. 5 reports only mean scores per LLM with no measure of dispersion or significance testing, yet the text makes comparative claims (e.g., Gemini and Kimi scoring 0.14 and 0.10 below average in Fluency); error bars or a paired test would make these claims checkable.","section":"IV-E, Fig. 5"},{"comment":"In Eq. (3), the compression ratio is credited only when FuncSim(x̂_i, x̂) = 1.0 holds against the code regenerated from the original summary, x̂, rather than against the original source code x; if x̂ is behaviorally wrong because the summary is wrong, an informative but not-yet-correct summary can be penalized for compressibility, so the Conciseness score should be interpreted with this caveat stated.","section":"III-A3, Eq. (3)"},{"comment":"In Table II, the Translation row lists two metrics ('CA@1 [53], Edit Distance [54]') inside a single cell; please separate the metrics into distinct rows or columns for readability.","section":"Table II"},{"comment":"The sensitivity analysis in RQ4 uses 1,000 'randomly sampled summaries'; please state whether these come from the same 500-function pool and whether they overlap with the RQ1 or RQ2 summary sets.","section":"IV-F1"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the validation of the two load-bearing components — the differential-fuzzing oracle and the assumption that regeneration depends on the summary text — leans heavily on self-cited prior work [31], [32] and on annotations by the paper's own authors. There is no artifact availability statement or external check of the fuzzer. The '14X' claim in the abstract also appears difficult to reconcile with Table III as printed and should be corrected. The paper's topic, an evaluation-methodology contribution for code summarization, fits the journal's scope, provided the RQ2 confounding issue is resolved and the computational feasibility of the framework (large numbers of LLM regeneration calls) is documented."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper introduces a genuinely new way to evaluate code summaries: instead of comparing against a gold reference, it regenerates code from the summary plus minimal context, then scores correctness by functional similarity, abstraction by diversity of regenerated implementations, and conciseness by compressibility. That is a real conceptual step beyond BLEU/ROUGE/BERTScore and beyond LLM-as-a-judge. Second, the main empirical claim—that Correctness and Abstraction predict downstream LLM performance much better than baseline metrics—is plausible but not yet airtight. The design leaves open a specific confound: the context given to the regenerator includes the function name and signature, and the evaluation pool is drawn from public benchmarks (HumanEval, MBPP, BigCodeBench, The Vault) whose names are often descriptive and whose solutions are memorized. If the generator can reconstruct the code from the name alone, the correctness score stops measuring summary quality. The threats section admits the influence of pretrained knowledge but never quantifies it. A control—say, regenerating with the summary omitted or with a wrong summary—would settle this.\n\nWhat the paper does well: the abstraction dimension is a novel and interesting operationalization; the empirical breadth is real (11 summarizers, four downstream tasks, three task LLMs); the sensitivity analysis for generator choice is a good sign; and the finding that what matters varies by task is a useful contribution regardless of the specific numbers. The RQ3 model comparison is also informative.\n\nThe soft spots, in proportion. The fuzzer test set T is never described: size, coverage, generation method all absent, and the oracle is self-cited. That is a reasonable concern but downstream of the leakage issue—even a perfect oracle doesn't help if the regeneration isn't driven by the summary. The RQ1 ground truth is two authors, unanimous pairs only, with no external annotators; the agreement rate is computed on a filtered subset, so it may not generalize. And there is no artifact release: no code, no data, no prompts. The 'up to 14X' headline is a max over tasks and baselines; the paper should report the full distribution rather than the single best case.\n\nWho is this for: anyone working on code summarization evaluation, LLM-based SE agents, or task-aware generation. The idea has legs.\n\nMy recommendation: send it to peer review, but insist on major revision: release artifacts, add a control showing regeneration depends on summary content, report T in detail, and get external annotations for at least a subset of the preference pairs. If those land, this becomes a solid and citable contribution.","headline":"SONAR is a novel reference-free, regeneration-based evaluation framework for code summaries that deserves serious review, but its core Correctness signal needs a control experiment to rule out function-name leakage before the '14X' claim can be trusted.","tokens_in":20208,"tokens_out":3329,"would_cite":true,"duration_ms":30857,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SONAR evaluates code summaries by regenerating code from them, and shows that Correctness and Abstraction scores predict LLM performance on downstream software tasks up to 14 times better than reference-based baselines.","keywords":["code summarization","reference-free evaluation","round-trip correctness","code regeneration","task-aware summarization","differential fuzzing","summary quality dimensions","LLM-as-consumer"],"falsifier":"Recompute SONAR's Correctness and Conciseness for the same summary-code pairs using a fixed, exhaustive or mutation-based test oracle, then re-measure the Spearman correlations with downstream task performance; if the 1.3 to 14 times advantage over reference-based baselines does not survive the oracle change, the predictive power is an artifact of the fuzzer's input distribution rather than a property of the summary.","tokens_in":19125,"feed_emoji":"🤖","tokens_out":8923,"duration_ms":66750,"temperature":0.7,"pith_summary":"The paper claims that the quality of a code summary should be judged by how well it serves the LLM that consumes it, not by how closely it matches developer-written documentation. To make this measurable without gold references, SONAR uses a 'regenerate, then evaluate' loop: a code-generator LLM rebuilds the source code from the summary, and the rebuilt code is probed to score the summary on four dimensions—Correctness, Abstraction, Conciseness, and Fluency. Across four downstream software engineering tasks and eleven summarizer LLMs, the paper finds that Correctness, followed by Abstraction, significantly correlates with task performance, with correlations up to 14 times higher than the best reference-based baseline, while Conciseness and Fluency show little signal. If correct, this reframes summary evaluation as task- and consumer-dependent and opens the way to reference-free, execution-grounded summary scoring.","feed_headline":"Reference-free code summary scores predict LLM tasks 14x better","feed_subtitle":"Regenerating code from a summary reveals which summaries actually help LLM agents on software engineering tasks.","key_machinery":"The central mechanism is round-trip correctness adapted as 'regenerate, then evaluate': code is turned into a summary, the summary is fed back to a code-generator LLM, and the regenerated code is analyzed. The load-bearing oracle is differential fuzzing: functional similarity is the fraction of automatically generated test inputs on which original and regenerated code produce identical outputs, and this same oracle underlies both Correctness and Conciseness. Abstraction is operationalized as the complement of average pairwise similarity, combining structural (code-property-graph edge) and lexical (token-overlap) similarity across a pool of regenerated implementations from several generators.","core_discovery":"SONAR's central discovery is that a summary's code-regeneration behavior is a usable proxy for its quality as an input to LLM-based software engineering. Correctness is measured by differential-fuzzing functional equivalence between original and summary-regenerated code; Abstraction by the diversity of a pool of regenerated implementations (via structural and lexical similarity of code-property graphs); Conciseness by the maximum compression ratio that still preserves functional equivalence; and Fluency by inverse perplexity. Validated against human ground-truth labels, SONAR agrees with annotators 90% of the time overall. In the downstream-task study, every task metric has at least one significantly correlated SONAR dimension, with predictive gains of a factor of 1.3 to 14 over the best reference-based baseline; correctness dominates for semantics-preserving tasks while abstraction leads for retrieval and translation diversity. The same scores reveal that modern LLMs trade off abstraction against correctness, and that dimension-aware prompting can improve abstraction by up to 10 percentage points without retraining.","pith_inferences":["The same regeneration-based logic could extend beyond function summaries to other natural-language software artifacts—commit messages, issue reports, API documentation—wherever the artifact is meant to specify behavior; the abstraction measure would then quantify how much implementation freedom the artifact leaves.","If abstraction truly predicts cross-language retrieval, one testable extension is to use SONAR's abstraction score as a filter or training signal for retrieval-oriented summarizers, something embedding-based objectives do not explicitly reward.","The paper shows human-valued conciseness does not matter to LLMs, but it does not measure whether human developers reading the same summaries show the same task-performance pattern; a direct human-vs-agent comparison would test whether the two consumers genuinely diverge.","Because the fuzzing oracle is the linchpin, a robustness check would be to re-run the downstream correlations on a subsample using an independent oracle (e.g., bounded exhaustive test generation); if the advantage over baselines collapses, the fuzzer's input distribution, not the summary itself, carries the result."],"forward_implications":["Summaries can be scored without gold references, human judgments, or LLM-as-judge: code regeneration supplies an execution-grounded signal.","Evaluation becomes task-aware: downstream tasks map to the dimension that predicts them, e.g., correctness for translation and optimization, abstraction for retrieval and translation diversity.","LLM summarizers can be steered cheaply: dimension-aware prompting improves abstraction by up to 10 percentage points, giving a lightweight way to tailor summaries to a consumer.","Conciseness and fluency appear unimportant to LLM consumers, so pipelines serving agents should optimize correctness and abstraction instead of human-preferred brevity.","SONAR scores are stable under internal LLM substitution (rank correlations 0.88–0.94), so the framework is not tied to one generator model."],"supporting_citations":[{"why":"Supplies the round-trip correctness idea that SONAR adapts into its regenerate-then-evaluate loop.","marker":"[17]"},{"why":"Grounds the differential-fuzzing approach used as the functional-equivalence oracle in Eq. 1.","marker":"[31]"},{"why":"Extends the differential-fuzzing oracle to LLM-generated code, the basis for the Correctness and Conciseness scorers.","marker":"[32]"},{"why":"The reference-free baseline whose design SONAR contrasts with, since it still relies on training toward developer-written documentation.","marker":"[6]"},{"why":"One of the two executable-code benchmarks that supply both the functions to summarize and the human-written descriptions used as reference baselines.","marker":"[45]"},{"why":"The second executable-code benchmark providing functions and reference descriptions for summary generation and downstream tasks.","marker":"[46]"},{"why":"Supplies the test-oracle-generation task where summaries are consumed as input, one of the four downstream tasks.","marker":"[14]"},{"why":"Supplies the code-translation task that uses summaries as a pivot, one of the four downstream tasks.","marker":"[16]"},{"why":"Provides the program-analysis representation used to compute structural similarity for the Abstraction scorer.","marker":"[36]"},{"why":"Supplies the cross-language retrieval setup and candidate pool used for the retrieval task.","marker":"[13]"}],"fun_headline_variants":["Regenerate code to score summaries for LLM agents","SONAR: reference-free summary scores predict LLM tasks 14x better","Summary quality for LLMs? Check if code can be regenerated","Why code summaries need task-aware scoring, not human preference","Trading abstraction for correctness: LLMs' summary weakness exposed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The Correctness and Conciseness scores, and therefore the downstream correlations, depend entirely on the differential fuzzer discovering every behavioral difference between original and regenerated code; the paper does not report the generated test set's size or coverage, inheriting the fuzzer's validity from prior work rather than demonstrating it here.","fun_headline_variants_meta":{"raw":{"variants":["Regenerate code to score summaries for LLM agents","SONAR: reference-free summary scores predict LLM tasks 14x better","Summary quality for LLMs? Check if code can be regenerated","Why code summaries need task-aware scoring, not human preference","Trading abstraction for correctness: LLMs' summary weakness exposed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000261,"raw_usage":{"total_tokens":1634,"prompt_tokens":1028,"completion_tokens":606,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":519}},"tokens_in":644,"tokens_out":606,"duration_ms":5855,"temperature":1.0,"reasoning_tokens":519,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:41:44.696810+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute SONAR's Correctness and Conciseness for the same summary-code pairs using a fixed, exhaustive or mutation-based test oracle, then re-measure the Spearman correlations with downstream task performance; if the 1.3 to 14 times advantage over reference-based baselines does not survive the oracle change, the predictive power is an artifact of the fuzzer's input distribution rather than a property of the summary.","supporting_citations":[{"cited_title":"Unsupervised evaluation of code llms with round-trip correctness,","cited_arxiv_id":null,"evidence_quote":"Supplies the round-trip correctness idea that SONAR adapts into its regenerate-then-evaluate loop."},{"cited_title":"A differential fuzzing-based evaluation of functional equivalence in llm-generated code refactorings,","cited_arxiv_id":null,"evidence_quote":"Extends the differential-fuzzing oracle to LLM-generated code, the basis for the Correctness and Conciseness scorers."},{"cited_title":"Summarize and generate to back-translate: Unsupervised translation of programming languages,","cited_arxiv_id":null,"evidence_quote":"Supplies the code-translation task that uses summaries as a pivot, one of the four downstream tasks."},{"cited_title":"Joern - the bug hunter’s workbench,","cited_arxiv_id":null,"evidence_quote":"Provides the program-analysis representation used to compute structural similarity for the Abstraction scorer."},{"cited_title":"Unicor: Modality collaboration for robust cross-language hybrid code retrieval,","cited_arxiv_id":null,"evidence_quote":"Supplies the cross-language retrieval setup and candidate pool used for the retrieval task."}],"review_version":2}