Pith. sign in

REVIEW 4 major objections 6 minor 68 references

SONAR: Task-Aware Code Summary Evaluation for LLM Consumers Without References

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SONAR evaluates code summaries by regenerating code from them, and shows that Correctness and Abstraction scores predict LLM performance on downstream software tasks up to 14 times better than reference-based baselines.

desk verdict SONAR is a novel reference-free, regeneration-based evaluation framework for code summaries that deserves serious review, but its core Correctness signal needs a control experiment to rule out function-name leakage before the '14X' claim can be trusted. read the letter →

arxiv 2608.04195 v1 pith:NGJABKUR submitted 2026-08-04 cs.SE

Simantika Bhattacharjee Dristi , Matthew B. Dwyer This is my paper · ORCID
classification cs.SE
keywords codesummarizationreference-freeevaluationround-tripcorrectnessregenerationtask-awaredifferentialfuzzingsummaryqualitydimensionsLLM-as-consumer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the quality of a code summary should be judged by how well it serves the LLM that consumes it, not by how closely it matches developer-written documentation. To make this measurable without gold references, SONAR uses a 'regenerate, then evaluate' loop: a code-generator LLM rebuilds the source code from the summary, and the rebuilt code is probed to score the summary on four dimensions—Correctness, Abstraction, Conciseness, and Fluency. Across four downstream software engineering tasks and eleven summarizer LLMs, the paper finds that Correctness, followed by Abstraction, significantly correlates with task performance, with correlations up to 14 times higher than the best reference-based baseline, while Conciseness and Fluency show little signal. If correct, this reframes summary evaluation as task- and consumer-dependent and opens the way to reference-free, execution-grounded summary scoring.

What carries the argument

The central mechanism is round-trip correctness adapted as 'regenerate, then evaluate': code is turned into a summary, the summary is fed back to a code-generator LLM, and the regenerated code is analyzed. The load-bearing oracle is differential fuzzing: functional similarity is the fraction of automatically generated test inputs on which original and regenerated code produce identical outputs, and this same oracle underlies both Correctness and Conciseness. Abstraction is operationalized as the complement of average pairwise similarity, combining structural (code-property-graph edge) and lexical (token-overlap) similarity across a pool of regenerated implementations from several generators.

What would settle it

Recompute SONAR's Correctness and Conciseness for the same summary-code pairs using a fixed, exhaustive or mutation-based test oracle, then re-measure the Spearman correlations with downstream task performance; if the 1.3 to 14 times advantage over reference-based baselines does not survive the oracle change, the predictive power is an artifact of the fuzzer's input distribution rather than a property of the summary.

Watch

Extended reading notes

Core claim

SONAR's central discovery is that a summary's code-regeneration behavior is a usable proxy for its quality as an input to LLM-based software engineering. Correctness is measured by differential-fuzzing functional equivalence between original and summary-regenerated code; Abstraction by the diversity of a pool of regenerated implementations (via structural and lexical similarity of code-property graphs); Conciseness by the maximum compression ratio that still preserves functional equivalence; and Fluency by inverse perplexity. Validated against human ground-truth labels, SONAR agrees with annotators 90% of the time overall. In the downstream-task study, every task metric has at least one significantly correlated SONAR dimension, with predictive gains of a factor of 1.3 to 14 over the best reference-based baseline; correctness dominates for semantics-preserving tasks while abstraction leads for retrieval and translation diversity. The same scores reveal that modern LLMs trade off abstraction against correctness, and that dimension-aware prompting can improve abstraction by up to 10 percentage points without retraining.

Load-bearing premise

The Correctness and Conciseness scores, and therefore the downstream correlations, depend entirely on the differential fuzzer discovering every behavioral difference between original and regenerated code; the paper does not report the generated test set's size or coverage, inheriting the fuzzer's validity from prior work rather than demonstrating it here.

Editorial extensions

If this is right

  • Summaries can be scored without gold references, human judgments, or LLM-as-judge: code regeneration supplies an execution-grounded signal.
  • Evaluation becomes task-aware: downstream tasks map to the dimension that predicts them, e.g., correctness for translation and optimization, abstraction for retrieval and translation diversity.
  • LLM summarizers can be steered cheaply: dimension-aware prompting improves abstraction by up to 10 percentage points, giving a lightweight way to tailor summaries to a consumer.
  • Conciseness and fluency appear unimportant to LLM consumers, so pipelines serving agents should optimize correctness and abstraction instead of human-preferred brevity.
  • SONAR scores are stable under internal LLM substitution (rank correlations 0.88–0.94), so the framework is not tied to one generator model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same regeneration-based logic could extend beyond function summaries to other natural-language software artifacts—commit messages, issue reports, API documentation—wherever the artifact is meant to specify behavior; the abstraction measure would then quantify how much implementation freedom the artifact leaves.
  • If abstraction truly predicts cross-language retrieval, one testable extension is to use SONAR's abstraction score as a filter or training signal for retrieval-oriented summarizers, something embedding-based objectives do not explicitly reward.
  • The paper shows human-valued conciseness does not matter to LLMs, but it does not measure whether human developers reading the same summaries show the same task-performance pattern; a direct human-vs-agent comparison would test whether the two consumers genuinely diverge.
  • Because the fuzzing oracle is the linchpin, a robustness check would be to re-run the downstream correlations on a subsample using an independent oracle (e.g., bounded exhaustive test generation); if the advantage over baselines collapses, the fuzzer's input distribution, not the summary itself, carries the result.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes SONAR, a reference-free framework for evaluating LLM-generated source code summaries along four dimensions: Correctness, Abstraction, Conciseness, and Fluency. The framework follows a 'regenerate, then evaluate' principle: a summary, together with minimal code context (function name, return type, parameter names, imports), is given to a code generator, and the functional similarity between the original and regenerated code, computed via differential fuzzing, serves as the quality signal. Correctness is this functional similarity directly; Abstraction is the diversity of a pool of regenerated implementations; Conciseness is the complement of the largest behavior-preserving compression ratio; Fluency is inverse perplexity. The authors validate the three regeneration-based dimensions against human preference labels (90% overall agreement), correlate SONAR scores with LLM performance across four downstream software engineering tasks (retrieval, translation, optimization, test oracle generation) with six metrics and three task LLMs, evaluate 11 summarizer LLMs under three prompting strategies, and run a sensitivity analysis across scorer LLMs.

Significance. If the findings hold, this is a useful contribution to code summarization evaluation: it offers an execution-grounded, reference-free alternative to BLEU/ROUGE/BERTScore-style metrics, and it gives quantitative support to the increasingly common intuition that what makes a summary useful to an LLM consumer differs from what human developers value. The experimental breadth is a genuine strength: 11 summarizer LLMs, four downstream tasks, three task LLMs, six metrics, sensitivity analysis, and partial correlations. The differential-fuzzing oracle and the use of downstream task performance as an external test are the right kind of evidence for a reference-free metric, and the Abstraction-as-diversity construct is novel and interesting, though it rests on the a priori axiom that implementation diversity measures level of abstraction.

major comments (4)
  1. [III-A, Eq. (1); IV-A] The paper's central mechanism, namely that regenerated code 'empirically grounds' summary evaluation, is not tested against the possibility that regeneration depends on the minimal context c rather than on the summary s. Because c includes the function name, return type, and parameter names (Section III-A), and because the evaluation pool is drawn from publicly available benchmarks (HumanEval, MBPP, BigCodeBench, The Vault) whose function names are often highly descriptive (e.g., greatest_common_divisor in Fig. 1), the generator G may reconstruct a canonical implementation from c alone or from memorized benchmark solutions, making FuncSim(x, G(s,c)) high regardless of s. The threats section (§V-A) concedes 'the influence of their pretrained knowledge' as a factor, but no experiment quantifies it. I ask for a control that computes FuncSim(x, G(c)) with the summary omitted and with a deliberately incorrect summary, reporting how often and by how much the summary changes the regenerated code, along with the distribution of Correctness scores; without this, the claim that Correctness measures summary quality rather than the function's name and benchmark notoriety is unsubstantiated.
  2. [IV-D3, Table III] The RQ2 correlations pool multiple summaries per source function (roughly 14–21 summaries per function across the approximately 100 sampled HumanEval/MBPP functions), so the reported Spearman correlations can be driven by between-function difficulty rather than by summary quality: functions that are inherently easier also tend to receive higher SONAR Correctness and Abstraction scores and higher downstream success rates. This confound directly affects the conclusion in §IV-D4 that 'the summary itself drives downstream performance.' I ask the authors to recompute the significant correlations in Table III within functions (e.g., function-stratified or with function and summarizer as covariates) and to report whether the signals survive; if they do not, the task-dependence claim would need to be restricted to between-function comparisons.
  3. [III-A1, Eq. (1)] FuncSim in Eq. (1) is the cornerstone of both the Correctness and Conciseness scorers, but the test input set T is not characterized: the paper does not report |T|, how T is generated per function, what coverage is achieved, or any calibration of the fuzzer's ability to detect behavioral differences (for example, mutation-based false-negative rates). The fuzzer's validity is inherited from two self-cited papers [31], [32] rather than demonstrated here, and the stated fallback to reference-based code evaluation metrics for code that cannot be run in isolation introduces a second oracle whose share in the reported scores is unspecified. I ask that the authors report the distribution of |T| and of FuncSim across the function pool, validate the oracle on functions with known behavioral variants, and state explicitly how many summaries in each experiment (RQ1, RQ2, RQ4) were scored with the reference-based proxy instead of execution.
  4. [IV-C1, Table I] The RQ1 validation protocol starts with 100 summary pairs per dimension, discards any pair on which the two annotators do not unanimously agree (retaining 84, 86, and 89 pairs for Correctness, Abstraction, and Conciseness, respectively), and reports agreement only on the retained pairs. This makes the reported 87–96% agreement rates conditional on the annotators having agreed, which inflates the rates relative to an unbiased estimate, and it makes disagreements and 'Not Sure' responses invisible. I ask the authors to report agreement on the full 100 pairs with disagreement and 'Not Sure' counted as non-agreement, to report inter-annotator agreement (e.g., Cohen's kappa on the full set), and, since both annotators are authors of the paper and designers of SONAR, to have the annotation audit performed or repeated by annotators not involved in designing the framework.
minor comments (6)
  1. [Abstract; IV-D4] The abstract and §IV-D4 state that correlations are 'up to 14X higher' than the best baseline, but the ratios implied by Table III are mostly between roughly 1.2 and 4 (for example, 0.59/0.15 for Pass@1 and 0.41/0.20 for Success Rate); please specify exactly which task metric and which baseline produce the 14× ratio and how that ratio is computed.
  2. [Tables III and IV] Many correlations are tested across six task metrics and multiple dimensions without any multiplicity control; please add a note on how many of the starred entries survive, for instance, an FDR correction at 5%.
  3. [IV-E, Fig. 5] Fig. 5 reports only mean scores per LLM with no measure of dispersion or significance testing, yet the text makes comparative claims (e.g., Gemini and Kimi scoring 0.14 and 0.10 below average in Fluency); error bars or a paired test would make these claims checkable.
  4. [III-A3, Eq. (3)] In Eq. (3), the compression ratio is credited only when FuncSim(x̂_i, x̂) = 1.0 holds against the code regenerated from the original summary, x̂, rather than against the original source code x; if x̂ is behaviorally wrong because the summary is wrong, an informative but not-yet-correct summary can be penalized for compressibility, so the Conciseness score should be interpreted with this caveat stated.
  5. [Table II] In Table II, the Translation row lists two metrics ('CA@1 [53], Edit Distance [54]') inside a single cell; please separate the metrics into distinct rows or columns for readability.
  6. [IV-F1] The sensitivity analysis in RQ4 uses 1,000 'randomly sampled summaries'; please state whether these come from the same 500-function pool and whether they overlap with the RQ1 or RQ2 summary sets.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SONAR scores are defined independently and validated against external downstream task performance, so the central correlations are not forced by construction.

full rationale

SONAR's derivation chain is not circular. Correctness (Def. 1) is defined as FuncSim(x, G(s,c)) using differential fuzzing (Eq. 1), and the RQ2 claim is an empirical Spearman correlation between these pre-defined scores and downstream task metrics (Table III). The downstream tasks are not used to fit or define the SONAR scores, so a correlation is independent evidence rather than a construction. Abstraction (Def. 2) and Conciseness (Def. 3) are also fixed operationalizations; they are not derived from the downstream results. The self-citations [31], [32] supply a functional-equivalence oracle, but the oracle's validity is an external methodological dependency, not an assumption of the target conclusion. RQ1's human-annotation validation is a potential bias threat because the annotators are the authors, but the paper reports that annotators did not have access to SONAR scores during labeling, and any such bias is a validity concern, not a circular reduction. The threats section explicitly concedes that scores may be affected by pretrained knowledge and fuzzing noise, which is a limitation acknowledgment rather than evidence that the claim is definitionally forced. No equation in the paper reduces to another by construction, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The framework's scores depend on several hand-chosen hyperparameters (U, V, n, fuzzer test set size, temperature) and on domain assumptions about LLM behavior and fuzzing validity. The abstraction construct is defined by the authors and validated only against their own annotations. No new physical entities are introduced.

free parameters (5)
  • U (number of generator models in Abstraction pool) = 3
    Chosen by hand; the diversity pool size affects Abstraction scores and the stability of the measure.
  • V (samples per generator) = 3
    Chosen by hand; together with U it determines the number of regenerated implementations k=9 used in ImplDiv.
  • n (number of compressed summaries in Conciseness) = 3
    Chosen by hand; more compressions improve reliability but add cost; the conciseness score is the complement of the maximum compression ratio.
  • Fuzzer test input set T size and coverage = not reported
    The functional similarity oracle in Eq. (1) depends on T, but the paper does not state how many inputs are generated or what coverage is targeted; this directly affects all Correctness and Conciseness scores.
  • Decoding temperature = not reported
    All LLM generations are run with identical decoding parameters and temperature settings, but the actual temperature value is not given; this affects regeneration diversity and scores.
assumptions (4)
  • domain assumption Differential fuzzing with byte-level generated test inputs yields a valid functional equivalence oracle.
    Correctness and Conciseness scores rely on Eq. (1); the validity of the oracle is taken from self-cited prior work [31], [32] and is not independently established in this paper.
  • ad hoc to paper Diversity of regenerated implementations measures summary abstraction.
    Definition 2 defines abstraction as implementation-level diversity; this construct is new and is validated only against the authors' own preference annotations.
  • domain assumption Code generator LLMs follow the instruction to preserve implementation-specific details stated in the summary.
    The Abstraction scorer instructs generators to preserve details when present; if they fail to do so, the diversity signal is weakened or distorted.
  • domain assumption Inverse perplexity measures fluency.
    Fluency is defined as inverse GPT-2 perplexity, following prior work [41], [42]; this is a standard approximation but does not measure human-perceived readability directly.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SONAR: Task-Aware Code Summary Evaluation for LLM Consumers Without References." pith.science (2026). https://pith.science/paper/NGJABKUR

@misc{pith2026260804195,
  author       = {Pith},
  title        = {Pith review of: SONAR: Task-Aware Code Summary Evaluation for LLM Consumers Without References},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NGJABKUR}},
  note         = {Machine review of arXiv:2608.04195}
}
read the original abstract

Source code summaries have traditionally been evaluated from a human developer's perspective, with quality determined by how closely they resemble developer-written references and how well they align with human preferences. But this overlooks a growing reality: LLM-based tools and agents increasingly consume code summaries as inputs for software engineering (SE) tasks, and what makes a summary useful for a consuming agent on a task remains largely unexplored. To bridge this gap, we propose SONAR, a reference-free framework that evaluates source code summaries along four dimensions: Correctness, Abstraction, Conciseness, and Fluency. Rather than optimizing for a pre-written "gold standard", SONAR introduces a novel code regeneration-based approach that uses a summary to regenerate code and leverages that reconstruction as a quality signal of the summary. This provides an empirical grounding that requires neither a reference summary nor the subjective judgment of humans or LLMs. We evaluate SONAR's dimensions on their ability to influence LLM performance across four downstream SE tasks. We find that Correctness, followed by Abstraction, significantly correlates with LLM performance, with correlations up to 14X higher than the best baseline. Conciseness and Fluency, though widely valued by human developers, remain mostly insignificant to an LLM consumer, suggesting that what makes a summary useful is task- and consumer-dependent. Through a large-scale evaluation of 11 popular LLMs using SONAR, we further identify the strengths and weaknesses of different models across each quality dimension, while offering insights to facilitate future research on task-aware summarization.

Figures

Figures reproduced from arXiv: 2608.04195 by the authors.

Figure 1
Figure 1. LLM-generated summaries vary across quality dimen [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The SONAR Framework 2) Abstraction Scorer: This module approximates abstrac￾tion in a code summary through the diversity of regenerated implementations: the more implementation details a summary encodes, the more constrained the regeneration space is, and the less diverse the regenerated implementations will be. Definition 2 (Abstraction): Given source code x, summary s, and a set of regenerated implementations, Xˆ,… view at source ↗
Figure 3
Figure 3. Implementation details in a summary affect the diversity of regenerated implementations. Red borders highlight [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Prompt templates used for summary generation. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 6
Figure 6. Figure 6: Effect of prompting strategy on S [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 39 canonical work pages

  1. [31]

    Analyzing and mitigating surface bias in code evaluation metrics,

    S. B. Dristi and M. B. Dwyer, “Analyzing and mitigating surface bias in code evaluation metrics,” 2025. [Online]. Available: https: //arxiv.org/abs/2509.15397

    arXiv 2025

  2. [32]

    A differential fuzzing-based evaluation of functional equivalence in llm-generated code refactorings,

    ——, “A differential fuzzing-based evaluation of functional equivalence in llm-generated code refactorings,” 2026. [Online]. Available: https://arxiv.org/abs/2602.15761

    work page arXiv 2026

  3. [1]

    A survey on machine learning techniques for source code analysis,

    T. Sharma, M. Kechagia, S. Georgiou, R. Tiwari, I. Vats, H. Moazen, and F. Sarro, “A survey on machine learning techniques for source code analysis,” 2022. [Online]. Available: https://arxiv.org/abs/2110.09610

    arXiv 2022

  4. [2]

    Source code summarization in the era of large language models,

    W. Sun, Y . Miao, Y . Li, H. Zhang, C. Fang, Y . Liu, G. Deng, Y . Liu, and Z. Chen, “Source code summarization in the era of large language models,” 2025. [Online]. Available: https://arxiv.org/abs/2407.07959

    arXiv 2025

  5. [3]

    Calibration of large language models on code summarization,

    Y . Virk, P. Devanbu, and T. Ahmed, “Calibration of large language models on code summarization,”Proc. ACM Softw. Eng., vol. 2, no. FSE, Jun. 2025. [Online]. Available: https://doi.org/10.1145/3729400

    doi:10.1145/3729400 2025

  6. [4]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” inProceedings of the 40th Annual Meeting on Association for Computational Linguistics, ser. ACL ’02. USA: Association for Computational Linguistics, 2002, p. 311–318. [Online]. Available: https://doi.org/10.3115/1073083.1073135

    arXiv 2002

  7. [5]

    Simllm: Calculating semantic similarity in code summaries using a large language model-based approach,

    X. Jin and Z. Lin, “Simllm: Calculating semantic similarity in code summaries using a large language model-based approach,”Proc. ACM Softw. Eng., vol. 1, no. FSE, Jul. 2024. [Online]. Available: https://doi.org/10.1145/3660769

    work page doi:10.1145/3660769 2024

  8. [6]

    Evaluating code summarization techniques: A new metric and an empirical characterization,

    A. Mastropaolo, M. Ciniselli, M. D. Penta, and G. Bavota, “Evaluating code summarization techniques: A new metric and an empirical characterization,” 2023. [Online]. Available: https: //arxiv.org/abs/2312.15475

    arXiv 2023

Show all 68 references
  1. [7]

    METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,” inProceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, J. Goldstein, A. Lavie, C.-Y...

    2005

  2. [8]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” 01 2004, p. 10

    2004

  3. [9]

    On the effectiveness of llm-as-a-judge for code generation and summarization,

    G. Crupi, R. Tufano, A. Velasco, A. Mastropaolo, D. Poshyvanyk, and G. Bavota, “On the effectiveness of llm-as-a-judge for code generation and summarization,” 2025. [Online]. Available: https: //arxiv.org/abs/2507.16587

    2025 arXiv

  4. [10]

    Can large language models serve as evaluators for code summarization?

    Y . Wu, Y . Wan, Z. Chu, W. Zhao, Y . Liu, H. Zhang, X. Shi, H. Jin, and P. S. Yu, “Can large language models serve as evaluators for code summarization?”IEEE Transactions on Software Engineering, vol. 51, no. 12, pp. 3205–3217, 2025

    2025

  5. [11]

    Artificial intelligence for software engi- neering: The journey so far and the road ahead,

    I. Ahmed, A. Aleti, H. Cai, A. Chatzigeorgiou, P. He, X. Hu, M. Pezz `e, D. Poshyvanyk, and X. Xia, “Artificial intelligence for software engi- neering: The journey so far and the road ahead,”ACM Transactions on Software Engineering and Methodology, vol. 34, no. 5, pp. 1–27, 2025

    2025

  6. [12]

    Optimizing datasets for code summarization: Is code-comment coherence enough?

    A. Vitale, A. Mastropaolo, R. Oliveto, M. D. Penta, and S. Scalabrino, “Optimizing datasets for code summarization: Is code-comment coherence enough?” 2025. [Online]. Available: https://arxiv.org/abs/ 2502.07611

    2025 arXiv

  7. [13]

    Unicor: Modality collaboration for robust cross-language hybrid code retrieval,

    Y . Yang, L. Kuang, J. Liu, Z. Liu, Y . Xia, and D. Lo, “Unicor: Modality collaboration for robust cross-language hybrid code retrieval,”

  8. [14]

    Doc2oracll: Investigating the impact of documentation on llm-based test oracle generation,

    S. B. Hossain, R. Taylor, and M. Dwyer, “Doc2oracll: Investigating the impact of documentation on llm-based test oracle generation,”Proc. ACM Softw. Eng., vol. 2, no. FSE, Jun. 2025. [Online]. Available: https://doi.org/10.1145/3729354

    2025 doi

  9. [15]

    Nl in the middle: Code translation with llms and intermediate representations,

    C.-e. A. Tai, P. Nie, L. Golab, and A. Wong, “Nl in the middle: Code translation with llms and intermediate representations,” in2025 IEEE International Conference on Collaborative Advances in Software and COmputiNg (CASCON), 2025, pp. 295–300

    2025

  10. [16]

    Summarize and generate to back-translate: Unsupervised translation of programming languages,

    W. U. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “Summarize and generate to back-translate: Unsupervised translation of programming languages,” inProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, A. Vlachos and I....

    2023

  11. [17]

    Unsupervised evaluation of code llms with round-trip correctness,

    M. Allamanis, S. Panthaplackel, and P. Yin, “Unsupervised evaluation of code llms with round-trip correctness,” inProceedings of the 41st Inter- national Conference on Machine Learning, ser. ICML’24. JMLR.org, 2024

    2024

  12. [18]

    Simple and effective baselines for code summarisation evaluation,

    J. Robinson and J. K. Kummerfeld, “Simple and effective baselines for code summarisation evaluation,” 2025. [Online]. Available: https://arxiv.org/abs/2505.19392

    2025 arXiv

  13. [19]

    Beyond accuracy: Evaluating self-consistency of code large language models with identitychain,

    M. J. Min, Y . Ding, L. Buratti, S. Pujar, G. Kaiser, S. Jana, and B. Ray, “Beyond accuracy: Evaluating self-consistency of code large language models with identitychain,” 2024. [Online]. Available: https://arxiv.org/abs/2310.14053

    2024 arXiv

  14. [20]

    Reassessing automatic evaluation metrics for code summarization tasks,

    D. Roy, S. Fakhoury, and V . Arnaoudova, “Reassessing automatic evaluation metrics for code summarization tasks,” inProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 202...

    2021

  15. [21]

    Bertscore: Evaluating text generation with bert,

    T. Zhang*, V . Kishore*, F. Wu*, K. Q. Weinberger, and Y . Artzi, “Bertscore: Evaluating text generation with bert,” inInternational Conference on Learning Representations, 2020. [Online]. Available: https://openreview.net/forum?id=SkeHuCVFDr

    2020

  16. [22]

    Bleurt: Learning robust metrics for text generation,

    T. Sellam, D. Das, and A. Parikh, “Bleurt: Learning robust metrics for text generation,” inProceedings of the 58th annual meeting of the association for computational linguistics, 2020, pp. 7881–7892

    2020

  17. [23]

    Sentence-bert: Sentence embeddings using siamese bert-networks,

    N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” 2019. [Online]. Available: https: //arxiv.org/abs/1908.10084

    2019 arXiv

  18. [24]

    A large-scale empirical study on code-comment inconsistencies,

    F. Wen, C. Nagy, G. Bavota, and M. Lanza, “A large-scale empirical study on code-comment inconsistencies,” inProceedings of the 27th International Conference on Program Comprehension, ser. ICPC ’19. IEEE Press, 2019, p. 53–64. [Online]. Available: https://doi.org/10.1109/ICPC....

    2019

  19. [25]

    Automated test generation from program documentation encoded in code comments,

    G. Denaro and L. Guglielmo, “Automated test generation from program documentation encoded in code comments,” 2025. [Online]. Available: https://arxiv.org/abs/2504.21161

    2025 arXiv

  20. [26]

    Automatic generation of oracles for exceptional behaviors,

    A. Goffi, A. Gorla, M. D. Ernst, and M. Pezz `e, “Automatic generation of oracles for exceptional behaviors,” inProceedings of the 25th International Symposium on Software Testing and Analysis, ser. ISSTA 2016. New York, NY , USA: Association for Computing Machinery, 2016, p. ...

    2016

  21. [27]

    An approach of method-level bug localization,

    Z. Ni, L. Bo, B. Li, T. Chen, X. Sun, and X. Wu, “An approach of method-level bug localization,”IET Software, vol. 16, no. 4, p. 422–437, May 2022. [Online]. Available: https://doi.org/10.1049/sfw2.12060

    2022 doi

  22. [28]

    Natural language summarization enables multi-repository bug localization by llms in microservice architectures,

    A. R. Oskooei, S. S. Yukcu, M. C. Bozoglan, and M. S. Aktas, “Natural language summarization enables multi-repository bug localization by llms in microservice architectures,” 2025. [Online]. Available: https://arxiv.org/abs/2512.05908

    2025

  23. [29]

    Human or llm? a comparative study on accessible code generation capability,

    H. Suh, M. Tafreshipour, S. Malek, and I. Ahmed, “Human or llm? a comparative study on accessible code generation capability,” 2025. [Online]. Available: https://arxiv.org/abs/2503.15885

    2025 arXiv

  24. [30]

    Inside out: Uncovering how comment internalization steers llms for better or worse,

    A. Imani, M. Moshirpour, and I. Ahmed, “Inside out: Uncovering how comment internalization steers llms for better or worse,” 2025. [Online]. Available: https://arxiv.org/abs/2512.16790

    2025

  25. [33]

    Codebleu: a method for automatic evaluation of code synthesis,

    S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma, “Codebleu: a method for automatic evaluation of code synthesis,” 2020. [Online]. Available: https://arxiv.org/abs/2009.10297

    2020 arXiv

  26. [34]

    Codesearchnet challenge: Evaluating the state of semantic code search,

    H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “Codesearchnet challenge: Evaluating the state of semantic code search,” 2020. [Online]. Available: https://arxiv.org/abs/1909.09436

    2020 arXiv

  27. [35]

    A neural model for generating natural language summaries of program subroutines,

    A. LeClair, S. Jiang, and C. McMillan, “A neural model for generating natural language summaries of program subroutines,” inProceedings of the 41st International Conference on Software Engineering, ser. ICSE ’19. IEEE Press, 2019, p. 795–806. [Online]. Available: https://doi.o...

    2019

  28. [36]

    Joern - the bug hunter’s workbench,

    The Joern Team, “Joern - the bug hunter’s workbench,” https://joern.io/, 2026, accessed: 2026-06-29

    2026

  29. [37]

    Deep learning code fragments for code clone detection,

    M. White, M. Tufano, C. Vendome, and D. Poshyvanyk, “Deep learning code fragments for code clone detection,” in2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE), 2016, pp. 87–98

    2016

  30. [38]

    Supervised deep features for software functional clone detection by exploiting lexical and syntactical information in source code,

    H.-H. Wei and M. Li, “Supervised deep features for software functional clone detection by exploiting lexical and syntactical information in source code,” inProceedings of the 26th International Joint Conference on Artificial Intelligence, ser. IJCAI’17. AAAI Press, 2017, p. 3034–3040

    2017

  31. [39]

    A systematic literature review on source code similarity measurement and clone detection: Techniques, applications, and challenges,

    M. Zakeri-Nasrabadi, S. Parsa, M. Ramezani, C. Roy, and M. Ekhtiarzadeh, “A systematic literature review on source code similarity measurement and clone detection: Techniques, applications, and challenges,”J. Syst. Softw., vol. 204, no. C, Oct. 2023. [Online]. Available: https...

    2023

  32. [40]

    Sebyte: Scalable clone and similarity search for bytecode,

    I. Keivanloo, C. K. Roy, and J. Rilling, “Sebyte: Scalable clone and similarity search for bytecode,”Science of Computer Programming, vol. 95, pp. 426–444, 2014, special Issue on Software Clones (IWSC’12). [Online]. Available: https://www.sciencedirect.com/science/ article/pii...

    2014

  33. [41]

    DExperts: Decoding-time controlled text generation with experts and anti-experts,

    A. Liu, M. Sap, X. Lu, S. Swayamdipta, C. Bhagavatula, N. A. Smith, and Y . Choi, “DExperts: Decoding-time controlled text generation with experts and anti-experts,” inProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internat...

    2021

  34. [42]

    A contrastive framework for neural text generation,

    Y . Su, T. Lan, Y . Wang, D. Yogatama, L. Kong, and N. Collier, “A contrastive framework for neural text generation,” inProceedings of the 36th International Conference on Neural Information Processing Systems, ser. NIPS ’22. Red Hook, NY , USA: Curran Associates Inc., 2022

    2022

  35. [43]

    Summarizing source code with transferred api knowledge,

    X. Hu, G. Li, X. Xia, D. Lo, S. Lu, and Z. Jin, “Summarizing source code with transferred api knowledge,” inProceedings of the 27th International Joint Conference on Artificial Intelligence, ser. IJCAI’18. AAAI Press, 2018, p. 2269–2275

    2018

  36. [44]

    A parallel corpus of python functions and documentation strings for automated code documentation and code generation,

    A. V . M. Barone and R. Sennrich, “A parallel corpus of python functions and documentation strings for automated code documentation and code generation,” 2017. [Online]. Available: https://arxiv.org/abs/1707.02275

    2017 arXiv

  37. [45]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter,...

    2021 arXiv

  38. [46]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, “Program synthesis with large language models,” 2021. [Online]. Available: https://arxiv.org/abs/2108.07732

    2021 arXiv

  39. [47]

    Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions,

    T. Y . Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paul, S. Brunner, C. Gong, T. Hoang, A. R. Zebaze, X. Hong, W.-D. Li, J. Kaddour, M. Xu, Z. Zhang, P. Yadav, N. Jain, A. Gu, Z. Cheng, J. Liu, Q. Liu, Z. Wang, B. Hui, N. Muennighoff...

    2025 arXiv

  40. [48]

    The vault: A comprehensive multilingual dataset for advancing code understanding and generation,

    D. Nguyen, L. Nam, A. Dau, A. Nguyen, K. Nghiem, J. Guo, and N. Bui, “The vault: A comprehensive multilingual dataset for advancing code understanding and generation,” inFindings of the Association for Computational Linguistics: EMNLP 2023, H. Bouamor, J. Pino, and K. Bali, Ed...

    2023

  41. [49]

    Olmo 3: Charting a path through the model flow to lead open- source ai,

    Ai2, “Olmo 3: Charting a path through the model flow to lead open- source ai,” https://allenai.org/blog/olmo3, Nov. 2025, accessed: 2026-06- 29

    2025

  42. [50]

    Bigcodebench (hard) leaderboard,

    LM Market Cap, “Bigcodebench (hard) leaderboard,” https://lmmarketcap.com/benchmarks/bigcodebench, 2026, accessed: 2026-06-29

    2026

  43. [51]

    Docstringeval: Evaluating the effectiveness of language models for code explanation through docstring generation,

    G. Sundaram, B. Venktesh V , and S. K B, “Docstringeval: Evaluating the effectiveness of language models for code explanation through docstring generation,” in2025 International Conference on Emerging Technologies in Computing and Communication (ETCC), 2025, pp. 1–7

    2025

  44. [52]

    Best ai for reasoning in 2026,

    LLM Stats Research, “Best ai for reasoning in 2026,” https://llm-stats. com/leaderboards/best-ai-for-reasoning, 2026, accessed: 2026-06-27

    2026

  45. [53]

    Data augmentation for code translation with comparable corpora and multiple references,

    Y . Xie, A. Naik, D. Fried, and C. Rose, “Data augmentation for code translation with comparable corpora and multiple references,” in Findings of the Association for Computational Linguistics: EMNLP 2023, H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Comput...

    2023

  46. [54]

    Who evaluates the evaluators? on automatic metrics for assessing ai-based offensive code generators,

    P. Liguori, C. Improta, R. Natella, B. Cukic, and D. Cotroneo, “Who evaluates the evaluators? on automatic metrics for assessing ai-based offensive code generators,”Expert Systems with Applications, vol. 225, p. 120073, 2023. [Online]. Available: https://www.sciencedirect.com/...

    2023

  47. [55]

    Coffe: A code efficiency benchmark for code generation,

    Y . Peng, J. Wan, Y . Li, and X. Ren, “Coffe: A code efficiency benchmark for code generation,”Proc. ACM Softw. Eng., vol. 2, no. FSE, Jun. 2025. [Online]. Available: https://doi.org/10.1145/3715727

    2025 doi

  48. [56]

    Togll: Correct and strong test oracle generation with llms,

    S. B. Hossain and M. B. Dwyer, “Togll: Correct and strong test oracle generation with llms,” inProceedings of the IEEE/ACM 47th International Conference on Software Engineering, ser. ICSE ’25. IEEE Press, 2025, p. 1475–1487. [Online]. Available: https: //doi.org/10.1109/ICSE55...

    2025

  49. [57]

    CoIR: A comprehensive benchmark for code information retrieval models,

    X. Li, K. Dong, Y . Q. Lee, W. Xia, H. Zhang, X. Dai, Y . Wang, and R. Tang, “CoIR: A comprehensive benchmark for code information retrieval models,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. N...

    2025

  50. [58]

    [research paper] crolsim: Cross language software similarity detector using api documentation,

    K. W. Nafi, B. Roy, C. K. Roy, and K. A. Schneider, “[research paper] crolsim: Cross language software similarity detector using api documentation,” in2018 IEEE 18th International Working Conference on Source Code Analysis and Manipulation (SCAM), 2018, pp. 139–148

    2018

  51. [59]

    Directional diffusion-style code editing pre-training,

    Q. Liang, Z. Sun, Q. Zhu, J. Hu, Y . Zhao, Y . Chen, M. Zhu, G. Wang, and L. Zhang, “Directional diffusion-style code editing pre-training,”

  52. [60]

    Naturaledit: Code modification through direct interaction with adaptive natural language representation,

    N. Tang, D. Meininger, G. Xu, Y . Shi, Y . Huang, C. McMillan, and T. J.- J. Li, “Naturaledit: Code modification through direct interaction with adaptive natural language representation,” 2026. [Online]. Available: https://arxiv.org/abs/2510.04494

    2026

  53. [61]

    Bridging functional correctness and runtime efficiency gaps in llm-based code translation,

    L. Zhang, J. Wang, C. Hu, B. Liang, J. Li, and M. Zhang, “Bridging functional correctness and runtime efficiency gaps in llm-based code translation,” 2026. [Online]. Available: https://arxiv.org/abs/2606.17683

    2026 arXiv

  54. [62]

    The fault in our stars: Quality assessment of code generation benchmarks,

    M. L. Siddiq, S. Dristi, J. Saha, and J. C. S. Santos, “The fault in our stars: Quality assessment of code generation benchmarks,” in2024 IEEE International Conference on Source Code Analysis and Manipulation (SCAM), 2024, pp. 201–212

    2024

  55. [63]

    The proof and measurement of association between two things,

    C. Spearman, “The proof and measurement of association between two things,”The American Journal of Psychology, vol. 15, no. 1, pp. 72–101, 1904

    1904

  56. [64]

    An analysis of variance test for normality (complete samples)†,

    S. S. SHAPIRO and M. B. WILK, “An analysis of variance test for normality (complete samples)†,”Biometrika, vol. 52, no. 3-4, pp. 591–611, 12 1965. [Online]. Available: https://doi.org/10.1093/biomet/ 52.3-4.591

    1965 doi

  57. [65]

    Averaging correlations: Expected values and bias in combined pearson rs and fisher’s z transformations,

    D. M. Corey, W. P. Dunlap, and M. J. Burke, “Averaging correlations: Expected values and bias in combined pearson rs and fisher’s z transformations,”The Journal of General Psychology, vol. 125, no. 3, pp. 245–261, 1998. [Online]. Available: https: //doi.org/10.1080/00221309809595548

    1998 doi

  58. [66]

    Odds ratios and logistic regression: further examples of their use and interpretation,

    S. M. Hailpern and P. F. Visintainer, “Odds ratios and logistic regression: further examples of their use and interpretation,”Stata Journal, vol. 3, no. 3, pp. 213–225, September 2003. [Online]. Available: https://ideas.repec.org/a/tsj/stataj/v3y2003i3p213-225.html

    2003

  59. [2025]

    Available: https://arxiv.org/abs/2501.12079

    [Online]. Available: https://arxiv.org/abs/2501.12079

  60. [2026]

    Available: https://arxiv.org/abs/2512.10452

    [Online]. Available: https://arxiv.org/abs/2512.10452

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.