Pith. sign in

REVIEW 3 major objections 7 minor 69 references

MoST claims that strategy-guided code optimization improves when strategies are mined from multiple knowledge sources and transferred across languages or architectures.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 10:02 UTC pith:2BAA6HIG

load-bearing objection Solid multi-source extension of SemOpt with real gains on low-resource languages; the Codex comparison in the real-world experiment is apples-to-oranges and needs rework. the 3 major comments →

arxiv 2607.20353 v1 pith:2BAA6HIG submitted 2026-07-22 cs.SE

Multi-Source and Cross-Scenario Strategy-Guided Code Optimization

Yuwei Zhao , Qianyu Xiao , Ye Cui , Yijun Yu , Yingfei Xiong This is my paper
classification cs.SE
keywords code optimizationlarge language modelsstrategy-guided optimizationstatic analysis rulescross-language transferevidence objectsweighted clusteringprogram performance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that strategy-guided LLM code optimization can be substantially improved by drawing on more than one knowledge source and by letting optimization strategies cross scenario boundaries such as programming language and target architecture. It proposes MoST, which converts items from optimization commits and an optimization manual into uniform evidence objects, clusters them by natural-language description, transfers examples into the target scenario when direct examples are missing, and generates validated static-analysis rules that locate optimizable spots for an LLM. Compared with the prior SemOpt system on 351 historical optimization tasks, MoST reports 24.44%–180.00% more exact-match patches and 21.88%–37.50% more semantically equivalent patches across C/C++, Python, and Rust. On 15 real-world projects it reports higher maximum and average performance improvements than SemOpt and a Codex agent. The sympathetic reading is that multi-source strategy coverage plus cross-scenario rule validation is the mechanism behind these gains.

Core claim

The central claim is that the two coverage limits of strategy-guided optimization—single-source strategy libraries and scenario-bound formalization—can be overcome by representing every piece of optimization knowledge as an evidence object (a natural-language description, a before/after code example, scenario tags, and a source type), clustering across sources with source-weighted density clustering, and transferring examples into the target scenario before generating static-analysis rules. The paper argues that this pipeline lets strategies be discovered from documents as well as commits, and makes them applicable to scenarios where no direct example exists, while the functional rule valida

What carries the argument

The load-bearing object is the evidence object e=⟨P,E,T,y⟩, a uniform representation of a strategy from any source, consisting of a natural-language description, a before/after code-modification example, applicable scenario tags, and a source-type label. The self-balanced weighted clustering selects the document weight, similarity threshold, and minimum cluster size so that cross-source clustering preserves single-scenario strategy counts, yielding 356 strategy clusters. The example-transfer procedure, with three cases for exact-tag, partial, and absent target-scenario examples, plus the functional rule validation that checks match-before and reject-after behavior, is what makes cross-scenar

Load-bearing premise

The paper assumes that the single-scenario strategy-cluster counts used as the reference for hyperparameter selection are a correct, noise-free ground truth; if those counts are unreliable because of duplicate or noisy LLM-generated descriptions, the chosen weights would preserve that noise and degrade every downstream step.

What would settle it

Compute the single-scenario cluster counts K_T with a different LLM or after deduplicating descriptions; if the selected hyperparameters change materially and the exact-match and semantic-equivalence results drop by more than the reported margins, the self-balanced selection claim is undermined. Alternatively, run the same benchmark with a rule-validation step that also checks a held-out set of negative examples to see whether the match-before/reject-after check is sufficient to guarantee precision.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Multi-source strategy construction expands coverage beyond historical commits; documentation information appears in 45.73% of effective C/C++ optimization results even though it forms a tiny fraction of the evidence.
  • Cross-scenario example transfer matters most for low-resource scenarios: 75.53% of effective Python results and 90.91% of effective Rust results came from post-transfer target-scenario candidates.
  • Weighted clustering protects high-quality, low-frequency sources; removing it reduces successful optimizations by 8.97%–10.71% in the ablation.
  • Reliable rule generation is critical: disabling example transfer and validation reduces EM by 43.59%–50.00%.
  • On 15 real-world projects, MoST reports maximum performance improvements of 19.72%–717.42% and average improvements of 4.44%–258.17%, outperforming SemOpt and Codex agents.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The evidence-object representation likely generalizes to other knowledge sources such as API documentation, compiler optimization guides, or online Q&A; the current implementation ingests only one optimization manual, so the measured documentation contribution may underestimate what a richer multi-source library would yield.
  • The self-balanced selection criterion is a proxy for downstream quality; a more direct validation against pilot task outcomes, or a reference count robust to duplicate LLM-generated descriptions, might change the chosen hyperparameters.
  • If the gains are driven mainly by rule-guided localization rather than by the strength of the optimizer model, MoST should remain effective with a smaller or cheaper model—an easily testable extension of the paper's claim.
  • Cross-scenario transfer could be extended beyond language and architecture to other scenario properties such as library versions, parallelism models, or cache configurations, which the paper's scenario-tag representation already accommodates.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper presents MoST, an LLM-based code optimization framework that converts heterogeneous knowledge sources (historical commits and an Intel optimization manual) into uniform evidence objects, clusters them across sources and scenarios using a self-balanced weighted clustering algorithm, transfers examples to target scenarios when necessary, and generates validated Semgrep rules to guide LLM optimization. The evaluation has two parts: reproduction of 351 historical optimization tasks in C/C++, Python, and Rust, and optimization of 15 real-world projects. The authors report that MoST exceeds SemOpt on historical-task EM/SemEqv by 24.44%–180.00% and 21.88%–37.50% respectively, and that on real-world projects it achieves larger Max/Avg performance improvements than SemOpt and Codex, with ablations and a provenance analysis supporting the design choices.

Significance. MoST addresses two genuine limitations of prior strategy-guided optimizers: it broadens the knowledge sources beyond commits and enables cross-scenario (e.g., cross-language) strategy transfer. The evaluation is broad and mostly careful: three languages, a new Rust benchmark, three baselines, an ablation of both core components, a pilot-based parameter selection, statistical significance checks for performance measurements, and available artifacts. If the claimed results hold, the work is a solid advance for LLM-based code optimization. My main reservation is that the headline comparison with Codex is based on an asymmetric evaluation protocol; a second, lesser concern is that the clustering hyperparameters are selected on the same corpus used for evaluation. Both are addressable and do not, in my view, invalidate the core ideas.

major comments (3)
  1. [V-A5, Table IV, RQ4] The real-world comparison between MoST/SemOpt and Codex Agent is asymmetric. For MoST and SemOpt, the reported Max/Avg and #≥5%/#≥10% are computed from a final project version assembled by (i) generating candidate optimizations for hotspot functions, (ii) retaining only candidates that improve at least one data point by >5% and degrade none by >2%, and (iii) for each hotspot, keeping the eligible variant with the highest total improvement. Codex Agent, however, contributes a single project patch that is evaluated as that result ('Codex Agent's project patch is evaluated as that result'). Thus the comparison is best-of-many (filtered and composed) versus one-shot. The enormous relative Avg improvements over Codex (up to 27,288.89% per the RQ4 answer) are consistent with this asymmetry. To support the claim that MoST 'significantly outperforms Codex,' the authors must run Codex through the
  2. [III-B (Eqs. 3–6), IV-C] The self-balanced clustering parameters (w_d=3, τ=0.76, s_min=5) are chosen by minimizing the balance loss L(θ) on the full evidence corpus, and the historical evaluation tasks are drawn from that same corpus (with exact commit/code matches excluded). The 45-task pilot set is disjoint from the formal benchmark and provides supporting evidence that the criterion tracks EM, but the pilot is small and the parameter search is still on the same data distribution as the evaluation. Because the ablation in Table III attributes only 6 EM points (50 vs 56) to weighted clustering, the overall comparative claims are unlikely to be overturned by a different θ; nevertheless, the paper should report a sensitivity analysis over the configurations in Table II for the main benchmark (or at least for a subset), to demonstrate that the reported improvements are not tied to this particular selection.
  3. [IV-A5, Fig. 3] Success is defined as 'at least one of three runs' satisfying EM/SemEqv. All LLM experiments are run at temperature 0, so the source of run-to-run variation should be clarified; if the three runs are not truly independent, the definition is ambiguous. Even if this protocol is inherited from SemOpt, the reported relative improvements are computed from counts that can be sensitive to a single run (e.g., small-count settings such as Rust with 14 vs 9 successes). I recommend reporting the distribution (e.g., mean±std over the three runs, or majority success) and the per-run rates, so that the stability of the claimed percentages can be assessed. This does not necessarily bias the comparison against baselines, but it is load-bearing for the precise effect sizes in the abstract.
minor comments (7)
  1. [V-A2] The fallback for Rust SemOpt when no candidate passes the >5% filter should be described more precisely; if SemOpt is not given the same composition opportunities as MoST in such cases, the Rust comparison may still favor MoST.
  2. [IV-D] The manual applicability evaluation (89.70%) should report how many suggestions were inspected, the criteria for 'valuable', and inter-rater agreement; comparing to SemOpt's 89.86% with a different LLM (DeepSeek-V3) is not apples-to-apples.
  3. [III-A, Table V] Only 189 document-derived evidence objects are used versus 48,440 commit-derived objects; the claim that documentation contributes 45.73% of effective C/C++ results should be accompanied by counts/confidence intervals, since the absolute numbers are small.
  4. [V-C] RQ5 counts multiple effective results from the same hotspot/rule; please clarify whether the counts are at the level of data points and report unique hotspot/rule counts as a sanity check.
  5. [VI] The data-leakage mitigation excludes only exact commit/code matches; near-duplicate versions of the same optimization could still appear in the strategy library. A similarity-based deduplication or an additional leakage analysis would strengthen the threat-to-validity discussion.
  6. [Table IV] Some entries are hard to parse because spaces are missing between Avg and #≥5%/#≥10% columns (e.g., RocksDB). Please format the table so each column is clearly separated.
  7. [Abstract and references] Typos: 'otherknowledge' in the abstract should be 'other knowledge'; reference [55] URL contains a space in 'line profiler'; 'super parameters' in Section II-C should be 'hyperparameters'.

Circularity Check

0 steps flagged

No significant circularity: the central claims are empirical comparisons, and no equation reduces to a fitted value or to a self-citation chain.

full rationale

The paper's central claims are empirical comparisons on externally defined benchmarks (151 C/C++, 150 Python, and 50 Rust historical tasks; 15 real-world projects). The only parameter-selection mechanism, Eqs. (3)-(6), defines a clustering-only balance loss used to choose (document weight, threshold, min size); it does not produce the reported EM/SemEqv or performance numbers, and the selected configuration is checked against a disjoint 45-task pilot set (Table II) before application. No fitted parameter is renamed as a prediction, and no evaluation metric enters the selection loss. Self-citations to SemOpt [15] are used as a baseline and benchmark source, but the MoST-vs-SemOpt results are re-run in this paper rather than imported; the claim that RAPGen and Clang-Tidy are weaker is a published, externally falsifiable comparison and is not load-bearing for the main MoST-vs-SemOpt/Codex results. The real-world evaluation asymmetry (filtered best-of-many composition for MoST/SemOpt versus single-patch Codex) is a protocol-fairness threat, not a circularity of derivation. Therefore no step reduces to its own inputs.

Axiom & Free-Parameter Ledger

8 free parameters · 6 axioms · 1 invented entities

The framework's central claim rests on assumptions about LLM reliability, clustering validity, and evaluation protocol rather than on mathematical derivation. The most fragile is that the balance-loss reference counts come from single-scenario clustering.

free parameters (8)
  • document weight w_d = 3
    Chosen by the self-balanced criterion from a candidate set; balances commit vs document evidence in clustering.
  • description-similarity threshold tau = 0.76
    Chosen by the self-balanced criterion; determines which evidence objects merge into the same strategy cluster.
  • minimum cluster size s_min = 5
    Chosen by the self-balanced criterion; minimum weighted evidence support to retain a cluster.
  • loss hyperparameters epsilon, lambda = 1, 1
    Set by hand in the balance loss (Eq. 4); no sensitivity analysis is provided.
  • rule-generation budget b = 5
    Inherited from the SemOpt setting; number of Semgrep rules generated per example.
  • top N retained pairs = 25
    Phase 4 keeps the top 25 matched location–strategy pairs for LLM optimization.
  • hotspot threshold = 0.1% of runtime
    Functions whose execution time exceeds 0.1% of total runtime are treated as hotspots in the real-project experiment.
  • composition eligibility thresholds = >5% improvement, <=2% degradation
    Candidates are eligible for composition only if they improve at least one data point by more than 5% and degrade no other data point by more than 2%.
axioms (6)
  • domain assumption LLM conversion of documents/commits into evidence objects is faithful (Phase 1).
    Section III-A relies on DeepSeek-V4-Pro to summarize NL descriptions, normalize code snippets, and assign scenario tags; errors propagate to clustering and rules.
  • domain assumption Weighted density clustering on NL descriptions groups semantically identical strategies across sources (Phase 2).
    Section III-B assumes that semantic similarity of descriptions aligns with strategy identity, which is not formally guaranteed.
  • domain assumption Single-scenario cluster counts K_T are a stable reference for cross-scenario balance (Eq. 3).
    The self-balanced criterion preserves counts from single-scenario clustering; if those counts are noisy, the optimization is misguided.
  • domain assumption The LLM's cross-scenario applicability check is reliable (Phase 3, case 3).
    When a cluster has no exact-tag evidence, an LLM decides whether the strategy applies to the target scenario; a wrong decision either wastes budget or blocks a useful strategy.
  • domain assumption Rule validation on a single before/after example is sufficient for rule quality (Phase 3).
    The paper checks that a rule matches the before-example and rejects the after-example; passing one example does not imply precision on unseen code.
  • domain assumption EM, SemEqv, and the project performance protocol capture optimization quality (Sections IV-A4, V-A4).
    Exact and semantic match to historical developer patches are proxies for correctness; performance tests may not represent real workloads.
invented entities (1)
  • Evidence object no independent evidence
    purpose: Uniform representation of strategies from heterogeneous sources (description, example, scenario tags, source type).
    This is a paper-internal data structure; no external handle is provided beyond the end-to-end evaluation.

pith-pipeline@v1.3.0-alltime-deepseek · 18568 in / 13750 out tokens · 106297 ms · 2026-08-01T10:02:36.441830+00:00 · methodology

0 comments
read the original abstract

Automated code optimization improves program performance by refactoring source code, and recent studies use LLMs to generate optimization patches. The newest approaches are strategy-guided: they summarize strategies from historical optimization commits as static analysis rules, and use these rules to match code locations for LLMs to optimize. However, these approaches have two limitations: (1) the strategies may come from other knowledge sources, such as textbooks and web pages, but the existing approaches cannot utilize them; (2) a strategy may be applicable to different scenarios, e.g., different programming languages, but existing approaches can only formalize strategies for the scenario to which the source commit belongs. To address these limitations, we propose MoST, an LLM-based code optimization framework that integrates multiple knowledge sources across scenarios. MoST uniformly represents items in different knowledge sources as evidence objects, clusters them in a cross-source and cross-scenario manner to identify strategies, and transfers them to the target scenario when necessary for generating static analysis rules. To implement this process, MoST employs a novel self-balanced weighted clustering algorithm to balance evidence objects from different knowledge sources, and a novel example transfer procedure to ensure the quality of the generated rules when transferring across scenarios. On a benchmark containing 151 C/C++, 150 Python, and 50 Rust historical optimization tasks, compared with SemOpt, MoST yields 24.44%-180.00% and 21.88%-37.50% more patches that are exactly the same as or semantically equivalent to developer patches, respectively. When optimizing 15 real-world projects, MoST achieves 19.72%-717.42% maximum improvements and 4.44%-258.17% average improvements for the performance tests in the projects, significantly outperforming SemOpt and Codex.

Figures

Figures reproduced from arXiv: 2607.20353 by Qianyu Xiao, Ye Cui, Yijun Yu, Yingfei Xiong, Yuwei Zhao.

Figure 1
Figure 1. Figure 1: A document-sourced, cross-scenario motivating example. The Intel optimization document provides excerpted strategy [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The overall workflow of MoST. It first constructs evidence objects from multiple sources, then discovers strategy [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance comparison across languages and models. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

69 extracted references · 1 canonical work pages

  1. [1]

    ISO/IEC 25010:2011 Systems and software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – System and software quality models,

    ISO/IEC, “ISO/IEC 25010:2011 Systems and software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – System and software quality models,” 2011, accessed: 2026-07-01. [Online]. Available: https://www.iso.org/standard/35733.html

    2011

  2. [2]

    Discovering, reporting, and fixing per- formance bugs,

    A. Nistor, T. Jiang, and L. Tan, “Discovering, reporting, and fixing per- formance bugs,” in2013 10th Working Conference on Mining Software Repositories (MSR). IEEE, 2013, pp. 237–246

    2013

  3. [3]

    Catch me if you can: performance bug detection in the wild,

    M. Jovic, A. Adamoli, and M. Hauswirth, “Catch me if you can: performance bug detection in the wild,” inProceedings of the 2011 ACM International Conference on Object Oriented Programming Systems Languages and Applications, 2011, pp. 155–170

    2011

  4. [4]

    PerfScope: Practical online server performance bug inference in production cloud computing infrastructures,

    D. J. Dean, H. Nguyen, X. Gu, H. Zhang, J. Rhee, N. Arora, and G. Jiang, “PerfScope: Practical online server performance bug inference in production cloud computing infrastructures,” inProceedings of the ACM Symposium on Cloud Computing, 2014, pp. 1–13

    2014

  5. [5]

    Peephole optimization,

    W. M. McKeeman, “Peephole optimization,”Communications of the ACM, vol. 8, no. 7, pp. 443–444, 1965

    1965

  6. [6]

    Global common subexpression elimination,

    J. Cocke, “Global common subexpression elimination,” inProceedings of a Symposium on Compiler Optimization, 1970, pp. 20–24

    1970

  7. [7]

    CodeGen: An open large language model for code with multi-turn program synthesis,

    E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y . Zhou, S. Savarese, and C. Xiong, “CodeGen: An open large language model for code with multi-turn program synthesis,” inInternational Conference on Learning Representations (ICLR), 2023, arXiv:2203.13474. [Online]. Available: https://openreview.net/forum?id=iaYcJKpY2B

    Pith/arXiv arXiv 2023

  8. [8]

    StarCoder: may the source be with you!

    R. Li, L. B. Allal, Y . Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chimet al., “StarCoder: may the source be with you!”arXiv preprint arXiv:2305.06161, 2023. [Online]. Available: https://arxiv.org/abs/2305.06161

    Pith/arXiv arXiv 2023

  9. [9]

    DeepSeek-Coder: When the large language model meets programming – the rise of code intelligence,

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . K. Li, F. Luo, Y . Xiong, and W. Liang, “DeepSeek-Coder: When the large language model meets programming – the rise of code intelligence,”arXiv preprint arXiv:2401.14196, 2024. [Online]. Available: https://arxiv.org/abs/2401.14196

    Pith/arXiv arXiv 2024

  10. [10]

    Large language models for software engineering: Sur- vey and open problems,

    A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Sur- vey and open problems,” in2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). IEEE, 2023, pp. 31–53

    2023

  11. [11]

    Learning performance-improving code edits,

    A. Shypula, A. Madaan, Y . Zeng, U. Alon, J. R. Gardner, Y . Yang, M. Hashemi, G. Neubig, P. Ranganathan, O. Bastani, and A. Yazdanbakhsh, “Learning performance-improving code edits,” in The Twelfth International Conference on Learning Representations (ICLR), 2024, arXiv:2302.07867. [Online]. Available: https://openrevi ew.net/forum?id=ix7rLVHXyY

    Pith/arXiv arXiv 2024

  12. [12]

    RAPGen: An approach for fixing code inefficiencies in zero-shot,

    S. Garg, R. Z. Moghaddam, and N. Sundaresan, “RAPGen: An approach for fixing code inefficiencies in zero-shot,” in2025 IEEE/ACM 47th In- ternational Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 2025, pp. 124–135, arXiv:2306.17077

    Pith/arXiv arXiv 2025

  13. [13]

    Search-based LLMs for code optimization,

    S. Gao, C. Gao, W. Gu, and M. R. Lyu, “Search-based LLMs for code optimization,” in2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), 2025, pp. 578–590, arXiv:2408.12159

    Pith/arXiv arXiv 2025

  14. [14]

    PerfCodeGen: Improving performance of LLM generated code with execution feedback,

    Y . Peng, A. D. Gotmare, M. R. Lyu, C. Xiong, S. Savarese, and D. Sahoo, “PerfCodeGen: Improving performance of LLM generated code with execution feedback,” in2025 IEEE/ACM Second International Conference on AI Foundation Models and Software Engineering (Forge). IEEE, 2025, pp. 1–13

    2025

  15. [15]

    SemOpt: LLM- driven code optimization via rule-based analysis,

    Y . Zhao, Y .-A. Xiao, Q. Xiao, Z. Zhang, and Y . Xiong, “SemOpt: LLM- driven code optimization via rule-based analysis,”ACM Transactions on Software Engineering and Methodology, 2026

    2026

  16. [16]

    Semgrep,

    Semgrep, “Semgrep,” 2026, accessed: 2026-07-01. [Online]. Available: https://github.com/semgrep/semgrep

    2026

  17. [17]

    Intel® 64 and IA-32 Architectures Optimization Reference Manual V olume 1,

    Intel Corporation, “Intel® 64 and IA-32 Architectures Optimization Reference Manual V olume 1,” 2023, version 050; Accessed: 2026-07-

    2023

  18. [18]

    Available: https://www.intel.com/content/www/us/en/cont ent-details/671488/intel-64-and-ia-32-architectures-optimization-refer ence-manual-volume-1.html

    [Online]. Available: https://www.intel.com/content/www/us/en/cont ent-details/671488/intel-64-and-ia-32-architectures-optimization-refer ence-manual-volume-1.html

  19. [19]

    A density-based algorithm for discovering clusters in large spatial databases with noise,

    M. Ester, H.-P. Kriegel, J. Sander, and X. Xu, “A density-based algorithm for discovering clusters in large spatial databases with noise,” inProceedings of the Second International Conference on Knowledge Discovery and Data Mining, 1996, pp. 226–231. [Online]. Available: https://aaai.org/papers/KDD96-037-a-density-based-algorithm-for-dis covering-clusters...

    1996

  20. [20]

    Random search for hyper-parameter optimization,

    J. Bergstra and Y . Bengio, “Random search for hyper-parameter optimization,”Journal of Machine Learning Research, vol. 13, no. 10, pp. 281–305, 2012. [Online]. Available: https://jmlr.org/papers/v13/be rgstra12a.html

    2012

  21. [21]

    The statistical analysis of compositional data,

    J. Aitchison, “The statistical analysis of compositional data,”Journal of the Royal Statistical Society: Series B (Methodological), vol. 44, no. 2, pp. 139–160, 1982

    1982

  22. [22]

    KNighter: Transforming static analysis with LLM-synthesized checkers,

    C. Yang, Z. Zhao, Z. Xie, H. Li, and L. Zhang, “KNighter: Transforming static analysis with LLM-synthesized checkers,” inProceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, 2025, pp. 655–669

    2025

  23. [23]

    Write your own CodeChecker: An automated test-driven checker development approach with LLMs,

    J. Liu, Y . Xie, J. Yan, J. Huang, J. Yan, and J. Zhang, “Write your own CodeChecker: An automated test-driven checker development approach with LLMs,” inIEEE/ACM International Conference on Software Engineering (ICSE), 2026, arXiv:2411.06796. [Online]. Available: https://arxiv.org/abs/2411.06796

    Pith/arXiv arXiv 2026

  24. [24]

    Neuro-symbolic static analysis with LLM-generated vulnerability patterns,

    P. Li, S. Yao, J. S. Korich, C. Luo, J. Yu, Y . Cao, and J. Yang, “Neuro-symbolic static analysis with LLM-generated vulnerability patterns,”arXiv preprint arXiv:2504.16057, 2025. [Online]. Available: https://arxiv.org/abs/2504.16057

    Pith/arXiv arXiv 2025

  25. [25]

    What makes good in-context demonstrations for code intelligence tasks with LLMs?

    S. Gao, X.-C. Wen, C. Gao, W. Wang, H. Zhang, and M. R. Lyu, “What makes good in-context demonstrations for code intelligence tasks with LLMs?” in2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2023, pp. 761–773

    2023

  26. [26]

    Clang-Tidy,

    LLVM Project, “Clang-Tidy,” 2026, accessed: 2026-07-01. [Online]. Available: https://clang.llvm.org/extra/clang-tidy/

    2026

  27. [27]

    Models & Pricing: DeepSeek-V4-Pro,

    DeepSeek, “Models & Pricing: DeepSeek-V4-Pro,” 2026, accessed: 2026-07-01. [Online]. Available: https://api-docs.deepseek.com/quick s tart/pricing

    2026

  28. [28]

    GPT-5.2 Model,

    OpenAI, “GPT-5.2 Model,” 2026, accessed: 2026-07-01. [Online]. Available: https://developers.openai.com/api/docs/models/gpt-5.2

    2026

  29. [29]

    RocksDB,

    Facebook, “RocksDB,” 2026, accessed: 2026-07-01. [Online]. Available: https://github.com/facebook/rocksdb

    2026

  30. [30]

    [Online]

    Redis, “Redis,” 2026, accessed: 2026-07-01. [Online]. Available: https://github.com/redis/redis

    2026

  31. [31]

    [Online]

    gRPC Authors, “gRPC,” 2026, accessed: 2026-07-01. [Online]. Available: https://github.com/grpc/grpc

    2026

  32. [32]

    LevelDB,

    Google, “LevelDB,” 2026, accessed: 2026-07-01. [Online]. Available: https://github.com/google/leveldb

    2026

  33. [33]

    [Online]

    Gabi Melman, “spdlog,” 2026, accessed: 2026-07-01. [Online]. Available: https://github.com/gabime/spdlog

    2026

  34. [34]

    Click Documentation,

    Click Project, “Click Documentation,” 2026, accessed: 2026-07-01. [Online]. Available: https://click.palletsprojects.com/

    2026

  35. [35]

    Flask Documentation,

    Flask Project, “Flask Documentation,” 2026, accessed: 2026-07-01. [Online]. Available: https://flask.palletsprojects.com/

    2026

  36. [36]

    Jinja Documentation,

    Jinja Project, “Jinja Documentation,” 2026, accessed: 2026-07-01. [Online]. Available: https://jinja.palletsprojects.com/

    2026

  37. [37]

    Requests,

    Python Software Foundation, “Requests,” 2026, accessed: 2026-07-01. [Online]. Available: https://github.com/psf/requests

    2026

  38. [38]

    [Online]

    Scrapy, “Scrapy,” 2026, accessed: 2026-07-01. [Online]. Available: https://github.com/scrapy/scrapy

    2026

  39. [39]

    [Online]

    Dimforge, “Rapier,” 2026, accessed: 2026-07-01. [Online]. Available: https://github.com/dimforge/rapier

    2026

  40. [40]

    [Online]

    image-rs, “image,” 2026, accessed: 2026-07-01. [Online]. Available: https://github.com/image-rs/image

    2026

  41. [41]

    [Online]

    Rust Project Developers, “regex,” 2026, accessed: 2026-07-01. [Online]. Available: https://github.com/rust-lang/regex

    2026

  42. [42]

    datafusion-sqlparser-rs,

    Apache Software Foundation, “datafusion-sqlparser-rs,” 2026, accessed: 2026-07-01. [Online]. Available: https://github.com/apache/datafusion-s qlparser-rs

    2026

  43. [43]

    [Online]

    Tokio Contributors, “Tokio,” 2026, accessed: 2026-07-01. [Online]. Available: https://github.com/tokio-rs/tokio

    2026

  44. [44]

    [Online]

    OpenAI, “Codex,” 2026, accessed: 2026-07-01. [Online]. Available: https://developers.openai.com/codex

    2026

  45. [45]

    Codex Models: gpt-5.3-codex-spark,

    OpenAI Codex, “Codex Models: gpt-5.3-codex-spark,” 2026, accessed: 2026-07-01. [Online]. Available: https://developers.openai.com/codex/ models

    2026

  46. [46]

    GPT-5.4 mini Model,

    OpenAI API, “GPT-5.4 mini Model,” 2026, accessed: 2026-07-01. [Online]. Available: https://developers.openai.com/api/docs/models/gp t-5.4-mini

    2026

  47. [47]

    GPT-5.4 Model,

    OpenAI, “GPT-5.4 Model,” 2026, accessed: 2026-07-01. [Online]. Available: https://developers.openai.com/api/docs/models/gpt-5.4

    2026

  48. [48]

    The generalization of Student’s problem when several different population variances are involved,

    B. L. Welch, “The generalization of Student’s problem when several different population variances are involved,”Biometrika, vol. 34, no. 1-2, pp. 28–35, 1947

    1947

  49. [49]

    Robust benchmarking in noisy environments,

    J. Chen and J. Revels, “Robust benchmarking in noisy environments,” arXiv preprint arXiv:1608.04295, 2016. [Online]. Available: https: //arxiv.org/abs/1608.04295

    Pith/arXiv arXiv 2016

  50. [50]

    Minotaur: A SIMD-oriented syn- thesizing superoptimizer,

    Z. Liu, S. Mada, and J. Regehr, “Minotaur: A SIMD-oriented syn- thesizing superoptimizer,”Proceedings of the ACM on Programming Languages, vol. 8, no. OOPSLA2, pp. 1561–1585, 2024

    2024

  51. [51]

    Unveiling overlooked performance variance in serverless computing,

    J. Wen, Z. Chen, F. Sarro, and S. Wang, “Unveiling overlooked performance variance in serverless computing,”Empirical Software Engineering, vol. 30, no. 2, p. 59, 2025

    2025

  52. [52]

    On measuring large language models performance with inferential statistics,

    J. M. Fraile-Hern ´andez and A. Pe ˜nas, “On measuring large language models performance with inferential statistics,”Information, vol. 16, no. 9, p. 817, 2025

    2025

  53. [53]

    perf: Linux profiling with performance counters,

    Linux Kernel Organization, “perf: Linux profiling with performance counters,” 2026, accessed: 2026-07-01. [Online]. Available: https: //perfwiki.github.io/

    2026

  54. [54]

    [Online]

    Free Software Foundation, “Gcov,” 2026, accessed: 2026-07-01. [Online]. Available: https://gcc.gnu.org/onlinedocs/gcc/Gcov.html

    2026

  55. [55]

    The Python profilers,

    Python Software Foundation, “The Python profilers,” 2026, accessed: 2026-07-01. [Online]. Available: https://docs.python.org/3/library/profil e.html

    2026

  56. [56]

    line profiler,

    pyutils, “line profiler,” 2026, accessed: 2026-07-01. [Online]. Available: https://github.com/pyutils/line profiler

    2026

  57. [57]

    Performance problems you can fix: A dynamic analysis of memoization opportunities,

    L. Della Toffola, M. Pradel, and T. R. Gross, “Performance problems you can fix: A dynamic analysis of memoization opportunities,”ACM SIGPLAN Notices, vol. 50, no. 10, pp. 607–622, 2015

    2015

  58. [58]

    CADET: Debugging and fixing misconfigurations using counterfactual reasoning,

    R. Krishna, M. S. Iqbal, M. A. Javidian, B. Ray, and P. Jamshidi, “CADET: Debugging and fixing misconfigurations using counterfactual reasoning,”arXiv preprint arXiv:2010.06061, 2020. [Online]. Available: https://arxiv.org/abs/2010.06061

    Pith/arXiv arXiv 2010

  59. [59]

    Genetic optimisation of C++ applications,

    R. Giavrimis, A. Butler, C. C. Petrescu, M. Basios, and S. K. Dash, “Genetic optimisation of C++ applications,” in2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2021, pp. 1180–1182

    2021

  60. [60]

    CLion: A cross-platform IDE for C and C++,

    JetBrains, “CLion: A cross-platform IDE for C and C++,” 2026, accessed: 2026-07-01. [Online]. Available: https://www.jetbrains.com/ clion/

    2026

  61. [61]

    Learning to improve code efficiency,

    B. Chen, D. Tarlow, K. Swersky, M. Maas, P. Heiber, A. Naik, M. Hashemi, and P. Ranganathan, “Learning to improve code efficiency,”arXiv preprint arXiv:2208.05297, 2022. [Online]. Available: https://arxiv.org/abs/2208.05297

    Pith/arXiv arXiv 2022

  62. [62]

    Supersonic: Learning to generate source code optimizations in C/C++,

    Z. Chen, S. Fang, and M. Monperrus, “Supersonic: Learning to generate source code optimizations in C/C++,”IEEE Transactions on Software Engineering, vol. 50, no. 11, pp. 2849–2864, 2024

    2024

  63. [63]

    DeepDev-PERF: a deep learning-based approach for improving software performance,

    S. Garg, R. Z. Moghaddam, C. B. Clement, N. Sundaresan, and C. Wu, “DeepDev-PERF: a deep learning-based approach for improving software performance,” inProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022, pp. 948–958

    2022

  64. [64]

    Knowledge transfer from high-resource to low-resource programming languages for code LLMs,

    F. Cassano, J. Gouwar, F. Lucchetti, C. Schlesinger, A. Freeman, C. J. Anderson, M. Q. Feldman, M. Greenberg, A. Jangda, and A. Guha, “Knowledge transfer from high-resource to low-resource programming languages for code LLMs,”Proceedings of the ACM on Programming Languages, vol. 8, no. OOPSLA2, pp. 677–708, 2024

    2024

  65. [65]

    Enhancing code gener- ation for low-resource languages: No silver bullet,

    A. Giagnorio, A. Martin-Lopez, and G. Bavota, “Enhancing code gener- ation for low-resource languages: No silver bullet,” in2025 IEEE/ACM 33rd International Conference on Program Comprehension (ICPC), 2025, pp. 478–488, arXiv:2501.19085

    Pith/arXiv arXiv 2025

  66. [66]

    A survey on LLM-based code gen- eration for low-resource and domain-specific programming languages,

    S. Joel, J. J. W. Wu, and F. H. Fard, “A survey on LLM-based code gen- eration for low-resource and domain-specific programming languages,” ACM Transactions on Software Engineering and Methodology, 2025

    2025

  67. [67]

    EffiCoder: Enhancing code generation in large language models through efficiency-aware fine- tuning,

    D. Huang, G. Zeng, J. Dai, M. Luo, H. Weng, Y . Qing, H. Cui, Z. Guo, and J. M. Zhang, “EffiCoder: Enhancing code generation in large language models through efficiency-aware fine- tuning,”arXiv preprint arXiv:2410.10209, 2024. [Online]. Available: https://arxiv.org/abs/2410.10209

    Pith/arXiv arXiv 2024

  68. [68]

    PEACE: Towards efficient project-level efficiency optimization via hybrid code editing,

    X. Ren, J. Wan, Y . Peng, Z. Liu, M. Liang, D. Chen, W. Jiang, and Y . Li, “PEACE: Towards efficient project-level efficiency optimization via hybrid code editing,” in2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2025, pp. 1831– 1843

    2025

  69. [69]

    POLO: An LLM-powered project-level code performance optimization framework,

    J. Bai, R. Xu, S. Wu, D. Yang, J. Zhao, and G. Chen, “POLO: An LLM-powered project-level code performance optimization framework,” inProceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25. International Joint Conferences on Artificial Intelligence Organization, 2025, pp. 7319–7328. [Online]. Available: https:/...

    work page doi:10.24963/ijcai.2025/814 2025