Pith. sign in

REVIEW 5 major objections 6 minor 60 references

AdaptiveLLM: A Framework for Selecting Optimal Cost-Efficient LLM for Code-Generation Based on CoT Length

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Chain-of-thought length can be the automatic difficulty signal that routes a coding task to the cheapest language model able to solve it.

desk verdict CoT-length routing idea is fresh, but the training labels in Eq. (1) can pick failing cheap models over passing expensive ones, so the empirical claims need a careful re-run before trusting them. read the letter →

arxiv 2506.10525 v1 pith:XE4OSSO3 submitted 2025-06-12 cs.SE

Junhang Cheng , Fang Liu , Chengru Wu , Li Zhang This is my paper
classification cs.SE
keywords largelanguagemodelselectionchain-of-thoughtlengthcodegenerationdifficultyassessmentXGBoostclassifierBERTfine-tuningcost-efficientinferencepass@1
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AdaptiveLLM claims that the length of a reasoning model's chain of thought - the step-by-step text produced before the final answer - is a dependable automatic measure of how hard a coding problem is, and that a router built on this signal can pick the cheapest code LLM that will actually solve each problem. The framework clusters CoT lengths into three difficulty bands, fine-tunes CodeBERT to encode those bands into problem embeddings, and trains an XGBoost classifier to select among eight code models per problem. On three coding benchmarks it reports a 7.86% higher pass@1 than the ComplexityNet baseline while cutting inference cost by about 89%, and it argues the CoT signal reflects LLM difficulty better than human difficulty labels. If correct, this would remove the need for human-annotated difficulty tags in model routing and let practitioners save most of their inference budget without sacrificing code-generation accuracy.

What carries the argument

The load-bearing object is the scoring identity that defines ground-truth optimal-model labels: $Score_i = \log(\max_j Tokens_j \times MaxPrice) \times pass_i - \log(Tokens_i \times Price_i)$, where $pass_i$ is the fraction of a model's five responses that pass all tests and $Tokens_i \times Price_i$ is the cost penalty. Around it, the framework is a pipeline: median CoT length per problem, k-means clustering into three difficulty clusters, triplet-contrastive fine-tuning of CodeBERT, then an XGBoost classifier over the resulting embeddings. The CoT length does the work of replacing human difficulty annotations, and the score formula does the work of defining the optimal-model label for supervised training.

What would settle it

Using the released dataset, recompute the Eq. (1) label for every problem and check how often the argmax model passes at least one test; a high fraction of failing 'optimal' models would invalidate the training labels. A cleaner falsification is to retrain the same XGBoost router with labels defined as the cheapest model that passes at least one test, and see whether the 7.86% pass@1 advantage over ComplexityNet survives.

Watch

Extended reading notes

Core claim

The paper's central discovery is that chain-of-thought length, measured as the median length of ten reasoning traces from DeepSeek-R1-Distill-Qwen-32B, tracks how well code LLMs will actually perform on a problem, while human difficulty labels (acceptance rate, contest rating) do not. It operationalizes this by k-means clustering CoT lengths into three difficulty levels, using triplet contrastive loss to fine-tune CodeBERT so problem embeddings carry difficulty information, and training an XGBoost classifier to map each embedding to the model that maximizes $Score_i = \log(\max_j Tokens_j \times MaxPrice) \times pass_i - \log(Tokens_i \times Price_i)$. Reported results include pass@1 of 44.94% versus 37.08% for ComplexityNet, with inference cost reduced by 88.3% (pass@1) to 88.9% (pass@5), and an ablation showing that the difficulty-aware fine-tuning contributes 1.12 pass@1 points.

Load-bearing premise

The ground-truth 'optimal model' for each problem is whatever maximizes the score formula, and that formula never precisely defines the pass-rate variable and never disqualifies models that fail every test, so a cheap failing model can outrank a costlier model that actually solves the problem; every reported gain inherits whatever distortion this label choice introduces.

Editorial extensions

If this is right

  • A router built on CoT length can be deployed on unlabeled coding problems without human difficulty annotations, because the reasoning model is invoked only during training.
  • On the three evaluated benchmarks, replacing ComplexityNet with AdaptiveLLM would raise pass@1 by 7.86 points while cutting inference cost by roughly 89% on the same candidate pool.
  • The ablation shows that difficulty-aware CodeBERT fine-tuning alone adds 1.12 pass@1 points, confirming that embedding CoT-derived difficulty labels into problem vectors improves routing choices.
  • Because AdaptiveLLM needs only the problem text at decision time, it avoids the repeated LLM invocations of cascading or ensembling routers and makes a single call to the selected model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A head-to-head comparison of routers trained on CoT-derived labels versus human difficulty labels on the same eight-model pool would directly test the paper's claim that CoT is 'more reliable'; the current box-plot evidence is suggestive but not a routing experiment.
  • Recomputing ground-truth labels with explicit handling of models that fail every test could change the measured 7.86% advantage, since the published score formula never excludes failing models; a corrected label definition (cheapest model that passes at least one test) is a straightforward robustness check.
  • The same CoT-length signal could extend to other reasoning-heavy generation tasks, such as math word problems or code repair, though the fixed three-cluster quantization would likely need per-domain recalibration.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes AdaptiveLLM, a framework for routing code-generation problems to one of eight open-source LLMs. Problem difficulty is estimated automatically from the length of chain-of-thought generated by DeepSeek-R1-Distill-Qwen-32B, clustered into three levels via k-means; CodeBERT is fine-tuned with triplet loss to produce difficulty-aware embeddings, and an XGBoost classifier is trained to select the optimal model per problem. The optimal model labels are derived from a scoring formula that trades pass rate against token count and price. Experiments on HumanEval, LeetCodeSample, and CodeContests report a 7.86% pass@1 improvement over the ComplexityNet baseline with an 88.9% cost reduction, and an ablation shows a small gain from the CodeBERT fine-tuning.

Significance. If the central result holds, AdaptiveLLM would be a practical contribution: it replaces human difficulty annotations with an automated CoT-length signal for LLM routing, and it demonstrates that a routing framework can simultaneously improve code-generation accuracy and reduce inference cost. The paper ships a replication package, which is a strength, and the idea of validating difficulty labels against LLM behavior rather than human judgment is worth pursuing. However, the current evidence is weakened by a ground-truth label formula that can select failing models as optimal, an unstated normalization assumption in the contrastive loss, the absence of error bars or significance tests, and a partially self-referential difficulty validation. These issues are fixable, but they make the reported improvements not yet convincing.

major comments (5)
  1. [Section 4.1.2, Eq. (1)] The ground-truth label formula is underspecified: pass_i is described only as "the proportion of the model that passes all the test cases in five responses," which does not clarify whether it is a fraction over responses, over test cases, or over both. More importantly, the formula does not exclude models with pass_i = 0. A model with zero passing responses receives Score_i = -log(Tokens_i * Price_i), which can exceed the score of a more expensive model with pass_i > 0. Using the aggregate values in Table 4, a 1.5B model at $0.14/M with zero passes scores approximately -3.83, while Qwen2.5-Coder-32B at $1.26/M with pass_i = 0.2 scores approximately -5.50, so the failing model would be labeled optimal. Because the XGBoost classifier is trained on these labels, any systematic occurrence of such cases corrupts the learning target and all downstream results. The paper must define pass_i precisely, state whether zero-pass models are eligible, report the fraction of training labels with pass_i = 0, and re-run the label construction and evaluation with a corrected objective or an explicit eligibility constraint.
  2. [Section 4.2.2, Eq. (2)] The triplet loss is written as max(0, (a·n)/(||a|| ||n||) - (a·p)/(||a|| ||p||) + margin) and then simplified to max(0, a·n - a·p + margin). This simplification is valid only if the embeddings are L2-normalized before the dot product, but the paper never states that normalization step. Without normalization, the loss is scale-dependent, the margin is not in cosine-similarity units, and the training signal changes with the magnitude of the CodeBERT embeddings. Please state explicitly whether embeddings are normalized and include that operation in the implementation description, or use the cosine form throughout.
  3. [Table 4 and Section 6.2] All reported results are point estimates with no error bars, no repeated runs over different random seeds, and no statistical significance tests. The headline claims of a 7.86% pass@1 improvement over ComplexityNet and a 1.12% ablation improvement from fine-tuning could easily fall within run-to-run variance, especially for a small test set of 178 problems. The paper should report means and standard deviations over multiple runs and, for the main comparisons, a significance test or a confidence interval.
  4. [Section 6.1.2, RQ1-2] The validation that CoT difficulty "more accurately reflects LLM perception" than human labels is partly self-referential. The experiment selects problems that Qwen2.5-Coder-32B-Instruct answers correctly, then shows that the CoT lengths for these problems tend to be short. Since CoT length is the same signal used to define the difficulty clusters, observing short CoT on a model-correct subset does not independently establish that CoT captures LLM difficulty. Stronger validation would use out-of-sample models, multiple models, or the correlation between CoT-based difficulty and pass rates across the entire candidate pool.
  5. [Abstract and Section 6.2.1] The abstract claims that "compared to a single model, AdaptiveLLM demonstrates an approximately 15% accuracy improvement, while maintaining the same level of cost consumption," but no row in Table 4 substantiates this exact comparison. The best single model in the pool (Qwen2.5-Coder-32B-Instruct) achieves 57.87% pass@1, higher than AdaptiveLLM's 44.94%; if the comparison is to a different single model, to the pool average, or to a cost-equated baseline, that baseline and the cost-equation procedure must be specified explicitly.
minor comments (6)
  1. [Section 4.1.2] The definition of pass_i should distinguish between the fraction of generated responses that pass and the fraction of test cases passed within a response; the current phrase "passes all the test cases in five responses" is ambiguous.
  2. [Section 4.2.2] Positive samples are described as problems with "similar CoT length" from the same cluster, which is redundant after k-means clustering by length; clarify whether similarity within a cluster is an additional criterion or simply cluster membership.
  3. [Section 5.2.2] The ComplexityNet baseline replaces GPT-4 with gpt-4o-2024-11-20 and DaVinci-002 with Qwen2.5-7B-Instruct; since the original ComplexityNet uses those specific models, the substitutions may affect the comparison, and a sensitivity analysis or a note on expected equivalence should be provided.
  4. [Table 4] In the GPT4o row, the pass@1 price value (551.62e-05) is identical to the pass@1 token value (551.62), which looks like a typographical error; please verify the price calculation.
  5. [Section 6.1.1] For the comparison between CoT-based and human difficulty, only confusion matrices are shown; reporting quantitative agreement metrics such as ARI and FMI, as already used in Table 3, would make the comparison easier to interpret.
  6. [Throughout] There are several typos and wording issues, including "whitch" in Section 2.1, "AdativeLLM" in the Section 6.2 headings, and "promblems" in Section 6.1.2; a careful proofread is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No load-bearing circularity: the supervised labels come from measured pass rates and prices, while the CoT-based difficulty features are constructed independently, and the final evaluation is on a held-out test split.

full rationale

The derivation chain in AdaptiveLLM is self-contained. Optimal-model labels are defined by Eq. (1) from measured per-problem pass rates, token counts, and API prices (Section 4.1.2); these quantities are independent of the CoT-length features used by the classifier. The difficulty signal is obtained by k-means clustering of DeepSeek-R1-Distill-Qwen-32B reasoning lengths (Section 4.2.1), which does not use the candidate models' pass rates or the Eq. (1) score. CodeBERT fine-tuning and XGBoost training use these independently measured inputs, and the reported pass@1/cost results are computed on a 70/30 held-out split (Section 5.3). The RQ1-2 validation of CoT-based difficulty is methodologically weak—it selects problems correctly answered by one strong LLM and then observes that CoT lengths are low for that subset—but it is an empirical consistency check, not a definitional reduction: CoT length is not constructed from correctness, and no equation makes the claimed conclusion true by construction. There are no load-bearing self-citations or imported uniqueness theorems. The underspecified pass_i in Eq. (1) and the possibility that zero-pass cheap models receive high scores are valid correctness and label-quality concerns, but they do not make the framework circular. Overall, the central selection pipeline does not reduce to its own inputs.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The framework introduces no new physical or conceptual entities, but it depends on several hand-chosen hyperparameters and a chain of unvalidated domain assumptions, most notably the CoT-length difficulty correlation and the parameter-size price proxy.

free parameters (8)
  • k (number of difficulty clusters) = 3
    Chosen for k-means clustering of CoT lengths; no sensitivity analysis is provided.
  • margin (triplet loss) = 1
    Default positive value in Eq. (2); no tuning is described.
  • temperature / top_p / top_k for candidate models = 0.3 / 0.95 / 20
    Uniformly set for models without official defaults; these values affect the generated responses and measured pass rates.
  • max tokens for reasoning model = 16384
    Cap on CoT length; longer CoTs are set to 0 and excluded from the dataset.
  • max tokens for code models = 2048
    Cap for generated code; chosen to avoid truncation for all problems.
  • train/test split ratio = 70/30
    Random split into 411 training and 178 test problems; no stratification by dataset is reported.
  • CodeBERT fine-tuning hyperparameters = not reported
    Learning rate, epochs, and batch size are deferred to the public repository.
  • XGBoost hyperparameters = not reported
    Configuration is documented only in the public repository.
assumptions (5)
  • domain assumption CoT length of a reasoning model correlates with problem difficulty for code generation
    Core premise behind the difficulty labeling in Section 4.2.1; if false, the difficulty labels are meaningless.
  • domain assumption k-means with k=3 produces meaningful difficulty tiers
    Three clusters are used without validation of cluster quality or stability.
  • domain assumption Models with similar parameter sizes have similar inference prices
    Used in Sections 4.1.1 and 5.1.1 to proxy prices for models not deployed on SiliconFlow; acknowledged as approximate in the threats to validity.
  • domain assumption SandboxFusion correctly executes and tests generated code
    All correctness labels depend on the sandbox's test results.
  • ad hoc to paper Embeddings are L2-normalized in Eq. (2)
    The identity cos(a,n) - cos(a,p) = a·n - a·p only holds for unit vectors; normalization is not stated in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AdaptiveLLM: A Framework for Selecting Optimal Cost-Efficient LLM for Code-Generation Based on CoT Length." pith.science (2026). https://pith.science/paper/XE4OSSO3

@misc{pith2026250610525,
  author       = {Pith},
  title        = {Pith review of: AdaptiveLLM: A Framework for Selecting Optimal Cost-Efficient LLM for Code-Generation Based on CoT Length},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XE4OSSO3}},
  note         = {Machine review of arXiv:2506.10525}
}
read the original abstract

While Large Language Models (LLMs) have significantly advanced code generation efficiency, they face inherent challenges in balancing performance and inference costs across diverse programming tasks. Dynamically selecting the optimal LLM based on task difficulty and resource constraints offers a promising approach to achieve an optimal balance between efficiency and performance. However, existing model selection methods are resource-intensive and often neglect cost efficiency. Moreover, these approaches rely on human-annotated difficulty labels that are frequently inaccessible in real-world settings and may not align with the LLM's own assessment of task difficulty. In this paper, we introduce AdaptiveLLM, a framework that dynamically selects optimal LLMs for a given coding task by automatically assessing task difficulty. Our framework first estimates task difficulty using Chain-of-Thought lengths generated by reasoning model, clusters these into three difficulty levels via k-means, and fine-tunes CodeBERT to embed difficulty-aware features. A trained XGBoost classifier then selects the best model for each problem, optimizing the performance-cost trade-off. Experimental results show that AdaptiveLLM achieves a 7.86% improvement in pass@1 score while reducing resource consumption by 88.9% compared to baseline method ComplexityNet. When compared to a single model, AdaptiveLLM demonstrates an approximately 15% accuracy improvement, while maintaining the same level of cost consumption. Apart from that, the difficulty assessment using CoT provides more reliable selection criteria than human evaluation. Our replication package is available at https://github.com/cjhCoder7/AdaptiveLLM.

Figures

Figures reproduced from arXiv: 2506.10525 by the authors.

Figure 1
Figure 1. The Use of AdaptiveLLM. 1 Introduction Large Language Models have emerged as transformative tools in code understanding and generation, driving significant advance￾ments in programming efficiency through intelligent assistants such as GitHub Copilot [17] and Cursor [4]. These systems lever￾age LLM’s contextual reasoning capabilities to predict and auto￾complete code snippets, reducing development time and effort. Ho… view at source ↗
Figure 2
Figure 2. The performance of eight code LLMs on HumanEval, LeetCodeSample and CodeContests. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The relationship between Cost and Performance. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The overall architecture of AdaptiveLLM. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: The process of LLM ranking dataset construction. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: The process of CoT difficulty labeling. • Positive sample: Randomly select a problem with a similar CoT length from the difficulty cluster 𝐶𝑖 that is the same as the an￾chored sample. • Negative sample: Randomly choose one of the two clusters𝐶𝑗 and 𝐶𝑘 with different di…
Figure 7
Figure 7. Figure 7: The confusion matrix between CoT difficulty and [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

60 extracted references · 19 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report.arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    Anysphere. 2009. Codeforces. https://codeforces.com/

    work page 2009

  3. [3]

    Anysphere. 2015. LeetCode. https://leetcode.com/

    work page 2015

  4. [4]

    Anysphere. 2023. Cursor. https://www.cursor.com/

    work page 2023

  5. [5]

    Zeliha Ergul Aydin and Zehra Kamisli Ozturk. 2021. Performance analysis of XGBoost classifier with missing data.Manchester Journal of Artificial Intelligence and Applied Sciences (MJAIAS)2, 02 (2021), 2021

    work page 2021

  6. [6]

    Henry Bae, Aghyad Deeb, Alex Fleury, and Kehang Zhu. 2023. Complexitynet: Increasing llm inference efficiency by learning task complexity.arXiv preprint arXiv:2312.11511(2023)

    arXiv 2023

  7. [7]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners.Advances in neural information processing systems33 (2020), 1877–1901

    2020

  8. [8]

    G Ann Campbell. 2018. Cognitive Complexity-A new way of measuring under- standability.SonarSource SA10 (2018)

    work page 2018

Show all 60 references
  1. [9]

    Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps- Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. 2022. Multipl-e: A scalable and extensible approach to benchmarking neural code generation.arXiv preprint...

    2022 arXiv

  2. [10]

    Lingjiao Chen, Matei Zaharia, and James Zou. 2023. Frugalgpt: How to use large language models while reducing cost and improving performance.arXiv preprint arXiv:2305.05176(2023)

    2023 arXiv

  3. [11]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374(2021)

    2021 arXiv

  4. [12]

    Shuhao Chen, Weisen Jiang, Baijiong Lin, James Kwok, and Yu Zhang. 2024. RouterDC: Query-based router by dual contrastive learning for assembling large language models.Advances in Neural Information Processing Systems37 (2024), 66305–66328

    2024

  5. [13]

    Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. InProceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. 785–794

    2016

  6. [14]

    Zhuo Chen, Fu Jiang, Yijun Cheng, Xin Gu, Weirong Liu, and Jun Peng. 2018. XGBoost classifier for DDoS attack detection and analysis in SDN-based cloud. In2018 IEEE international conference on big data and smart computing (bigcomp). IEEE, 251–256

    2018

  7. [15]

    Christof Ebert, James Cain, Giuliano Antoniol, Steve Counsell, and Phillip La- plante. 2016. Cyclomatic complexity.IEEE software33, 6 (2016), 27–29

    2016

  8. [16]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. Codebert: A pre-trained model for programming and natural languages.arXiv preprint arXiv:2002.08155 (2020)

    2020 arXiv

  9. [17]

    Github. 2021. Github Copilot. https://github.com/features/copilot

    2021

  10. [18]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  11. [19]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al . 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948(2025)

    2025 arXiv

  12. [20]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence.arXiv preprint arXiv:2401.14196(2024)

    2024 arXiv

  13. [21]

    Neha Gupta, Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Aditya Krishna Menon, and Sanjiv Kumar. 2024. Language model cascades: Token-level uncertainty and beyond.arXiv preprint arXiv:2404.10136(2024)

    2024 arXiv

  14. [22]

    T Hariprasad, G Vidhyagaran, K Seenu, and Chandrasegar Thirumalai. 2017. Soft- ware complexity analysis using halstead metrics. In2017 international conference on trends in electronics and informatics (ICEI). IEEE, 1109–1113

    2017

  15. [23]

    Elad Hoffer and Nir Ailon. 2015. Deep metric learning using triplet network. In Similarity-based pattern recognition: third international workshop, SIMBAD 2015, Copenhagen, Denmark, October 12-14, 2015. Proceedings 3. Springer, 84–92

    2015

  16. [24]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical report.arXiv preprint arXiv:2409.12186(2024)

    2024 arXiv

  17. [25]

    Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C Park

  18. [26]

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion.arXiv preprint arXiv:2306.02561(2023)

    2023 arXiv

  19. [27]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770(2023)

    2023 arXiv

  20. [28]

    Bonan Kou, Shengmai Chen, Zhijie Wang, Lei Ma, and Tianyi Zhang. 2023. Is model attention aligned with human attention? an empirical study on large language models for code generation.arXiv preprint arXiv:2306.01220(2023)

    2023 arXiv

  21. [29]

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-level code generation with alphacode.Science378, 6624 (2022), 1092–1097

    2022

  22. [30]

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al . 2024. Deepseek- v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434(2024)

    2024 arXiv

  23. [31]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437(2024)

    2024 arXiv

  24. [32]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.Advances in Neural Information Processing Systems 36 (2023), 21558–21572

    2023

  25. [33]

    Siyao Liu, He Zhu, Jerry Liu, Shulin Xin, Aoyan Li, Rui Long, Li Chen, Jack Yang, Jinxiang Xia, ZY Peng, et al. 2024. Fullstack bench: Evaluating llms as full stack coder.arXiv preprint arXiv:2412.00535(2024)

    2024 arXiv

  26. [34]

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy- Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. 2024. Starcoder 2 and the stack v2: The next generation.arXiv preprint arXiv:2402.19173(2024)

    2024 arXiv

  27. [35]

    Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct.arXiv preprint arXiv:2308.09583(2023)

    2023 arXiv

  28. [36]

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. When not to trust language models: Investigat- ing effectiveness of parametric and non-parametric memories.arXiv preprint arXiv:2212.10511(2022)

    2022 arXiv

  29. [37]

    Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. 2023. Octopack: Instruction tuning code large language models. In NeurIPS 2023 Workshop on Instruction Tuning and Instructi...

    2023

  30. [38]

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748(2018)

    2018 arXiv

  31. [39]

    OpenAI. 2024. OpenAI o1. https://openai.com/o1/

    2024

  32. [40]

    OpenAI. 2025. OpenAI o3mini. https://openai.com/index/openai-o3-mini/

    2025

  33. [41]

    Gabriel Orlanski, Kefan Xiao, Xavier Garcia, Jeffrey Hui, Joshua Howland, Jonathan Malmaud, Jacob Austin, Rishabh Singh, and Michele Catasta. 2023. Measuring the impact of programming language distribution. InInternational Conference on Machine Learning. PMLR, 26619–26645

    2023

  34. [42]

    Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. 2025. An empirical study of the non-determinism of chatgpt in code generation.ACM Transactions on Software Engineering and Methodology34, 2 (2025), 1–28

    2025

  35. [43]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiao- qing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950 (2023)

    2023 arXiv

  36. [44]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530(2024)

    2024 arXiv

  37. [45]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupati- raju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295(2024)

    2024 arXiv

  38. [46]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. 2025. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599(2025)

    2025 arXiv

  39. [47]

    Mistral AI team. 2024. Codestral. https://mistral.ai/news/codestral

    2024

  40. [48]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288(2023)

    2023 arXiv

  41. [49]

    Chung-Yu Wang, Alireza DaghighFarsoodeh, and Hung Viet Pham. 2024. Selec- tion of Prompt Engineering Techniques for Code Generation through Predicting Code Complexity.arXiv preprint arXiv:2409.16416(2024). AdaptiveLLM: A Framework for Selecting Optimal Cost-Efficient LLM for C...

    2024 arXiv

  42. [50]

    Moshi Wei, Nima Shiri Harzevili, Yuchao Huang, Junjie Wang, and Song Wang

  43. [51]

    Kurt D Welker. 2001. The software maintainability index revisited.CrossTalk14 (2001), 18–21

    2001

  44. [52]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al . 2024. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115(2024)

    2024 arXiv

  45. [53]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. 2024. Qwen2. 5- math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122(2024)

    2024 arXiv

  46. [54]

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, et al . 2024. Yi: Open foundation models by 01. ai.arXiv preprint arXiv:2403.04652(2024)

    2024 arXiv

  47. [55]

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. Metamath: Bootstrap your own mathematical questions for large language models.arXiv preprint arXiv:2309.12284(2023)

    2023 arXiv

  48. [56]

    Murong Yue, Jie Zhao, Min Zhang, Liang Du, and Ziyu Yao. 2023. Large lan- guage model cascades with mixture of thoughts representations for cost-efficient reasoning.arXiv preprint arXiv:2310.03094(2023)

    2023 arXiv

  49. [57]

    Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, et al. 2023. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. InProceedings of the 29th ACM SIGKDD Conference on Knowled...

    2023

  50. [58]

    Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al . 2024. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence.arXiv preprint arXiv:2406.11931(2024)

    2024 arXiv

  51. [2022]

    InProceedings of the 44th International Conference on Software Engineering

    Clear: contrastive learning for api recommendation. InProceedings of the 44th International Conference on Software Engineering. 376–387

  52. [2024]

    Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity.arXiv preprint arXiv:2403.14403(2024)

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.