Pith. sign in

REVIEW 5 major objections 5 minor 75 references

How to Select Pre-Trained Code Models for Reuse? A Learning Perspective

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Learning-based transferability scores can identify high-performing pre-trained code models from a 100-model zoo in about 100 seconds, without brute-force fine-tuning of all candidates.

desk verdict A useful empirical benchmark for cheap code-model selection, but the ground-truth oracle rests on a single fine-tuning configuration that the paper itself admits may favor small models. read the letter →

arxiv 2501.03783 v1 pith:SLYORKRL submitted 2025-01-07 cs.SE cs.CL

Zhangqian Bi , Yao Wan , Zhaoyang Chu , Yufei Hu , Junyi Zhang , Hongyu Zhang , Guandong Xu , Hai Jin This is my paper · ORCID
classification cs.SEcs.CL
keywords pre-trainedcodemodelsmodelselectiontransferabilityreusefine-tuningintelligencerankingrepresentationlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a developer can choose which pre-trained code model to fine-tune by scoring how well each model's frozen internal representations line up with the target task, instead of brute-force fine-tuning every candidate. On three code classification tasks spanning 100 public pre-trained code models, it reports that learning-based selection lands within 6% of the best fine-tuned model's performance on related tasks while cutting selection time from roughly 2,700 hours to about 100 seconds. It also claims that simple rules, such as picking the largest model or the model trained on the most data, are unreliable guides to downstream performance. The paper's first research question is answered affirmatively: selecting high-performing models without fine-tuning is possible.

What carries the argument

The load-bearing object is the transferability score $\alpha(M,t)$, a scalar meant to predict how well a frozen pre-trained model $M$ would perform on task $t$ after fine-tuning. Proxy-based methods obtain $\alpha$ by training a k-nearest-neighbor, linear, or SVM classifier on model-extracted features and using its test accuracy; distribution-based methods compute pairwise feature-feature and label-label similarity matrices and use a correlation between them, instantiated as PARC (Pearson dissimilarity plus Spearman correlation) and H-Score (covariance-based redundancy and inter-class variance). The score ranks all candidates, and the user fine-tunes only the top $b$ models, where $b$ is the computing budget.

What would settle it

Fine-tune a sample of the candidate models twice, once with the paper's shared hyperparameters and once with per-model tuned hyperparameters, and compare the two best-validation-checkpoint rankings; a large divergence would show that the reported advantage of learning-based selection depends on the shared configuration used to build ground truth.

Watch

Extended reading notes

Core claim

The central claim is that model selection can be cast as a ranking problem solved with forward passes alone: each candidate model is scored by a transferability estimate, the candidates are ranked, and only the top few are fine-tuned under a computing budget. The paper reports that proxy-based methods, which train cheap classifiers on features extracted by the frozen model, and distribution-based methods, which measure how well feature geometry matches task labels, both beat selection by model size or pre-training data size. With a budget of 5 or 10 fine-tuned candidates out of 100, these learning-based methods place the best model in the selected set on the vulnerability detection task and outperform the intuitive baselines on all three tasks, while costing about 100 seconds instead of about 2,700 hours for brute-force fine-tuning. The paper states this as: it is possible to select high-performing models from a zoo of PCMs without fine-tuning.

Load-bearing premise

The ground-truth ranking of the 100 models comes from fine-tuning every model with one shared hyperparameter configuration, so if some models need different settings to perform at their best, the ranking used to judge all selection methods is not the true ranking.

Editorial extensions

If this is right

  • Developers can replace exhaustive fine-tuning with forward-pass scoring during the prototype phase, at a reported selection cost of about 100 seconds per 100 models.
  • With a budget of 10 candidates out of 100, learning-based methods put the best-performing model inside the selected set on the vulnerability detection task, while size- and data-based heuristics do not.
  • The advantage of learning-based selection persists when only 1,000 probe samples are available and grows as more samples or a larger budget are provided.
  • Selection quality remains stable as the model collection scales from 10 to 100 candidates, and learning-based methods consistently beat intuitive methods in all three tested model-zoo sizes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors leave implicit is that the same ranking machinery could make model hubs interactive: a user uploads a small labeled probe set and receives a ranked list of pre-trained models in seconds, with fine-tuning reserved for the top few.
  • The methods assume frozen representations already encode enough task-relevant structure; on downstream tasks where fine-tuning substantially reorganizes representations, the correlation between proxy score and fine-tuned accuracy could weaken, which is a boundary condition the paper does not test.
  • A testable extension is to feed the transferability scores back into an active-learning loop, starting with a small probe set and adding examples only where the top-k ranking is unstable, which could lower the probe-sample cost further.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper studies how to select a pre-trained code model (PCM) from a large model zoo for a downstream classification task, without brute-force fine-tuning every candidate. The authors build a benchmark of 100 open-source PCMs, ranging from 42.5M to 3B parameters, and evaluate three intuitive baselines (model size, pre-training dataset size, exhaustive fine-tuning) and five learning-based strategies: proxy-based methods (k-NN, linear classifier, SVM) and distribution-based methods (PARC, H-Score). The ground-truth rankings are obtained by fine-tuning all 100 models with a shared hyperparameter configuration and selecting the best validation checkpoint. The main results are reported as NDCG@k and Rel@k on three tasks: vulnerability detection (Devign), algorithm classification (POJ-104), and programming language identification (SCC). The paper claims that learning-based methods can identify high-performing models without fine-tuning, reducing selection time from about 2,700 hours to about 100 seconds with less than 6% performance degradation, and that they outperform size- and data-size-based selection across model-zoo scales (RQ1--RQ3).

Significance. If the central empirical claim holds, the paper would provide a practical and timely contribution to model reuse in software engineering: it is one of the first systematic studies of transferability-based PCM selection, and it applies established computer-vision transferability metrics to a code-model zoo with a realistic scale (100 models). The study explicitly contributes a reproducible benchmark, with code and data linked in the Data Availability section, and the evaluation design is generally sound: selection scores are computed from frozen features, the ground truth comes from independent fine-tuning, and random sampling is repeated five times. The work also produces a falsifiable claim about the unreliability of model size and pre-training data size as selection criteria, which is useful regardless of the specific ranking methods. However, as detailed in the major comments, the trustworthiness of the oracle ranking and the exact scope of the reported performance-degradation bound need to be established before the main claims can be accepted.

major comments (5)
  1. [§V-A and §VI-B] The ground-truth ranking is defined by fine-tuning all 100 models with one shared configuration (learning rate 2e-5, 5 epochs, batch size 8 or 32) and taking the best validation checkpoint. Section VI-B concedes that this setting 'may not be fair for all models and might favor smaller ones,' but the assertion that it does not affect selection conclusions is not supported by any evidence. This is load-bearing because every NDCG@k and Rel@k value in Tables II and III, as well as the 'larger models do not always win' finding in Figure 2, is computed against this ranking. The zoo spans 42.5M to 3B parameters and includes decoder-only models, and a fixed 5-epoch schedule can systematically underfit large autoregressive models. Please provide a stability analysis of the oracle: for a stratified subset of models (covering small/large and encoder/decoder/encoder-decoder), fine-tune with per-model adjusted hyperparameters (e.g., longer training, learning-rate search, larger batches) and report the correlation with the original ranking. At minimum, report per-model validation curves or convergence diagnostics to justify that 5 epochs is adequate for the 3B-parameter models. Without this, the measured advantage of learning-based selection over the size baseline may be an artifact of an unfair training budget.
  2. [§V-C, Table II, and Figure 6] On vulnerability detection, the top-1 results do not favor the learning-based methods: Table II reports for proxy-based selection NDCG@1 = 0.42 and Rel@1 = 0.35, while the model-size baseline achieves NDCG@1 = 0.45 and Rel@1 = 0.38. The abstract's 'less than 6% performance degradation' claim is also inconsistent with these numbers: Rel@1 = 0.35 implies about 65% degradation at k=1 on the main task, and the 6% figure only matches the proxy-based Rel@10 = 0.94 on vulnerability detection or the near-1.0 Rel@k values on the other two tasks. The paper should report standard deviations over the five sampling runs (Table II currently has no error bars), explicitly qualify the performance-degradation claim by budget k and task, and revise the RQ1 answer, which states that learning methods are more effective than size-based selection without noting the k=1 exception shown in Figure 6.
  3. [§V-B, Eq. (14)] Equation (14) defines Rel@k as max_{Mi in selected k} Acc(Mi) / Acc(M_best_in_zoo), but the text immediately adds that 'We normalize the Rel@k score by subtracting the accuracy of the model with the lowest performance.' The formula as written does not contain this subtraction, and the described normalization would change the metric to something like (max_selected - Acc_low) / (Acc_best - Acc_low) or require a similar adjustment. Since Rel@k is one of the two central evaluation metrics used in Tables II and III and in the abstract's degradation bound, this inconsistency must be fixed. Please state the exact formula used to produce the reported numbers and clarify whether the normalization is applied in all tables.
  4. [§IV-C and §V-A] The distribution-based methods, especially H-Score, depend on Eq. (11), which computes tr((Sf)^{-1} Sy) and therefore requires inverting the feature redundancy matrix Sf. With high-dimensional features and only 1,000 probe samples, Sf can be singular or ill-conditioned, yet the paper does not describe any regularization, pseudo-inverse, or numerical stabilization for this inverse. Similarly, the paper does not specify which layer of each PCM is used as the feature extractor, how token-level representations are pooled into a sequence-level feature, or whether the same extraction protocol is used across encoder-only, decoder-only, and encoder-decoder models. These details are needed to interpret the reported scores and to reproduce the experiments. Please add the feature-extraction protocol and the numerical treatment of Eq. (11).
  5. [§III-B and Table II] The dataset-size baseline in Table II is applied to all 100 models, but Section III-B evaluates the effect of pre-training data size only on CodeBERT and PLBART variants (Figure 3). It is unclear how pre-training dataset sizes were obtained for the full zoo, including StarCoder-3B (35B tokens in Table I) and variants whose training data is not documented in the model card, and how ties or missing values were handled. Because the conclusion that 'selecting models based on the size of the pre-training dataset is ineffective' is part of the motivation for the learning-based methods, the construction of this baseline for all 100 models should be described explicitly.
minor comments (5)
  1. [§II-B and Table I] The paper states in Section II-B and Section III-A that the smallest model, codet5-small, has 42.5M parameters, but Table I lists codet5-small as 60.49M; please harmonize these numbers.
  2. [Fig. 7] The figure legend contains the typo 'Datset Size' for 'Dataset Size'.
  3. [§V-D] The sentence 'To stimulate this scenario' should read 'To simulate this scenario.'
  4. [§IV-A and §IV-C] The symbol n is used both for the number of models in the zoo (Section IV-A) and for the number of samples in the feature-label matrices Sf and Sy (Eqs. (4)--(5)); please use distinct symbols to avoid confusion.
  5. [§III-C] The sentence 'The implementation follows Approaches A and B, with time costs for fine-tuning each model recorded' is unclear; please specify which aspects of Approaches A and B are followed.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: transferability scores are validated against an independently produced fine-tuning oracle.

full rationale

The paper's derivation chain is self-contained. The transferability scores in Section IV (proxy-based and distribution-based) are computed from PCM-extracted features and task labels (Eqs. 2-11); the ground-truth ranking used for NDCG@k and Rel@k is produced independently by brute-force fine-tuning all 100 models with the protocol in Section V-A. No equation defines a predicted score in terms of the fine-tuned ranking, and no parameter is fitted to that ranking: the proxy test accuracy and feature-label correlations are evaluated directly against the oracle in Eqs. 13-14. The self-citations ([44], [75]) appear only in related work and data availability and are not load-bearing. The principal caveat is Section VI-B, which concedes that the shared fine-tuning configuration 'may not be fair for all models and might favor smaller ones' but asserts without supporting evidence that it does not affect selection conclusions; that is a validity threat to the oracle, not a circularity in the selection scores.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new theoretical entities are introduced; the analysis rests on the validity of the brute-force fine-tuning ground truth and on the proxy/distribution assumptions. Hand-set values are method hyperparameters and the fixed fine-tuning configuration.

free parameters (4)
  • Shared fine-tuning configuration = learning rate 2e-5, 5 epochs, batch size 8 (Devign) or 32 (POJ-104, SCC)
    Used for all 100 models to create the ground-truth ranking; the choice can change which model looks best and is acknowledged as a validity threat in Section VI-B.
  • kNN neighborhood size k = 1, 3, 5
    Method hyperparameter evaluated across settings; not fit to the target tasks.
  • SVM regularization C = 1.0
    Set by hand following prior work; no tuning on the three tasks.
  • Probe sample budget = 1,000, 2,000, 5,000
    Defines how much target data is used to compute transferability scores; the main efficiency claim uses the 1,000-sample setting.
assumptions (4)
  • domain assumption Ground-truth ranking from single-configuration fine-tuning is valid
    Every selection method is judged against brute-force fine-tuning with one shared hyperparameter setting (Section V-A). If models need per-model tuning, the target labels for the benchmark are unreliable.
  • domain assumption Frozen latent features are sufficient signals of transferability
    Proxy and distribution methods see only a forward pass on each PCM and never update its parameters; the entire approach assumes those features correlate with fine-tuned accuracy (Section IV-B).
  • domain assumption Uniform 1,000-sample probe set represents the task
    Selection scores are computed on a small uniform sample of the training data (Section V-C); an unrepresentative sample would break the ranking.
  • domain assumption The three tasks are representative of code intelligence
    The paper generalizes from vulnerability detection, algorithm classification, and language identification; generation tasks are explicitly left out (Section VI-B).

how reviews work

0 comments
Cite this review

Pith. "Pith review of How to Select Pre-Trained Code Models for Reuse? A Learning Perspective." pith.science (2026). https://pith.science/paper/SLYORKRL

@misc{pith2026250103783,
  author       = {Pith},
  title        = {Pith review of: How to Select Pre-Trained Code Models for Reuse? A Learning Perspective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SLYORKRL}},
  note         = {Machine review of arXiv:2501.03783}
}
read the original abstract

Pre-training a language model and then fine-tuning it has shown to be an efficient and effective technique for a wide range of code intelligence tasks, such as code generation, code summarization, and vulnerability detection. However, pretraining language models on a large-scale code corpus is computationally expensive. Fortunately, many off-the-shelf Pre-trained Code Models (PCMs), such as CodeBERT, CodeT5, CodeGen, and Code Llama, have been released publicly. These models acquire general code understanding and generation capability during pretraining, which enhances their performance on downstream code intelligence tasks. With an increasing number of these public pre-trained models, selecting the most suitable one to reuse for a specific task is essential. In this paper, we systematically investigate the reusability of PCMs. We first explore three intuitive model selection methods that select by size, training data, or brute-force fine-tuning. Experimental results show that these straightforward techniques either perform poorly or suffer high costs. Motivated by these findings, we explore learning-based model selection strategies that utilize pre-trained models without altering their parameters. Specifically, we train proxy models to gauge the performance of pre-trained models, and measure the distribution deviation between a model's latent features and the task's labels, using their closeness as an indicator of model transferability. We conduct experiments on 100 widely-used opensource PCMs for code intelligence tasks, with sizes ranging from 42.5 million to 3 billion parameters. The results demonstrate that learning-based selection methods reduce selection time to 100 seconds, compared to 2,700 hours with brute-force fine-tuning, with less than 6% performance degradation across related tasks.

Figures

Figures reproduced from arXiv: 2501.03783 by the authors.

Figure 1
Figure 1. The pipeline of developing and using PCMs, from pre-training to fine-tuning [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The accuracy of each PCM when adapted to the vulnerability detection task via fine-tuning. The accuracy is represented by gradients, with deeper [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The accuracy of (a) CodeBERT models and (b) PLBART models [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The time cost of brute-force fine-tuning and various learning strategies [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: An illustration of the (a) proxy-based and (b) distribution-based model selection strategies [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 7
Figure 7. Figure 7: NDCG@5 and Rel@5 heatmap of each learning-based method for [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 6
Figure 6. Figure 6: NDCG Scores, Rel@k Accuracies, and Time Cost of model selection strategies on the vulnerability detection tasks aligned with the actual best model. Consequently, fine-tuning this subset of top models and then selecting the most effective one is likely to yield strong p…
Figure 8
Figure 8. Figure 8: The time cost of learning-based model selection strategies grows in [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 59 canonical work pages

  1. [1]

    Codebert: A pre-trained model for programming and natural languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “Codebert: A pre-trained model for programming and natural languages,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , 2020, pp. 1536–1547

    work page 2020

  2. [2]

    Unified pre- training for program understanding and generation,

    W. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “Unified pre- training for program understanding and generation,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Online: Association for Computational Linguistics, Jun. 2021, pp. 2655–2668

    work page 2021

  3. [3]

    Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,

    Y . Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , 2021, pp. 8696–8708

    2021

  4. [4]

    Codegen: An open large language model for code with multi-turn program synthesis,

    E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y . Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,” in Proceedings of the International Conference on Learning Representations , 2022

    work page 2022

  5. [5]

    Starcoder: may the source be with you!

    R. Li, L. B. Allal, Y . Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, and J. Chim, “Starcoder: may the source be with you!” arXiv preprint arXiv:2305.06161 , 2023

    arXiv 2023

  6. [6]

    Code llama: Open foundation models for code,

    B. Rozi `ere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. C. Ferrer, A. Grattafiori, W. Xiong, A. D ´efossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. Scialom, and G. Synnaeve, “Code llama: Open foundation models for code,” ar...

    arXiv 2023

  7. [7]

    BERT: pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019, pp. 4171–4186

    work page 2019

  8. [8]

    Demystifying gpt-3 language model: A technical overview,

    C. Li, “Demystifying gpt-3 language model: A technical overview,” https://lambdalabs.com/blog/demystifying-gpt-3/, 2020, [Online; ac- cessed 1-Aug-2022]

    work page 2020

Show all 75 references
  1. [9]

    Hugging Face,

    “Hugging Face,” https://www.huggingface.com, 2023, [Online; accessed 1-Feb-2023]

    2023

  2. [10]

    Tensorflow Hub,

    “Tensorflow Hub,” https://www.tensorflow.org/hub, 2019, [Online; ac- cessed 1-Aug-2022]

    2019

  3. [11]

    Onnx model zoo,

    “Onnx model zoo,” https://github.com/onnx/models, 2023, accessed: 1- Feb-2023

    2023

  4. [12]

    Pytorch hub,

    “Pytorch hub,” https://pytorch.org/hub/, 2023, accessed: 1-Feb-2023

    2023

  5. [13]

    What is the intended usage context of this model? an exploratory study of pre-trained models on various model repositories,

    L. Gong, J. Zhang, M. Wei, H. Zhang, and Z. Huang, “What is the intended usage context of this model? an exploratory study of pre-trained models on various model repositories,” ACM Transactions on Software Engineering and Methodology , vol. 32, no. 3, pp. 1–57, 2023

    2023

  6. [14]

    A survey on neural network interpretability,

    Y . Zhang, P. Ti ˇno, A. Leonardis, and K. Tang, “A survey on neural network interpretability,” IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 5, no. 5, pp. 726–742, 2021

    2021

  7. [15]

    Scaling laws for neural language models,

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361 , 2020

    2001 arXiv

  8. [16]

    Which model to transfer? finding the needle in the growing haystack,

    C. Renggli, A. S. Pinto, L. Rimanic, J. Puigcerver, C. Riquelme, C. Zhang, and M. Lu ˇci´c, “Which model to transfer? finding the needle in the growing haystack,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 9205–9214

    2022

  9. [17]

    Neural data server: A large- scale search engine for transfer learning data,

    X. Yan, D. Acuna, and S. Fidler, “Neural data server: A large- scale search engine for transfer learning data,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 3893–3902

    2020

  10. [18]

    Scalable transfer learning with expert models,

    J. Puigcerver, C. R. Ruiz, B. Mustafa, C. Renggli, A. S. Pinto, S. Gelly, D. Keysers, and N. Houlsby, “Scalable transfer learning with expert models,” in Proceedings of the International Conference on Learning Representations, 2020

    2020

  11. [19]

    Scalable diverse model selec- tion for accessible transfer learning,

    D. Bolya, R. Mittapalli, and J. Hoffman, “Scalable diverse model selec- tion for accessible transfer learning,” in Advances in Neural Information Processing Systems, vol. 34, 2021, pp. 19 301–19 312

    2021

  12. [20]

    An information-theoretic approach to transferability in task transfer learning,

    Y . Bao, Y . Li, S.-L. Huang, L. Zhang, L. Zheng, A. Zamir, and L. Guibas, “An information-theoretic approach to transferability in task transfer learning,” in Proceedings of the IEEE International Conference on Image Processing. IEEE, 2019, pp. 2309–2313

    2019

  13. [21]

    Codesearchnet challenge: Evaluating the state of semantic code search,

    H. Husain, H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “Codesearchnet challenge: Evaluating the state of semantic code search,” arXiv preprint arXiv:1909.09436 , vol. abs/1909.09436, 2019

    1909 arXiv

  14. [22]

    The stack: 3 tb of permissively licensed source code,

    D. Kocetkov, R. Li, L. B. Allal, J. Li, C. Mou, C. M. Ferrandis, Y . Jernite, M. Mitchell, S. Hughes, T. Wolf, D. Bahdanau, L. von Werra, and H. de Vries, “The stack: 3 tb of permissively licensed source code,” arXiv preprint arXiv:2211.15533 , 2022

    2022 arXiv

  15. [23]

    An extensive study on pre-trained models for program understanding and generation,

    Z. Zeng, H. Tan, H. Zhang, J. Li, Y . Zhang, and L. Zhang, “An extensive study on pre-trained models for program understanding and generation,” in Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis , 2022, pp. 39–51

    2022

  16. [24]

    Graphcodebert: Pre- training code representations with data flow,

    D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu, M. Tufano, S. K. Deng, C. B. Clement, D. Drain, N. Sundaresan, J. Yin, D. Jiang, and M. Zhou, “Graphcodebert: Pre- training code representations with data flow,” in Proceedings of the 9t...

    2021

  17. [25]

    Codexglue: A machine learning bench- mark dataset for code understanding and generation,

    S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. Gong, M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. Liu, “Codexglue: A machine learning bench- mark dataset for co...

    2021 arXiv

  18. [26]

    Codegeex: A pre- trained model for code generation with multilingual benchmarking on humaneval-x,

    Q. Zheng, X. Xia, X. Zou, Y . Dong, S. Wang, Y . Xue, L. Shen, Z. Wang, A. Wang, Y . Li, T. Su, Z. Yang, and J. Tang, “Codegeex: A pre- trained model for code generation with multilingual benchmarking on humaneval-x,” in Proceedings of the 29th ACM SIGKDD Conference on Knowled...

    2023

  19. [27]

    Competition- level code generation with alphacode,

    Y . Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. D. Lago, T. Hubert, P. Choy, C. de Masson d’Autume, I. Babuschkin, X. Chen, P.-S. Huang, J. Welbl, S. Gowal, A. Cherepanov, J. Molloy, D. J. Mankowitz, E. S. Robson, P. Ko...

    2022

  20. [28]

    Electra: Pre- training text encoders as discriminators rather than generators,

    K. Clark, M.-T. Luong, Q. V . Le, and C. D. Manning, “Electra: Pre- training text encoders as discriminators rather than generators,” in Proceedings of International Conference on Learning Representations , 2020

    2020

  21. [29]

    BART: Denoising sequence-to- sequence pre-training for natural language generation, translation, and comprehension,

    M. Lewis, Y . Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V . Stoyanov, and L. Zettlemoyer, “BART: Denoising sequence-to- sequence pre-training for natural language generation, translation, and comprehension,” in Proceedings of the 58th Annual Meeting of the Associat...

    2020

  22. [30]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020

    2020

  23. [31]

    Devign: Effective vulner- ability identification by learning comprehensive program semantics via graph neural networks,

    Y . Zhou, S. Liu, J. Siow, X. Du, and Y . Liu, “Devign: Effective vulner- ability identification by learning comprehensive program semantics via graph neural networks,” in Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch ´...

    2019

  24. [32]

    Convolutional neural networks over tree structures for programming language processing,

    L. Mou, G. Li, L. Zhang, T. Wang, and Z. Jin, “Convolutional neural networks over tree structures for programming language processing,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 30, no. 1, 2016

    2016

  25. [33]

    Scc: Automatic classification of code snippets,

    K. Alreshedy, D. Dharmaretnam, D. M. German, V . Srinivasan, and T. A. Gulliver, “Scc: Automatic classification of code snippets,” in Proceedings of the IEEE International Working Conference on Source Code Analysis and Manipulation , 2018, pp. 203–208

    2018

  26. [34]

    CodeBERTScore: Eval- uating code generation with pretrained models of code,

    S. Zhou, U. Alon, S. Agarwal, and G. Neubig, “CodeBERTScore: Eval- uating code generation with pretrained models of code,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association f...

    2023

  27. [35]

    Code Parrot,

    “Code Parrot,” https://huggingface.co/datasets/codeparrot/ codeparrot-clean, 2023, [Online; accessed 1-Feb-2023]

    2023

  28. [36]

    On the transferability of pre-trained language models for low-resource programming languages,

    F. Chen, F. H. Fard, D. Lo, and T. Bryksin, “On the transferability of pre-trained language models for low-resource programming languages,” in Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension, 2022, pp. 401–412

    2022

  29. [37]

    How could neural networks understand programs?

    D. Peng, S. Zheng, Y . Li, G. Ke, D. He, and T.-Y . Liu, “How could neural networks understand programs?” in Proceedings of the International Conference on Machine Learning . PMLR, 2021, pp. 8476–8486

    2021

  30. [38]

    Cumulated gain-based evaluation of ir techniques,

    K. J ¨arvelin and J. Kek ¨al¨ainen, “Cumulated gain-based evaluation of ir techniques,” ACM Transactions on Information Systems (TOIS), vol. 20, no. 4, p. 422–446, oct 2002

    2002

  31. [39]

    Rank- ing neural checkpoints,

    Y . Li, X. Jia, R. Sang, Y . Zhu, B. Green, L. Wang, and B. Gong, “Rank- ing neural checkpoints,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 2663–2673

    2021

  32. [40]

    How stable are transferability metrics evaluations?

    A. Agostinelli, M. P ´andy, J. Uijlings, T. Mensink, and V . Ferrari, “How stable are transferability metrics evaluations?” in Proceedings of the European Conference on Computer Vision . Springer, 2022, pp. 303– 321

    2022

  33. [41]

    Sup- port vector machines,

    M. A. Hearst, S. T. Dumais, E. Osuna, J. Platt, and B. Scholkopf, “Sup- port vector machines,” IEEE Intelligent Systems and Their Applications , vol. 13, no. 4, pp. 18–28, 1998

    1998

  34. [42]

    Numerically stable, single-pass, parallel statistics algorithms,

    J. Bennett, R. Grout, P. P ´ebay, D. Roe, and D. Thompson, “Numerically stable, single-pass, parallel statistics algorithms,” in Proceedings of the IEEE International Conference on Cluster Computing and Workshops . IEEE, 2009, pp. 1–8

    2009

  35. [43]

    chatgpt,

    “chatgpt,” http://chat.openai.com, 2023, [Online; accessed 1-Feb-2023]

    2023

  36. [44]

    Deep learning for code intelligence: Survey, benchmark and toolkit,

    Y . Wan, Z. Bi, Y . He, J. Zhang, H. Zhang, Y . Sui, G. Xu, H. Jin, and P. Yu, “Deep learning for code intelligence: Survey, benchmark and toolkit,” ACM Computing Surveys , 2024

    2024

  37. [45]

    Deep code search,

    X. Gu, H. Zhang, and S. Kim, “Deep code search,” in Proceedings of 40th International Conference on Software Engineering , 2018, pp. 933– 944

    2018

  38. [46]

    Multi- modal attention network learning for semantic source code retrieval,

    Y . Wan, J. Shu, Y . Sui, G. Xu, Z. Zhao, J. Wu, and P. S. Yu, “Multi- modal attention network learning for semantic source code retrieval,” in Proceedings of 34th IEEE/ACM International Conference on Automated Software Engineering. IEEE, 2019, pp. 13–25

    2019

  39. [47]

    Deep code comment gener- ation,

    X. Hu, G. Li, X. Xia, D. Lo, and Z. Jin, “Deep code comment gener- ation,” in Proceedings of the IEEE/ACM 26th International Conference on Program Comprehension. IEEE, 2018, pp. 200–20 010

    2018

  40. [48]

    code2seq: Generating sequences from structured representations of code,

    U. Alon, S. Brody, O. Levy, and E. Yahav, “code2seq: Generating sequences from structured representations of code,” in Proceedings of the International Conference on Learning Representations , 2018

    2018

  41. [49]

    Improving automatic source code summarization via deep reinforce- ment learning,

    Y . Wan, Z. Zhao, M. Yang, G. Xu, H. Ying, J. Wu, and P. S. Yu, “Improving automatic source code summarization via deep reinforce- ment learning,” in Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. ACM, 2018, pp. 397– 407

    2018

  42. [50]

    Reinforcement-learning-guided source code summarization us- ing hierarchical attention,

    W. Wang, Y . Zhang, Y . Sui, Y . Wan, Z. Zhao, J. Wu, S. Y . Philip, and G. Xu, “Reinforcement-learning-guided source code summarization us- ing hierarchical attention,” IEEE Transactions on software Engineering , vol. 48, no. 1, pp. 102–119, 2020

    2020

  43. [51]

    Modeling hierarchical syntax structure with triplet position for source code summarization,

    J. Guo, J. Liu, Y . Wan, L. Li, and P. Zhou, “Modeling hierarchical syntax structure with triplet position for source code summarization,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 486–500

    2022

  44. [52]

    Deep learning code fragments for code clone detection,

    M. White, M. Tufano, C. Vendome, and D. Poshyvanyk, “Deep learning code fragments for code clone detection,” in Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineer- ing. ACM, 2016, pp. 87–98

    2016

  45. [53]

    Fcca: Hybrid code representation for functional clone detection using attention networks,

    W. Hua, Y . Sui, Y . Wan, G. Liu, and G. Xu, “Fcca: Hybrid code representation for functional clone detection using attention networks,” IEEE Transactions on Reliability , vol. 70, no. 1, pp. 304–318, 2020

    2020

  46. [54]

    Iterative refinement of project-level code context for precise code generation with compiler feedback,

    Z. Bi, Y . Wan, Z. Wang, H. Zhang, B. Guan, F. Lu, Z. Zhang, Y . Sui, H. Jin, and X. Shi, “Iterative refinement of project-level code context for precise code generation with compiler feedback,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Lin...

    2024

  47. [55]

    Tree-to-tree neural networks for program translation,

    X. Chen, C. Liu, and D. Song, “Tree-to-tree neural networks for program translation,” in Advances in Neural Information Processing Systems , 2018, pp. 2552–2562

    2018

  48. [56]

    Deep learning for code generation: A survey,

    H. Zhang, K. Zhang, Z. Li, J. Li, Y . Li, Y . Zhao, Y . Zhu, F. Liu, G. Li, and Z. Jin, “Deep learning for code generation: A survey,” SCIENCE CHINA Information Sciences , 2024

    2024

  49. [57]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971 , 2023

    2023 arXiv

  50. [58]

    What do they capture? - A structural analysis of pre-trained language models for source code,

    Y . Wan, W. Zhao, H. Zhang, Y . Sui, G. Xu, and H. Jin, “What do they capture? - A structural analysis of pre-trained language models for source code,” in Proceedings of the IEEE/ACM 44th International Conference on Software Engineering . ACM, 2022, pp. 2377–2388

    2022

  51. [59]

    Ast-probe: Recovering abstract syntax trees from hidden representations of pre-trained language models,

    J. A. Hern ´andez L ´opez, M. Weyssow, J. S. Cuadrado, and H. Sahraoui, “Ast-probe: Recovering abstract syntax trees from hidden representations of pre-trained language models,” in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering , se...

    2023

  52. [60]

    Better context makes better code language models: a case study on function call argument completion,

    H. Pei, J. Zhao, L. Lausen, S. Zha, and G. Karypis, “Better context makes better code language models: a case study on function call argument completion,” in Proceedings of the Thirty-Seventh AAAI Con- ference on Artificial Intelligence . AAAI Press, 2023

    2023

  53. [61]

    Towards efficient fine-tuning of pre-trained code models: An experi- mental study and beyond,

    E. Shi, Y . Wang, H. Zhang, L. Du, S. Han, D. Zhang, and H. Sun, “Towards efficient fine-tuning of pre-trained code models: An experi- mental study and beyond,” in Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis , 2023, pp. 39–51

    2023

  54. [62]

    Automating code-related tasks through transformers: The impact of pre-training,

    R. Tufano, L. Pascarella, and G. Bavota, “Automating code-related tasks through transformers: The impact of pre-training,” in Proceedings of the IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 2425–2437

    2023

  55. [63]

    Natural attack for pre-trained models of code,

    Z. Yang, J. Shi, J. He, and D. Lo, “Natural attack for pre-trained models of code,” in Proceedings of the 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022 . ACM, 2022, pp. 1482–1493

    2022

  56. [64]

    ReCode: Robustness evaluation of code generation models,

    S. Wang, Z. Li, H. Qian, C. Yang, Z. Wang, M. Shang, V . Kumar, S. Tan, B. Ray, P. Bhatia, R. Nallapati, M. K. Ramanathan, D. Roth, and B. Xiang, “ReCode: Robustness evaluation of code generation models,” in Proceedings of the 61st Annual Meeting of the Association for Computa...

    2023

  57. [65]

    Status report: Software reusability,

    R. Prieto-Diaz, “Status report: Software reusability,” IEEE Software , vol. 10, no. 3, pp. 61–66, 1993

    1993

  58. [66]

    Software reuse research: Status and future,

    W. B. Frakes and K. Kang, “Software reuse research: Status and future,” IEEE Transactions on Software Engineering, vol. 31, no. 7, pp. 529–536, 2005

    2005

  59. [67]

    Ranking and tuning pre-trained models: A new paradigm for exploiting model hubs,

    K. You, Y . Liu, Z. Zhang, J. Wang, M. I. Jordan, and M. Long, “Ranking and tuning pre-trained models: A new paradigm for exploiting model hubs,” Journal of Machine Learning Research , vol. 23, no. 209, pp. 1–47, 2022

    2022

  60. [68]

    Do better imagenet models trans- fer better?

    S. Kornblith, J. Shlens, and Q. V . Le, “Do better imagenet models trans- fer better?” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 2661–2671

    2019

  61. [69]

    An empirical study of pre- trained model reuse in the hugging face deep learning model registry,

    W. Jiang, N. Synovic, M. Hyatt, T. R. Schorlemmer, R. Sethi, Y .-H. Lu, G. K. Thiruvathukal, and J. C. Davis, “An empirical study of pre- trained model reuse in the hugging face deep learning model registry,” in Proceedings of the 2023 IEEE/ACM 45th International Conference on...

    2023

  62. [70]

    What makes a high-quality training dataset for large language models: A practition- ers’ perspective,

    X. Yu, Z. Zhang, F. Niu, X. Hu, X. Xia, and J. Grundy, “What makes a high-quality training dataset for large language models: A practition- ers’ perspective,” in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering , 2024, pp. 656–668

    2024

  63. [71]

    What to pre-train on? efficient intermediate task selection,

    C. Poth, J. Pfeiffer, A. R ¨uckl´e, and I. Gurevych, “What to pre-train on? efficient intermediate task selection,” arXiv preprint arXiv:2104.08247 , 2021

    2021 arXiv

  64. [72]

    Coding- ptms: How to find optimal code pre-trained models for code embedding in vulnerability detection?

    Y . Zhao, L. Gong, Z. Huang, Y . Wang, M. Wei, and F. Wu, “Coding- ptms: How to find optimal code pre-trained models for code embedding in vulnerability detection?” in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, 2024, pp. 1732–1744

    2024

  65. [73]

    Model spider: Learning to rank pre-trained models efficiently,

    Y .-K. Zhang, T.-J. Huang, Y .-X. Ding, D.-C. Zhan, and H.-J. Ye, “Model spider: Learning to rank pre-trained models efficiently,” in Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  66. [74]

    Guided recommendation for model fine-tuning,

    H. Li, C. Fowlkes, H. Yang, O. Dabeer, Z. Tu, and S. Soatto, “Guided recommendation for model fine-tuning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 3633–3642

    2023

  67. [75]

    Naturalcc: An open-source toolkit for code intelligence,

    Y . Wan, Y . He, Z. Bi, J. Zhang, Y . Sui, H. Zhang, K. Hashimoto, H. Jin, G. Xu, C. Xiong, and P. S. Yu, “Naturalcc: An open-source toolkit for code intelligence,” in Proceedings of 44th International Conference on Software Engineering, Companion Volume . ACM, 2022

    2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.