Pith. sign in

REVIEW 4 major objections 8 minor 71 references

On the Generalizability of Transformer Models to Code Completions of Different Lengths

T0 review · 4 major / 8 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that none of the four studied positional encoding schemes lets encoder-decoder Transformers generalize code completion to input lengths unseen during training.

desk verdict A useful negative empirical result—training on short code does not transfer to longer code for any tested positional encoding—but the abstract and conclusion overstate the mechanism because length buckets are confounded with code content and target difficulty. read the letter →

arxiv 2501.05051 v1 pith:DQ2UWLZE submitted 2025-01-09 cs.SE

Nathan Cooper , Rosalia Tufano , Gabriele Bavota , Denys Poshyvanyk This is my paper · ORCID
classification cs.SE
keywords codecompletionlengthgeneralizationpositionalencodingtransformerencoder-decoderALiBixPOSempiricalstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether code-completion Transformers can be trained on short code excerpts and still perform well on longer excerpts at inference, a property that would substantially cut training cost. It tests four positional encoding schemes—Sinusoidal, xPOS, ALiBi, and T5—on Java and Python completion tasks, holding the number of masked tokens fixed at 11 so that only input length changes. Across 32 trained models, every scheme degrades substantially when training and test lengths differ, and the degradation runs in both directions: short-trained models fail on long inputs and long-trained models fail on short inputs. The authors conclude that none of the studied schemes generalizes to unseen lengths and that including a representative mixture of lengths in training is the only reliable safeguard.

What carries the argument

The study's object is the sequence-to-sequence Transformer and four ways of injecting position information. Sinusoidal adds fixed sine and cosine position vectors to token embeddings at the network input. ALiBi adds a static, head-specific linear bias to attention scores so distant keys are down-weighted. T5 adds a learned, bucketed relative-position bias to attention. xPOS combines rotary relative-position embeddings with an exponential decay on attention resolution, with the paper using the Rotary encoder and the xPOS decoder. Each scheme is trained on function-length datasets labeled short, medium, long, and mix, with task complexity held constant by always masking 11 tokens, and evaluated by Exact Match, ChrF, and Rouge-L.

What would settle it

A concrete test would be to pad short functions with harmless filler tokens up to long lengths while keeping the 11 masked tokens identical: if any scheme keeps close-to-training performance on these padded inputs, the observed degradation is about length itself; if performance still collapses, dataset-content shift, not positional encoding, explains the paper's conclusion. Alternatively, finding one trained model in the released checkpoints that attains near-matched Exact Match on an unseen-length test set would directly refute the claim that none generalizes.

Watch

Extended reading notes

Core claim

The central empirical claim is that, for code completion in an encoder-decoder Transformer, none of the studied positional encoding schemes—Sinusoidal, xPOS, ALiBi, or T5—generalizes to input lengths outside the training range. The evidence is consistent across languages and metrics: for example, a Sinusoidal model trained on short Java functions reaches 10.81 percent Exact Match on short tests but only 2.91 percent when trained on medium functions and 0.50 percent when trained on long ones; T5, the best scheme, drops from 17.03 percent to 3.62 percent on long tests when trained on short instead of long inputs. The paper interprets this as length-specific behavior: attention mechanisms that extrapolate in decoder-only NLP settings do not transfer their length-extrapolation benefit to encoder-decoder code completion. The authors therefore conclude that there is no shortcut: training data should represent the lengths expected at inference time.

Load-bearing premise

The load-bearing premise is that holding the masked token count at 11 isolates input length as the only changed variable, so that short, medium, and long functions differ in length rather than in content, style, or difficulty.

Editorial extensions

If this is right

  • The train-short, test-long cost-saving strategy is not available with any of these encoding schemes for encoder-decoder code completion.
  • Models trained exclusively on short functions collapse on long completions, and models trained only on long functions collapse on short ones, so length-specific specialization is brittle.
  • Training on a mixture of lengths is the safe configuration: it avoids most of the degradation, even though it can give up a few points against a perfectly matched training length.
  • T5's learned relative bias is the strongest of the four in absolute performance and in relative drop, but it still fails the generalization test.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors leave implicit is that length generalization in decoder-only language models may depend on the task and architecture, not just on the attention bias; their negative result suggests encoder-decoder code completion is a harder extrapolation setting than language modeling.
  • A natural next experiment is to pad short functions to long lengths with neutral tokens while keeping the 11 masked tokens identical; if performance stays high, the failure is purely positional, whereas a drop would indicate that long functions differ in content or style, not just length.
  • The result also implies that cost-savings claims for training on short data should be re-examined for code: the real saving may require mixed-length or length-matched training, which removes much of the efficiency advantage.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The manuscript empirically investigates whether four positional encoding schemes (Sinusoidal, xPOS, ALiBi, and T5 relative bias) enable encoder-decoder Transformers to generalize to code completions of lengths not seen during training. The authors construct short, medium, long, and mixed datasets from Java (statement-level completion) and Python (block-level completion) functions, holding the number of masked tokens fixed at 11 and equalizing training set sizes across conditions. They train 32 models (4 schemes × 4 datasets × 2 languages) and evaluate them on held-out short/medium/long test sets using exact match, ChrF, and RougeL. The primary finding is that all schemes exhibit large performance degradation when training and test lengths differ, with T5 being the best-performing and least-degraded scheme; a secondary finding is that training on a mixture of lengths is a safer compromise than single-length training.

Significance. If the findings hold, this is a valuable negative result for the software engineering community: length-extrapolation positional encodings that work in some NLP decoder-only settings do not automatically transfer to encoder-decoder code completion, and practitioners must include representative lengths in training data. The study is carefully set up in several respects: duplicate removal, equal training sizes, a fixed masked-token count, and a shared hyperparameter configuration; the release of code, data, and checkpoints is a concrete asset. The main caveat is that the central 'length generalization' claim rests on the assumption that the length buckets differ only in input length, which is threatened by a content confound (Major Comment 1). The value of the paper therefore depends on either additional control analyses or a more carefully scoped conclusion.

major comments (4)
  1. [Section IV-A] The dataset construction does not actually isolate input length as the only changed variable. The authors sort functions by total token count and split into thirds, then mask the last 11 tokens of a statement or block in each function. While the number of masked tokens is held constant, the content of those 11 tokens and the surrounding code are not controlled; longer functions are likely to contain systematically different statements (e.g., more complex expressions, deeper nesting, different control-flow constructs) and thus a different target distribution. Consequently, the degradation observed in Tables II-IV when training on one length bucket and testing on another could be caused by a shift in code content and target difficulty, rather than by a failure of the positional encodings to extrapolate to unseen input lengths. This directly undermines the central claim in Section VII that 'none of the studied positional encoding schemes has the ability to generalize to unseen lengths.' I recommend either softening the conclusion to a statement about transfer across length-correlated code distributions, or adding a control analysis that matches the target statements across length buckets (e.g., by statement type, AST depth, or target vocabulary) to isolate the effect of input length.
  2. [Section IV-B and Tables II-VII] The paper presents only point estimates from a single training run per configuration, with no seed variance, error bars, or statistical significance tests. This is a problem for the finer-grained comparative claims, such as 'T5's positional encoding scheme ... is also better at generalizing to unseen lengths' (Section V) and the relative deltas in the mix tables (Tables V-VII), where some differences are small (e.g., Table VI, Java xPOS short, Δ = +1.16%). The main degradation effect is large and consistent, but the ranking and 'safe compromise' conclusions require a measure of variability to be credible. Please report multiple seeds for at least the main configurations, or provide bootstrap confidence intervals on the test sets.
  3. [Abstract and Section VII] The paper repeatedly refers to 'Large Language Models' and states general conclusions about 'Transformer models', but the experiments train small encoder-decoder Transformers from scratch (Table I: d_model=512, 6-8 layers; 280k training instances per dataset). The negative result may not transfer to large-scale pretrained code models (e.g., CodeT5, CodeGen, StarCoder) that use different training objectives, much larger corpora, and different decoding strategies. The conclusion should be explicitly scoped to the studied architecture, model size, and training setup, and the external-validity section should acknowledge this limitation.
  4. [Section V (Impact of training diversity) and Tables V-VII] The recommendation that training on a mixture of lengths 'might be the safest choice in most of cases' (Take Away #3) is not fully supported by the Python results. On Python, the mix-trained models are substantially worse than the length-matched models, with relative exact-match drops as large as -44.71% for T5 on the short test set (Table V), and similarly large drops for Sinusoidal and ALiBi. The paper acknowledges that 'training diversity helped less in Python,' but the recommended strategy still carries a large penalty for the best-performing scheme (T5) on one of the two languages. Please either quantify when mixing is beneficial versus harmful (e.g., by language, scheme, or test length) or soften the take-away accordingly.
minor comments (8)
  1. [Section VI] The sentence 'We investigated two Transformer architectures to mitigate the threats to external validity' is factually incorrect; the study uses one sequence-to-sequence Transformer architecture with four positional encoding variants. It should refer to two programming languages or two completion tasks.
  2. [Section V] There are several typos: 'Sonusoidal' should be 'Sinusoidal', 'syntaxt' should be 'syntax', and the abbreviation 'ChfR' is used inconsistently for 'ChrF'.
  3. [Section IV-A] The enumerated list of data-collection steps skips item (v), going from (iv) directly to (vi); this is a numbering error.
  4. [Section V (Impact of training diversity)] The text states 'the deltas in Tables V, III, and VII' when referring to the mix results; this should be 'Tables V, VI, and VII.'
  5. [Section IV-A] The sentence 'This may end up in generating duplicates due to different functions from which we masked the only part being different' is unclear and should be rephrased.
  6. [Section IV-B] The use of nucleus sampling (topp=0.95) for evaluation is not standard for exact-match measurement; since the study reports EM as a key metric, the authors should also provide greedy-decoding results to rule out sampling-induced variance.
  7. [Tables II-IV] The black-box, bold, and red highlighting described in Section V are not visible in the manuscript text provided; please ensure the visual encoding is clear in the camera-ready version.
  8. [Introduction] The paper uses 'LLM' to refer both to large pretrained models and to the small models trained here; this conflation should be corrected throughout, e.g., by using 'Transformer models' for the experiments.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim rests on held-out train/test length-bucket comparisons, not on fitted parameters or self-citations that assume the conclusion.

full rationale

The paper's central claim that none of the studied positional encoding schemes generalizes to unseen lengths is established by an empirical comparison of models trained on disjoint length buckets (short, medium, long, mix) and evaluated on held-out test sets of the same buckets. No parameter is fitted to the test data, and the metrics (EM, ChrF, RougeL) are computed on model predictions against ground-truth completions rather than being derived from the model definition or from any claimed equivalence between training and test conditions. The only adjacent conceptual concern is that the datasets are built by splitting functions on total token count while holding the masked-token count at 11, so longer functions may also differ in code content, target difficulty, or style; this is a construct-validity or dataset-shift threat to interpretation, not a circular derivation. Citations to prior work, including Ciniselli et al. [2], are used to motivate the code-completion tasks and to position the study, but the generalization result itself does not reduce to any cited theorem or to the authors' own prior claims. There are no fitted inputs renamed as predictions, no uniqueness arguments imported from self-citations, and no ansatz smuggled in through citation. The paper is self-contained as an empirical study: its conclusions are falsifiable by the released data and models, and its observed performance drops are reported rather than assumed. Accordingly, no circular step is present, and the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim is empirical, not derivational. The only numbers the claim depends on are experimental design constants (length thresholds, masked token count, hyperparameters). These are chosen by hand but not fitted to performance; they shape the scope of the conclusion rather than circularly guaranteeing it.

free parameters (3)
  • masked_token_count = 11
    Median statement/block length in the data; chosen to hold prediction complexity constant across length datasets. Not fitted to model performance.
  • length_range_thresholds = Java: short 6-96, medium 97-180, long 181-1024; Python: short 30-150, medium 151-309, long 310-1024
    Chosen by splitting sorted functions into three equal-size sets; these boundaries define what 'unseen length' means and affect the magnitude of observed degradation.
  • hyperparameters = learning rate 1e-4, batch 256, encoder/decoder layers 6/8, heads 8/6, max lengths 1024/128
    Fixed to Transformer paper defaults across all 32 models; not tuned per scheme, which may disadvantage some schemes.
assumptions (3)
  • domain assumption ChrF and RougeL are valid proxies for code completion quality.
    The paper relies on Evtikhiev et al. [66] for significance thresholds, but the transfer of those thresholds to this task is assumed.
  • domain assumption The GitHub-sourced functions are representative of real-world code completion scenarios.
    Top-3k projects by stars, master branch, with size/duplicate filters; acknowledged in threats to external validity.
  • domain assumption Unseen length generalization can be isolated by holding the number of masked tokens constant at 11.
    Assumes equal target length implies equal task difficulty, which may not hold across statements/blocks of different lengths.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Generalizability of Transformer Models to Code Completions of Different Lengths." pith.science (2026). https://pith.science/paper/DQ2UWLZE

@misc{pith2026250105051,
  author       = {Pith},
  title        = {Pith review of: On the Generalizability of Transformer Models to Code Completions of Different Lengths},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DQ2UWLZE}},
  note         = {Machine review of arXiv:2501.05051}
}
read the original abstract

The programming landscape is nowadays being reshaped by the advent of Large Language Models (LLMs) able to automate code-related tasks related to code implementation (e.g., code completion) and comprehension (e.g., code summarization). Such a paradigm shift comes with a number of implications related to how software will be written, maintained, and evolved. Also, these LLMs are extremely expensive to train, posing questions on their sustainability over time. Given their training cost, their ability to generalize, namely their ability to work on task instances different from those on which they have been trained, is an aspect worth being investigated. Previous work already showed that transformer models can successfully support code completion in a cross-project setting. However, it is unclear whether LLM are able to generalize to inputs having lengths not seen during training. For example, it is known that training a model on short instances allows to substantially reduce the training cost. However, the extent to which such a model would provide good performance on sequences having lengths not seen during training is not known. Many recent works in Natural Language Processing (NLP) tackled this problem in the context of decoder-only LLMs, i.e., xPOS and ALiBi. To assess if these solutions extend to encoder-decoder LLMs usually adopted in the code-related tasks, we present a large empirical study evaluating this generalization property of these and other encoding schemes proposed in the literature, namely Sinusoidal, xPOS, ALiBi, and T5. We found that none of these solutions successfully generalize to unseen lengths and that the only safe solution is to ensure the representativeness in the training set of all lengths likely to be encountered at inference time.

Figures

Figures reproduced from arXiv: 2501.05051 by the authors.

Figure 2
Figure 2. ALiBi Overview from the original paper [18]. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 1
Figure 1. Sequence to Sequence Transformer Overview from the original paper [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

71 extracted references · 39 canonical work pages

  1. [1]

    Toward deep learning software repositories,

    M. White, C. Vendome, M. Linares-Vasquez, and D. Poshyvanyk, “Toward deep learning software repositories,” in 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories , 2015, pp. 334– 345

    work page 2015

  2. [2]

    An empirical study on the usage of transformer models for code completion,

    M. Ciniselli, N. Cooper, L. Pascarella, A. Mastropaolo, E. Aghajani, D. Poshyvanyk, M. Di Penta, and G. Bavota, “An empirical study on the usage of transformer models for code completion,” IEEE Transactions on Software Engineering , 2021

    work page 2021

  3. [3]

    Using pre-trained models to boost code review automa- tion,

    R. Tufano, S. Masiero, A. Mastropaolo, L. Pascarella, D. Poshyvanyk, and G. Bavota, “Using pre-trained models to boost code review automa- tion,” in 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE), 2022, pp. 2291–2302

    work page 2022

  4. [4]

    Recommendations for datasets for source code summarization,

    A. LeClair and C. McMillan, “Recommendations for datasets for source code summarization,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, Volume 1 (Long and Short Papers), Jun. 2019, pp. 3931–3937

    work page 2019

  5. [5]

    An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation,

    M. Tufano, C. Watson, G. Bavota, M. D. Penta, M. White, and D. Poshyvanyk, “An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation,” ACM Transactions on Software Engineering and Methodology , vol. 28, no. 4, pp. 1–29, 2019

    work page 2019

  6. [6]

    Sequencer: Sequence-to-sequence learning for end-to- end program repair,

    Z. Chen, S. Kommrusch, M. Tufano, L. Pouchet, D. Poshyvanyk, and M. Monperrus, “Sequencer: Sequence-to-sequence learning for end-to- end program repair,” CoRR, vol. abs/1901.01808, 2019

    arXiv 1901

  7. [7]

    On learning meaningful assert statements for unit test cases,

    C. Watson, M. Tufano, K. Moran, G. Bavota, and D. Poshyvanyk, “On learning meaningful assert statements for unit test cases,” in 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE), 2020, pp. 1398–1409

    work page 2020

  8. [8]

    Learning how to mutate source code from bug-fixes,

    M. Tufano, C. Watson, G. Bavota, M. Di Penta, M. White, and D. Poshyvanyk, “Learning how to mutate source code from bug-fixes,” in 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2019, pp. 301–312

    work page 2019

Show all 71 references
  1. [9]

    A systematic literature review on the use of deep learning in software engineering research,

    C. Watson, N. Cooper, D. N. Palacio, K. Moran, and D. Poshyvanyk, “A systematic literature review on the use of deep learning in software engineering research,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 31, no. 2, pp. 1–58, 2022

    2022

  2. [10]

    Using transfer learning for code- related tasks,

    A. Mastropaolo, N. Cooper, D. N. Palacio, S. Scalabrino, D. Poshy- vanyk, R. Oliveto, and G. Bavota, “Using transfer learning for code- related tasks,” IEEE Transactions on Software Engineering , pp. 1–20, 2022

    2022

  3. [11]

    On learning meaningful code changes via neural machine translation,

    M. Tufano, J. Pantiuchina, C. Watson, G. Bavota, and D. Poshyvanyk, “On learning meaningful code changes via neural machine translation,” in 2019 IEEE/ACM 41st International Conference on Software Engi- neering (ICSE), 2019, pp. 25–36

    2019

  4. [12]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  5. [13]

    Learning internal representations by error propagation,

    D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning internal representations by error propagation,” California Univ San Diego La Jolla Inst for Cognitive Science, Tech. Rep., 1985

    1985

  6. [14]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021

    2021 arXiv

  7. [15]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le et al., “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732 , 2021

    2021 arXiv

  8. [16]

    Incoder: A generative model for code infilling and synthesis,

    D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, W.-t. Yih, L. Zettlemoyer, and M. Lewis, “Incoder: A generative model for code infilling and synthesis,” arXiv preprint arXiv:2204.05999, 2022

    2022 arXiv

  9. [17]

    Shortformer: Better language modeling using shorter inputs,

    O. Press, N. A. Smith, and M. Lewis, “Shortformer: Better language modeling using shorter inputs,” arXiv preprint arXiv:2012.15832, 2020

    2012 arXiv

  10. [18]

    Train short, test long: Attention with linear biases enables input length extrapolation,

    O. Press, N. Smith, and M. Lewis, “Train short, test long: Attention with linear biases enables input length extrapolation,” in International Conference on Learning Representations , 2022. [Online]. Available: https://openreview.net/forum?id=R8sQPpGCv0

    2022

  11. [19]

    Transformer-xl: Attentive language models beyond a fixed-length context,

    Z. Dai, Z. Yang, Y . Yang, J. Carbonell, Q. V . Le, and R. Salakhutdi- nov, “Transformer-xl: Attentive language models beyond a fixed-length context,” arXiv preprint arXiv:1901.02860 , 2019

    1901 arXiv

  12. [20]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, P. J. Liuet al., “Exploring the limits of transfer learning with a unified text-to-text transformer.” J. Mach. Learn. Res. , vol. 21, no. 140, pp. 1–67, 2020

    2020

  13. [21]

    A length-extrapolatable transformer,

    Y . Sun, L. Dong, B. Patra, S. Ma, S. Huang, A. Benhaim, V . Chaudhary, X. Song, and F. Wei, “A length-extrapolatable transformer,” arXiv preprint arXiv:2212.10554, 2022

    2022 arXiv

  14. [22]

    Github copilot your ai pair programmer,

    GitHub, “Github copilot your ai pair programmer,” 2023. [Online]. Available: https://copilot.github.com/

    2023

  15. [23]

    Roformer: Enhanced transformer with rotary position embedding,

    J. Su, Y . Lu, S. Pan, A. Murtadha, B. Wen, and Y . Liu, “Roformer: Enhanced transformer with rotary position embedding,” arXiv preprint arXiv:2104.09864, 2021

    2021 arXiv

  16. [24]

    Neural machine translation of rare words with subword units,

    R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” arXiv preprint arXiv:1508.07909, 2015

    2015 arXiv

  17. [25]

    On the naturalness of software,

    A. Hindle, E. T. Barr, M. Gabel, Z. Su, and P. Devanbu, “On the naturalness of software,” Communications of the ACM , vol. 59, no. 5, pp. 122–131, 2016

    2016

  18. [26]

    On the localness of software,

    Z. Tu, Z. Su, and P. Devanbu, “On the localness of software,” in Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering , 2014, pp. 269–280

    2014

  19. [27]

    When code completion fails: A case study on real-world completions,

    V . J. Hellendoorn, S. Proksch, H. C. Gall, and A. Bacchelli, “When code completion fails: A case study on real-world completions,” in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 2019, pp. 960–970

    2019

  20. [28]

    Code completion with statistical language models,

    V . Raychev, M. Vechev, and E. Yahav, “Code completion with statistical language models,” in Proceedings of the 35th ACM SIGPLAN Confer- ence on Programming Language Design and Implementation , 2014, pp. 419–428

    2014

  21. [29]

    Learning from examples to improve code completion systems,

    M. Bruch, M. Monperrus, and M. Mezini, “Learning from examples to improve code completion systems,” in Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering , 2009, pp. 213–222

    2009

  22. [30]

    The hidden cost of code completion: Understand- ing the impact of the recommendation-list length on its efficiency,

    X. Jin and F. Servant, “The hidden cost of code completion: Understand- ing the impact of the recommendation-list length on its efficiency,” in Proceedings of the 15th International Conference on Mining Software Repositories, 2018, pp. 70–73

    2018

  23. [31]

    Codefill: Multi-token code completion by jointly learning from structure and naming sequences,

    M. Izadi, R. Gismondi, and G. Gousios, “Codefill: Multi-token code completion by jointly learning from structure and naming sequences,” arXiv preprint arXiv:2202.06689 , 2022

    2022 arXiv

  24. [32]

    Learning autocompletion from real- world datasets,

    G. A. Aye, S. Kim, and H. Li, “Learning autocompletion from real- world datasets,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) . IEEE, 2021, pp. 131–139

    2021

  25. [33]

    Are deep neural networks the best choice for modeling source code?

    V . J. Hellendoorn and P. Devanbu, “Are deep neural networks the best choice for modeling source code?” in Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering , 2017, pp. 763–773

    2017

  26. [34]

    Towards a better code completion system by api grouping, filtering, and popularity-based ranking,

    D. Hou and D. M. Pletcher, “Towards a better code completion system by api grouping, filtering, and popularity-based ranking,” in Proceedings of the 2nd International Workshop on Recommendation Systems for Software Engineering, 2010, pp. 26–30

    2010

  27. [35]

    Combining program anal- ysis and statistical language model for code statement completion,

    S. Nguyen, T. Nguyen, Y . Li, and S. Wang, “Combining program anal- ysis and statistical language model for code statement completion,” in 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2019, pp. 710–721

    2019

  28. [36]

    How program history can improve code completion,

    R. Robbes and M. Lanza, “How program history can improve code completion,” in 2008 23rd IEEE/ACM International Conference on Automated Software Engineering . IEEE, 2008, pp. 317–326

    2008

  29. [37]

    Improving code completion with program history,

    ——, “Improving code completion with program history,” Automated Software Engineering, vol. 17, no. 2, pp. 181–212, 2010

    2010

  30. [38]

    Fast and memory-efficient neural code completion,

    A. Svyatkovskiy, S. Lee, A. Hadjitofi, M. Riechert, J. V . Franco, and M. Allamanis, “Fast and memory-efficient neural code completion,” in 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR). IEEE, 2021, pp. 329–340

    2021

  31. [39]

    Pythia: Ai- assisted code completion system,

    A. Svyatkovskiy, Y . Zhao, S. Fu, and N. Sundaresan, “Pythia: Ai- assisted code completion system,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019, pp. 2727–2735

    2019

  32. [40]

    A language model for statements of software code,

    Y . Yang, Y . Jiang, M. Gu, J. Sun, J. Gao, and H. Liu, “A language model for statements of software code,” in 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2017, pp. 682–687

    2017

  33. [41]

    Codegen: An open large language model for code with multi-turn program synthesis,

    E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y . Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,” arXiv preprint arXiv:2203.13474 , 2022

    2022 arXiv

  34. [42]

    Santacoder: don’t reach for the stars!

    L. B. Allal, R. Li, D. Kocetkov, C. Mou, C. Akiki, C. M. Ferrandis, N. Muennighoff, M. Mishra, A. Gu, M. Dey et al., “Santacoder: don’t reach for the stars!” arXiv preprint arXiv:2301.03988 , 2023

    2023 arXiv

  35. [43]

    Jungloid mining: helping to navigate the api jungle,

    D. Mandelin, L. Xu, R. Bod ´ık, and D. Kimelman, “Jungloid mining: helping to navigate the api jungle,” ACM Sigplan Notices, vol. 40, no. 6, pp. 48–61, 2005

    2005

  36. [44]

    Codegen: An open large language model for code with multi-turn program synthesis,

    E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y . Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,” in The Eleventh International Conference on Learning Representations , 2023. [Online]. Available: https://o...

    2023

  37. [45]

    Measuring coding challenge competence with apps,

    D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Songet al., “Measuring coding challenge competence with apps,” arXiv preprint arXiv:2105.09938 , 2021

    2021 arXiv

  38. [46]

    On the relation between position informa- tion and sentence length in neural machine translation,

    M. Neishi and N. Yoshinaga, “On the relation between position informa- tion and sentence length in neural machine translation,” in Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), 2019, pp. 328–338

    2019

  39. [47]

    Location Attention for Extrapolation to Longer Sequences,

    Y . Dubois, G. Dagan, D. Hupkes, and E. Bruni, “Location Attention for Extrapolation to Longer Sequences,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Online: Association for Computational Linguistics, Jul. 2020, pp. 403–413. [O...

    2020

  40. [48]

    The eos decision and length extrapolation,

    B. Newman, J. Hewitt, P. Liang, and C. D. Manning, “The eos decision and length extrapolation,” arXiv preprint arXiv:2010.07174 , 2020

    2010 arXiv

  41. [49]

    Shape: Shifted absolute position embedding for transformers,

    S. Kiyono, S. Kobayashi, J. Suzuki, and K. Inui, “Shape: Shifted absolute position embedding for transformers,” arXiv preprint arXiv:2109.05644, 2021

    2021 arXiv

  42. [50]

    Cape: Encoding relative positions with continuous augmented positional embeddings,

    T. Likhomanenko, Q. Xu, G. Synnaeve, R. Collobert, and A. Rogozh- nikov, “Cape: Encoding relative positions with continuous augmented positional embeddings,” Advances in Neural Information Processing Systems, vol. 34, pp. 16 079–16 092, 2021

    2021

  43. [51]

    Analysis of positional encodings for neural machine translation,

    J. Rosendahl, V . A. K. Tran, W. Wang, and H. Ney, “Analysis of positional encodings for neural machine translation,” in Proceedings of the 16th International Conference on Spoken Language Translation , 2019

    2019

  44. [52]

    Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks,

    B. Lake and M. Baroni, “Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks,” in International conference on machine learning. PMLR, 2018, pp. 2873– 2882

    2018

  45. [53]

    Compositionality decomposed: How do neural networks generalise?

    D. Hupkes, V . Dankers, M. Mul, and E. Bruni, “Compositionality decomposed: How do neural networks generalise?” Journal of Artificial Intelligence Research, vol. 67, pp. 757–795, 2020

    2020

  46. [54]

    Sampling projects in github for MSR studies,

    O. Dabic, E. Aghajani, and G. Bavota, “Sampling projects in github for MSR studies,” in 18th IEEE/ACM International Conference on Mining Software Repositories, MSR 2021 . IEEE, 2021, pp. 560–564

    2021

  47. [55]

    Unit test case generation with transform- ers,

    M. Tufano, D. Drain, A. Svyatkovskiy, S. K. Deng, and N. Sundaresan, “Unit test case generation with transform- ers,” CoRR, vol. abs/2009.05617, 2020. [Online]. Available: https://arxiv.org/abs/2009.05617

    2009 arXiv

  48. [56]

    Adam: A method for stochastic optimization,

    D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” International Conference on Learning Representations , 12 2014

    2014

  49. [57]

    X-transformers,

    P. Wang, “X-transformers,” GitHub, 2023. [Online]. Available: https://github.com/lucidrains/x-transformers

    2023

  50. [58]

    PyTorch Lightning,

    W. Falcon and The PyTorch Lightning team, “PyTorch Lightning,” 3

  51. [59]

    The curious case of neural text degeneration,

    A. Holtzman, J. Buys, L. Du, M. Forbes, and Y . Choi, “The curious case of neural text degeneration,” arXiv preprint arXiv:1904.09751 , 2019

    1904 arXiv

  52. [60]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics , 2002, pp. 311–318

    2002

  53. [61]

    chrf: character n-gram f-score for automatic mt evalu- ation,

    M. Popovi ´c, “chrf: character n-gram f-score for automatic mt evalu- ation,” in Proceedings of the Tenth Workshop on Statistical Machine Translation, 2015, pp. 392–395

    2015

  54. [62]

    Binary codes capable of correcting deletions, insertions, and reversals,

    V . I. Levenshtein et al., “Binary codes capable of correcting deletions, insertions, and reversals,” in Soviet physics doklady , vol. 10, no. 8. Soviet Union, 1966, pp. 707–710

    1966

  55. [63]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” in Text summarization branches out , 2004, pp. 74–81

    2004

  56. [64]

    Meteor: An automatic metric for mt evalua- tion with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evalua- tion with improved correlation with human judgments,” in Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization , 2005, pp. 65–72

    2005

  57. [65]

    Codexglue: A machine learning benchmark dataset for code understanding and generation,

    S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang et al., “Codexglue: A machine learning benchmark dataset for code understanding and generation,” arXiv preprint arXiv:2102.04664 , 2021

    2021 arXiv

  58. [66]

    Out of the bleu: how should we assess quality of the code generation models?

    M. Evtikhiev, E. Bogomolov, Y . Sokolov, and T. Bryksin, “Out of the bleu: how should we assess quality of the code generation models?” arXiv preprint arXiv:2208.03133 , 2022

    2022 arXiv

  59. [67]

    Cooper and R

    N. Cooper and R. Tufano. Replication package. [Online]. Available: https://github.com/RosaliaTufano/completeformer

  60. [68]

    Datasets: A community library for natural language processing,

    Q. Lhoest, A. Villanova del Moral, Y . Jernite, A. Thakur, P. von Platen, S. Patil, J. Chaumond, M. Drame, J. Plu, L. Tunstall, J. Davison, M. ˇSaˇsko, G. Chhablani, B. Malik, S. Brandeis, T. Le Scao, V . Sanh, C. Xu, N. Patry, A. McMillan-Major, P. Schmid, S. Gugger, C. Delan...

    2021

  61. [69]

    The adverse effects of code duplication in machine learning models of code,

    M. Allamanis, “The adverse effects of code duplication in machine learning models of code,” in Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software , 2019, pp. 143–153

    2019

  62. [70]

    Codebleu: a method for automatic evaluation of code synthesis,

    S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma, “Codebleu: a method for automatic evaluation of code synthesis,” arXiv preprint arXiv:2009.10297 , 2020

    2009 arXiv

  63. [2019]

    Available: https://github.com/Lightning-AI/lightning

    [Online]. Available: https://github.com/Lightning-AI/lightning

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.