{"id":"19df3ece-8917-403c-8155-e3f33056f305","arxiv_id":"2501.05051","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Across two languages and three metrics, no tested positional encoding scheme generalizes to code completion lengths unseen in training; mixed-length training is the recommended safe choice.","lead":"This paper tests whether four positional encoding schemes let code-completion models trained on short code handle longer code at inference time. It finds none of them generalize, and that training on a mixture of lengths is the only safe strategy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Length buckets are confounded with code content and target difficulty, so the claimed failure of positional encodings to generalize to unseen lengths is not isolated.","rationale":"The reader's weakest assumption identifies the same confound: holding the masked token count at 11 does not isolate input length because the short/medium/long buckets differ in content and difficulty. I agree with this assessment and consider it the most load-bearing concern because the paper's headline conclusion is a mechanistic statement about positional encodings' length-extrapolation ability. The practical conclusion that training on short functions does not transfer to long-function test sets may survive this concern, but the attribution to length per se does not. Other weaknesses, such as missing error bars and the broad wording of the abstract, are important but secondary: the magnitude of the drops makes noise an unlikely full explanation, whereas the dataset-construction design directly threatens the construct validity of 'unseen length' as the manipulated variable. The proposed matched-content control would settle the question by varying input length while holding target content fixed. Since the reader's verdict is already CONDITIONAL and this concern reinforces rather than overturns it, no change to the verdict is needed.","tokens_in":19827,"tokens_out":6087,"duration_ms":64212,"concrete_test":"Construct a matched-content control from the existing short test set: for each short instance, generate a 'lengthened' variant by prefixing the code with K syntactically valid, semantically neutral filler statements (e.g., no-op assignments) so that the input length falls in the medium/long ranges, while the masked target and its immediate context stay unchanged. Evaluate the models trained on the short dataset (all four encodings, both languages) on the original and lengthened versions. If EM/ChrF/RougeL on lengthened inputs remain close to the original short-test scores, the observed degradation is attributable to content shift rather than input length; if they drop substantially, the paper's length-generalization conclusion is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—'none of the studied positional encoding schemes has the ability to generalize to unseen lengths' (Section VII)—rests on the assumption that the short/medium/long datasets differ only in input length. Section IV-A builds the datasets by sorting functions by total token count and splitting into thirds (Java: 6–96, 97–180, 181–1024; Python: 30–150, 151–309, 310–1024). This ensures that longer functions differ systematically in properties correlated with length: statement vocabulary, nesting depth, control-flow constructs, and target-statement difficulty. Because the completion target is the last 11 tokens of a statement/block drawn from that function, the target distribution is not held constant across buckets. Thus, a model trained on short functions is also trained on a different distribution of completions. The observed performance drops in Tables II–IV could reflect domain shift in code content and target difficulty, not a failure of sinusoidal/xPOS/ALiBi/T5 to extrapolate to unseen input lengths. Holding the number of masked tokens at 11 (Section IV-A) controls task complexity only in a narrow sense; it does not control for what the 11 tokens are. The paper therefore overstates the mechanistic conclusion: it demonstrates that short-trained models do not transfer to long-function test sets, but it does not isolate length as the cause.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript empirically investigates whether four positional encoding schemes (Sinusoidal, xPOS, ALiBi, and T5 relative bias) enable encoder-decoder Transformers to generalize to code completions of lengths not seen during training. The authors construct short, medium, long, and mixed datasets from Java (statement-level completion) and Python (block-level completion) functions, holding the number of masked tokens fixed at 11 and equalizing training set sizes across conditions. They train 32 models (4 schemes × 4 datasets × 2 languages) and evaluate them on held-out short/medium/long test sets using exact match, ChrF, and RougeL. The primary finding is that all schemes exhibit large performance degradation when training and test lengths differ, with T5 being the best-performing and least-degraded scheme; a secondary finding is that training on a mixture of lengths is a safer compromise than single-length training.","tokens_in":20085,"tokens_out":12825,"duration_ms":112688,"significance":"If the findings hold, this is a valuable negative result for the software engineering community: length-extrapolation positional encodings that work in some NLP decoder-only settings do not automatically transfer to encoder-decoder code completion, and practitioners must include representative lengths in training data. The study is carefully set up in several respects: duplicate removal, equal training sizes, a fixed masked-token count, and a shared hyperparameter configuration; the release of code, data, and checkpoints is a concrete asset. The main caveat is that the central 'length generalization' claim rests on the assumption that the length buckets differ only in input length, which is threatened by a content confound (Major Comment 1). The value of the paper therefore depends on either additional control analyses or a more carefully scoped conclusion.","major_comments":[{"comment":"The dataset construction does not actually isolate input length as the only changed variable. The authors sort functions by total token count and split into thirds, then mask the last 11 tokens of a statement or block in each function. While the number of masked tokens is held constant, the content of those 11 tokens and the surrounding code are not controlled; longer functions are likely to contain systematically different statements (e.g., more complex expressions, deeper nesting, different control-flow constructs) and thus a different target distribution. Consequently, the degradation observed in Tables II-IV when training on one length bucket and testing on another could be caused by a shift in code content and target difficulty, rather than by a failure of the positional encodings to extrapolate to unseen input lengths. This directly undermines the central claim in Section VII that 'none of the studied positional encoding schemes has the ability to generalize to unseen lengths.' I recommend either softening the conclusion to a statement about transfer across length-correlated code distributions, or adding a control analysis that matches the target statements across length buckets (e.g., by statement type, AST depth, or target vocabulary) to isolate the effect of input length.","section":"Section IV-A"},{"comment":"The paper presents only point estimates from a single training run per configuration, with no seed variance, error bars, or statistical significance tests. This is a problem for the finer-grained comparative claims, such as 'T5's positional encoding scheme ... is also better at generalizing to unseen lengths' (Section V) and the relative deltas in the mix tables (Tables V-VII), where some differences are small (e.g., Table VI, Java xPOS short, Δ = +1.16%). The main degradation effect is large and consistent, but the ranking and 'safe compromise' conclusions require a measure of variability to be credible. Please report multiple seeds for at least the main configurations, or provide bootstrap confidence intervals on the test sets.","section":"Section IV-B and Tables II-VII"},{"comment":"The paper repeatedly refers to 'Large Language Models' and states general conclusions about 'Transformer models', but the experiments train small encoder-decoder Transformers from scratch (Table I: d_model=512, 6-8 layers; 280k training instances per dataset). The negative result may not transfer to large-scale pretrained code models (e.g., CodeT5, CodeGen, StarCoder) that use different training objectives, much larger corpora, and different decoding strategies. The conclusion should be explicitly scoped to the studied architecture, model size, and training setup, and the external-validity section should acknowledge this limitation.","section":"Abstract and Section VII"},{"comment":"The recommendation that training on a mixture of lengths 'might be the safest choice in most of cases' (Take Away #3) is not fully supported by the Python results. On Python, the mix-trained models are substantially worse than the length-matched models, with relative exact-match drops as large as -44.71% for T5 on the short test set (Table V), and similarly large drops for Sinusoidal and ALiBi. The paper acknowledges that 'training diversity helped less in Python,' but the recommended strategy still carries a large penalty for the best-performing scheme (T5) on one of the two languages. Please either quantify when mixing is beneficial versus harmful (e.g., by language, scheme, or test length) or soften the take-away accordingly.","section":"Section V (Impact of training diversity) and Tables V-VII"}],"minor_comments":[{"comment":"The sentence 'We investigated two Transformer architectures to mitigate the threats to external validity' is factually incorrect; the study uses one sequence-to-sequence Transformer architecture with four positional encoding variants. It should refer to two programming languages or two completion tasks.","section":"Section VI"},{"comment":"There are several typos: 'Sonusoidal' should be 'Sinusoidal', 'syntaxt' should be 'syntax', and the abbreviation 'ChfR' is used inconsistently for 'ChrF'.","section":"Section V"},{"comment":"The enumerated list of data-collection steps skips item (v), going from (iv) directly to (vi); this is a numbering error.","section":"Section IV-A"},{"comment":"The text states 'the deltas in Tables V, III, and VII' when referring to the mix results; this should be 'Tables V, VI, and VII.'","section":"Section V (Impact of training diversity)"},{"comment":"The sentence 'This may end up in generating duplicates due to different functions from which we masked the only part being different' is unclear and should be rephrased.","section":"Section IV-A"},{"comment":"The use of nucleus sampling (topp=0.95) for evaluation is not standard for exact-match measurement; since the study reports EM as a key metric, the authors should also provide greedy-decoding results to rule out sampling-induced variance.","section":"Section IV-B"},{"comment":"The black-box, bold, and red highlighting described in Section V are not visible in the manuscript text provided; please ensure the visual encoding is clear in the camera-ready version.","section":"Tables II-IV"},{"comment":"The paper uses 'LLM' to refer both to large pretrained models and to the small models trained here; this conflation should be corrected throughout, e.g., by using 'Transformer models' for the experiments.","section":"Introduction"}],"recommendation":"major_revision","confidential_remarks":"The paper is a well-executed empirical study with a useful benchmark and replication package, but the main mechanistic claim is not yet supported due to the length-content confound in the dataset construction. If the authors can provide a matched control or significantly soften the claim, the paper may be acceptable for publication. The scope mismatch with the term 'LLMs' and the lack of error bars are further concerns that the editorial process should weigh."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — the thing to know: this paper gives a clear negative answer to a practical question. For encoder-decoder code completion, training on short functions does not transfer to long-function test sets for Sinusoidal, xPOS, ALiBi, or T5. That is worth knowing, and the finding is consistent across Java and Python, across EM/ChrF/RougeL, and across all four schemes.\n\nWhat is new: the NLP literature on length extrapolation mostly targets decoder-only models, and this is the first controlled comparison of these four encoding schemes in encoder-decoder code completion on real GitHub-derived data. The dataset construction is careful—deduplication, equal training sizes, fixed eleven masked tokens, function-level splits, shared hyperparameters—and they ship code, data, checkpoints, and additional results. That is real evidence and deserves credit.\n\nWhere the soft spots are, in order. First, the length buckets are confounded with code content and target difficulty. Sorting functions by token count and splitting into thirds makes short, medium, and long differ in more than input length. The masked eleven tokens are drawn from different statements or blocks, so longer functions plausibly bring different syntax, vocabulary, and difficulty. The paper therefore demonstrates that short-trained models do not transfer to long-function test sets, but it does not isolate unseen length as the mechanism. The abstract and Section VII overstate the mechanistic conclusion; I would soften the claim to \"training on short code does not transfer to longer code in these settings,\" which is still practically important. Second, each model is trained once. No seed variance, no error bars. The external two-point significance heuristic is cited, but without run variance that is only a heuristic. The effects are large and monotone, so I doubt the main finding flips, but the confidence language should be quieter. Third, the mix-training results on Python are messier than the abstract suggests: T5 trained on the mix loses a lot relative to specialized training on every Python test set. They acknowledge this in the text, but the \"only safe solution\" framing is too strong.\n\nThe stress-test concern is real. It does not sink the paper, because the practical question—can I train on short examples and save cost—does not require isolating mechanism; the answer is no either way. But it does mean the paper is stronger as engineering guidance than as a statement about positional encodings and length generalization.\n\nBottom line: serious, reproducible empirical work, useful to DL4SE researchers and anyone building code-completion models. Send it to peer review, and ask for softened claims, seed-variance or repeated-run evidence, and an explicit discussion of the content confound.","headline":"A useful negative empirical result—training on short code does not transfer to longer code for any tested positional encoding—but the abstract and conclusion overstate the mechanism because length buckets are confounded with code content and target difficulty.","tokens_in":20572,"tokens_out":2243,"would_cite":true,"duration_ms":26731,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that none of the four studied positional encoding schemes lets encoder-decoder Transformers generalize code completion to input lengths unseen during training.","keywords":["code completion","length generalization","positional encoding","transformer","encoder-decoder","ALiBi","xPOS","empirical study"],"falsifier":"A concrete test would be to pad short functions with harmless filler tokens up to long lengths while keeping the 11 masked tokens identical: if any scheme keeps close-to-training performance on these padded inputs, the observed degradation is about length itself; if performance still collapses, dataset-content shift, not positional encoding, explains the paper's conclusion. Alternatively, finding one trained model in the released checkpoints that attains near-matched Exact Match on an unseen-length test set would directly refute the claim that none generalizes.","tokens_in":19672,"feed_emoji":"⚙️","tokens_out":5542,"duration_ms":53006,"temperature":0.7,"pith_summary":"The paper asks whether code-completion Transformers can be trained on short code excerpts and still perform well on longer excerpts at inference, a property that would substantially cut training cost. It tests four positional encoding schemes—Sinusoidal, xPOS, ALiBi, and T5—on Java and Python completion tasks, holding the number of masked tokens fixed at 11 so that only input length changes. Across 32 trained models, every scheme degrades substantially when training and test lengths differ, and the degradation runs in both directions: short-trained models fail on long inputs and long-trained models fail on short inputs. The authors conclude that none of the studied schemes generalizes to unseen lengths and that including a representative mixture of lengths in training is the only reliable safeguard.","feed_headline":"None of 4 positional encodings generalizes to unseen lengths","feed_subtitle":"Across 32 models and two languages, all four encoding schemes degrade sharply when training and test lengths differ.","key_machinery":"The study's object is the sequence-to-sequence Transformer and four ways of injecting position information. Sinusoidal adds fixed sine and cosine position vectors to token embeddings at the network input. ALiBi adds a static, head-specific linear bias to attention scores so distant keys are down-weighted. T5 adds a learned, bucketed relative-position bias to attention. xPOS combines rotary relative-position embeddings with an exponential decay on attention resolution, with the paper using the Rotary encoder and the xPOS decoder. Each scheme is trained on function-length datasets labeled short, medium, long, and mix, with task complexity held constant by always masking 11 tokens, and evaluated by Exact Match, ChrF, and Rouge-L.","core_discovery":"The central empirical claim is that, for code completion in an encoder-decoder Transformer, none of the studied positional encoding schemes—Sinusoidal, xPOS, ALiBi, or T5—generalizes to input lengths outside the training range. The evidence is consistent across languages and metrics: for example, a Sinusoidal model trained on short Java functions reaches 10.81 percent Exact Match on short tests but only 2.91 percent when trained on medium functions and 0.50 percent when trained on long ones; T5, the best scheme, drops from 17.03 percent to 3.62 percent on long tests when trained on short instead of long inputs. The paper interprets this as length-specific behavior: attention mechanisms that extrapolate in decoder-only NLP settings do not transfer their length-extrapolation benefit to encoder-decoder code completion. The authors therefore conclude that there is no shortcut: training data should represent the lengths expected at inference time.","pith_inferences":["An implication the authors leave implicit is that length generalization in decoder-only language models may depend on the task and architecture, not just on the attention bias; their negative result suggests encoder-decoder code completion is a harder extrapolation setting than language modeling.","A natural next experiment is to pad short functions to long lengths with neutral tokens while keeping the 11 masked tokens identical; if performance stays high, the failure is purely positional, whereas a drop would indicate that long functions differ in content or style, not just length.","The result also implies that cost-savings claims for training on short data should be re-examined for code: the real saving may require mixed-length or length-matched training, which removes much of the efficiency advantage."],"forward_implications":["The train-short, test-long cost-saving strategy is not available with any of these encoding schemes for encoder-decoder code completion.","Models trained exclusively on short functions collapse on long completions, and models trained only on long functions collapse on short ones, so length-specific specialization is brittle.","Training on a mixture of lengths is the safe configuration: it avoids most of the degradation, even though it can give up a few points against a perfectly matched training length.","T5's learned relative bias is the strongest of the four in absolute performance and in relative drop, but it still fails the generalization test."],"supporting_citations":[{"why":"Supplies the base sequence-to-sequence Transformer architecture, sinusoidal positional encoding, and the hyperparameters used for all models.","marker":"[12]"},{"why":"Introduces ALiBi and the train-short, test-long extrapolation claim that the study tests on code completion.","marker":"[18]"},{"why":"Introduces the T5 learned relative-position bias, the best-performing scheme in the study.","marker":"[20]"},{"why":"Introduces xPOS as a length-extrapolatable transformer and provides the decoder-side extension used in the experiments.","marker":"[21]"},{"why":"Introduces rotary position embedding, which forms the encoder-side component of the xPOS scheme.","marker":"[23]"},{"why":"Prior empirical study of T5 for code completion that motivates the task setup and the statement-level and block-level completion scenarios.","marker":"[2]"},{"why":"Provides the two-point significance threshold for ChrF and Rouge-L used to claim differences between encoding schemes.","marker":"[66]"},{"why":"Provides the open-source project sampling method used to build the Java and Python function datasets.","marker":"[54]"}],"fun_headline_variants":["No positional encoding generalizes to unseen code lengths","All four positional encodings fail on unseen lengths","Length extrapolation fails for code completion models","Code completion can't handle sequences of unseen lengths"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that holding the masked token count at 11 isolates input length as the only changed variable, so that short, medium, and long functions differ in length rather than in content, style, or difficulty.","fun_headline_variants_meta":{"raw":{"variants":["No positional encoding generalizes to unseen code lengths","All four positional encodings fail on unseen lengths","Length extrapolation fails for code completion models","Code completion can't handle sequences of unseen lengths"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000646,"raw_usage":{"total_tokens":3020,"prompt_tokens":1047,"completion_tokens":1973,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":663,"completion_tokens_details":{"reasoning_tokens":1914}},"tokens_in":663,"tokens_out":1973,"duration_ms":16925,"temperature":1.0,"reasoning_tokens":1914,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:19:04.571096+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test would be to pad short functions with harmless filler tokens up to long lengths while keeping the 11 masked tokens identical: if any scheme keeps close-to-training performance on these padded inputs, the observed degradation is about length itself; if performance still collapses, dataset-content shift, not positional encoding, explains the paper's conclusion. Alternatively, finding one trained model in the released checkpoints that attains near-matched Exact Match on an unseen-length test set would directly refute the claim that none generalizes.","supporting_citations":[{"cited_title":"Train short, test long: Attention with linear biases enables input length extrapolation,","cited_arxiv_id":null,"evidence_quote":"Introduces ALiBi and the train-short, test-long extrapolation claim that the study tests on code completion."},{"cited_title":"An empirical study on the usage of transformer models for code completion,","cited_arxiv_id":null,"evidence_quote":"Prior empirical study of T5 for code completion that motivates the task setup and the statement-level and block-level completion scenarios."},{"cited_title":"Sampling projects in github for MSR studies,","cited_arxiv_id":null,"evidence_quote":"Provides the open-source project sampling method used to build the Java and Python function datasets."}],"review_version":1}