{"id":"d3a1d225-69be-45c3-8b22-84326d9a2fab","arxiv_id":"2412.08063","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Feeding code-completion models a developer's recent file history, similar code, and key symbol definitions raised completion acceptance rates in internal ByteDance deployments by an estimated 15 percent.","lead":"ContextModule improves AI code completion by feeding the model three extra clues from the developer's repository: recent browsing and editing history, similar code, and definitions of key functions and structures. In internal ByteDance tests the additions raised completion acceptance by an estimated 15 percent or more.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Offline metric may be self-confirming: SEM as defined in §IV.A.3 makes correctness conditional on the retrieved context, and the user-behavior filter in §IV.B.1 may encode that same dependence; Tables II–IV would then reflect the scoring definition rather than an independent gain.","rationale":"The central claim is that repository-level context causes improvement in code completion. The cleanest causal evidence would be a metric that measures completion quality independently of the intervention, but SEM as published does not provide that: introducing \"the retrieved context helps\" into the correctness predicate makes the metric ambiguous at best and circular at worst. The user-behavior dataset's pre-filtering compounds this, because selecting only cases where the final code was judged correct before evaluating the no-context baseline is acceptable only if the judgment is made blind to the retrieved context. If it is not, the baseline is evaluated on a set enriched for cases where context is useful. I am not alleging fraud; the issue is that the paper's own text does not rule out the circular reading, and the datasets, code, and CKG are proprietary, so the reader cannot verify. The online acceptance-rate improvements are self-reported, sequential, and the combined >15% gain is explicitly an estimate, but those are weaker evidence rather than internally contradictory. The proposed context-blind rescoring would settle whether the offline gains are real; if they survive, the central claim has meaningful support, and if they collapse, the offline component of the claim fails. This keeps the verdict conditional rather than accept or reject outright, matching the reader's assessment.","tokens_in":11550,"tokens_out":5620,"duration_ms":59321,"concrete_test":"Rescore Tables II, III, and IV with a context-blind SEM: define a prediction as correct iff the ground-truth completion appears as a prefix of the generated output after stripping trailing tokens and whitespace (or use a fixed edit-similarity threshold), with the scorer never inspecting the retrieved context. If the reported deltas (e.g., +1.65–2.55 SEM for user behavior, +2.6–10 SEM for CKG-based symbols) persist under this criterion, the metric concern is resolved. Independently, rebuild the §IV.B.1 user-behavior dataset from raw telemetry with annotators blinded to the retrieved user-behavior snippets; if the filter cannot be applied blind, or if applying it blind removes the reported gain, then the user-behavior improvement is an artifact of the construction procedure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is that the primary offline metric, soft exact match (SEM), is defined in §IV.A.3 as counting a result correct \"as long as the retrieved context helps the model generate the correct answer.\" This wording makes correctness conditional on the very intervention being evaluated: if the scorer uses knowledge of whether retrieved context was present (for example, by checking whether the context contains the gold tokens, or by requiring the context to be necessary for the prediction to match), then any \"with context\" condition is scored against a standard that already assumes context is useful. The same circularity appears in the user-behavior dataset construction in §IV.B.1, where samples are pre-filtered to those in which the final code was judged \"correct\" via rule-based filtering and manual annotation, with no annotation protocol or inter-annotator agreement reported. If correctness was assessed with access to the retrieved user-behavior snippets, the no-context baseline is evaluated only on cases where context demonstrably helps, artificially inflating the reported 1.65–2.55 point SEM gains in Table II. This is not merely a benchmark-quality issue; it determines whether Tables II–IV measure an independent property of code completion or re-encode the treatment assignment. The online acceptance-rate numbers are less exposed to this particular flaw, but they cannot rescue the offline causal claim if the offline metric is definitionally circular.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ContextModule is an industry-oriented framework that augments LLM-based code completion with three repository-level context sources: user behavior code (recently browsed/edited snippets), similar code snippets retrieved by token-based similarity, and symbol definitions from a proprietary code knowledge graph (CKG). The paper describes retrieval strategies, latency optimizations (index caching, incremental parsing), offline evaluations on internal Go/Python/TypeScript datasets using a single base model (DeepSeek-Coder-6.7b-Base), and production acceptance-rate improvements of 7.1%, 6.3%, and 4.9% for the three strategies. The central claim is that repository-level context significantly improves completion accuracy and user acceptance.","tokens_in":11812,"tokens_out":5286,"duration_ms":52129,"significance":"If the results hold, ContextModule is a practical, deployable contribution to code completion in real IDEs, with useful engineering artifacts: a low-latency token-based retrieval pipeline, an incremental CKG for symbol definitions, and a production deployment with measured acceptance gains. The paper credibly addresses a real gap—most academic code-completion evaluations do not capture cross-file user behavior or industrial latency constraints. The strengths are the production evidence, the multi-language offline setup, and the explicit treatment of retrieval latency. However, the evaluative apparatus has load-bearing weaknesses, principally the vague and potentially circular definition of the primary metric 'soft exact match' and the construction of the user-behavior test set. These need to be resolved before the offline causal claims can be accepted.","major_comments":[{"comment":"The definition of 'soft exact match' is load-bearing for Tables II–IV but is not operationalizable as written: the paper says a result is correct 'as long as the retrieved context helps the model generate the correct answer.' This makes correctness conditional on the intervention being evaluated; if the scorer uses knowledge of whether retrieved context was present or was helpful, then the with-context condition is scored against a different standard than the no-context condition, and the reported gains partly reflect the scoring rule. Please give the exact algorithm (e.g., whether SEM checks that the prediction contains the gold label after stripping trailing tokens, and whether the check is symmetric across conditions), and report both exact match and edit similarity for all conditions.","section":"IV.A.3"},{"comment":"The user-behavior dataset is constructed by selecting instances where 'the generated code was correct' via rule-based filtering and manual annotation, but no annotation protocol or inter-annotator agreement is reported. If annotators had access to the retrieved user-behavior snippets, or if the correctness filter uses the same soft-exact-match criterion as the main metric, the no-context baseline is evaluated only on samples where context demonstrably helps, artificially inflating the gains in Table II. The ground truth also appears to be the final code at the completion point, which may be the model's own accepted suggestion; using it as supervision for the same model can reward reproducing model output. Please describe the annotation interface (whether context was visible), report inter-annotator agreement, and verify that the correctness filter is independent of the retrieval strategies.","section":"IV.B.1"},{"comment":"The online acceptance-rate improvements (7.1%, 6.3%, 4.9%) are reported for strategies implemented sequentially, with no control group, no experiment period, no population definition, and no adjustment for time-varying factors; the authors themselves note the indicators 'could not fully represent the effectiveness.' The subsequent claim that the combined improvement 'could achieve a relative increase of more than 15%' is an unsupported estimate. Please report the online experimental design (e.g., randomized rollout with holdout users, or at least pre/post acceptance rates with confounder controls) and present an uncertainty/range if the 15% figure is retained.","section":"V.D"},{"comment":"Effect sizes are small (1.65–2.55 SEM points, 1.8–2.4 edit-similarity points) and no error bars, confidence intervals, or significance tests are provided, despite 1500 samples per language. With a single base model (DeepSeek-Coder-6.7b-Base) and internal, unreleased datasets, the external validity of the 'significant improvement' claim is not established. Please add variance estimates or significance tests across samples, and ideally across multiple base models or at least justify the single-model choice.","section":"Tables II–IV"}],"minor_comments":[{"comment":"The table header 'Function Declaration' is inconsistent with the text in §IV.B.3 and §V.C, which both refer to 'function definition'; please unify the terminology.","section":"Table IV"},{"comment":"There is a typo in the sentence 'we sorte and filtered the results'; it should be 'sorted'.","section":"V.B.1"},{"comment":"The acronym SEM is introduced without being explicitly attached to 'soft exact match' in the text; please define it at first use and use it consistently throughout.","section":"IV.A.3"},{"comment":"The figure caption reads 'FrameWork of ContextModule'; the capitalization should be fixed to 'Framework'.","section":"Figure 1"},{"comment":"The LSP limitations are listed qualitatively but no measured latency or accuracy comparison is given; a small table comparing LSP vs. CKG retrieval times and hit rates would strengthen the motivation.","section":"III.C.1"},{"comment":"The statement 'We did not conduct offline experiments on context fusion' is a limitation; it should be acknowledged explicitly and, if possible, receive a short offline sanity check even on a subsample.","section":"V.D"}],"recommendation":"major_revision","confidential_remarks":"The paper is an industry contribution from ByteDance, and its evaluation rests entirely on internal, unreleased datasets and a proprietary CKG tool. The SEM metric's definition is sufficiently vague that it currently reads as self-confirming; this should be the central focus of the revision. I would advise the editor to ask for a precise metric specification, a transparent description of dataset construction (including annotation conditions), and either a proper online evaluation design or a clear statement of the limitations of the sequential deployment numbers. The engineering and deployment aspects are valuable, but the evidence as presented does not yet meet the bar for a strong archival claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this one with the stress-test note in hand. The concern lands. Section IV.A.3 defines soft exact match as counting a result correct \"as long as the retrieved context helps the model generate the correct answer.\" If the scorer has access to whether context was present or can see gold tokens in the context, then the \"with context\" condition is scored against a standard that already assumes the context is useful. Section IV.B.1's user-behavior filter has the same flavor: samples are kept only when the final code was judged correct, with no annotation protocol or agreement numbers. If correctness was assessed with the retrieved snippets visible, the no-context baseline is evaluated only on cases where context demonstrably helps. That would inflate Tables II and III.\n\nWhat is solid: the system itself is a legitimate engineering contribution. The user-behavior retrieval channel is the most distinctive piece and is not in RepoCoder or RepoHyper. The combination of three channels with caching and latency control is new as a system. The online acceptance-rate numbers are real-world evidence: sequential gains of 7.1%, 6.3%, and 4.9% are reported, and the authors honestly note the overlap and lack of separate ablations. That honesty counts. The case studies are illustrative.\n\nWhat is soft beyond the metric: datasets and CKG are proprietary and unreleased; only one base model is tested; there are no error bars or significance tests; there is no comparison to RepoCoder, RepoHyper, or other public baselines; the combined >15% online gain is an estimate. These are real but mostly fixable with artifacts and a stricter evaluation.\n\nWhether the central claim holds up: I think the direction is very likely right, and the online numbers are independent evidence that the system helps. But the offline causal claim is not established as written.\n\nThis paper is for researchers and practitioners building repository-level completion pipelines, especially in industry settings. A serious editor should send it to review rather than desk reject: the system contribution and production evidence are worth referee attention, and the eval issues are exactly what referees should push on.\n\nRecommendation: engage, but require the authors to release code/data or accept independent validation, re-define or drop SEM, report per-seed variance or significance, and add a public-baseline comparison.","headline":"A plausible, practically useful repository-context system whose offline evidence is weakened by a potentially self-confirming metric and unreleased data, but whose online acceptance numbers make it worth refereeing.","tokens_in":12392,"tokens_out":1947,"would_cite":false,"duration_ms":19911,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ContextModule claims that adding repository-level context—user-behavior code, similar snippets, and symbol definitions—to the completion prompt substantially improves accuracy and acceptance, with combined production gains above 15%.","keywords":["code completion","repository context","user behavior","code knowledge graph","similar code retrieval","large language models","low latency","production deployment"],"falsifier":"Run a production A/B test that toggles exactly one of the three context strategies at a time while holding the model and prompt template fixed; if removing user behavior code, similar code, or symbol definitions does not lower user acceptance, the central claim fails. A complementary offline check is to recompute the metrics with a strict exact-match or edit-distance criterion that does not count 'retrieved context helped' as a correct answer and see whether the improvements persist.","tokens_in":11307,"feed_emoji":"⌨️","tokens_out":6866,"duration_ms":59034,"temperature":0.7,"pith_summary":"This paper seeks to show that a code completion system can be improved substantially by retrieving repository-level context and injecting it into the prompt. It proposes three retrieval strategies: user behavior code, drawn from the developer's recent cross-file browsing and editing history; similar code snippets, found by token-based similarity search across the repository; and symbol definitions, served from a code knowledge graph built by static analysis. Offline experiments on Python, Go, and TypeScript show each strategy raises completion accuracy, and a production deployment reports relative acceptance-rate gains of 7.1%, 6.3%, and 4.9% from the three strategies applied sequentially, with an estimated combined gain above 15%. A sympathetic reader would care because these gains come without retraining the model and are paired with caching and incremental-parsing optimizations that keep retrieval within production latency budgets.","feed_headline":"Repository context lifts code-completion acceptance by over 15%","feed_subtitle":"User browsing history, similar code, and symbol definitions each add gains in a production LLM code-completion engine.","key_machinery":"The machinery is a retrieval-and-concatenation pipeline. User behavior code: the IDE records cursor clicks, file paths, and line numbers; the five most recently browsed files are split into sliding-window snippets, and the top snippets by click count are added to the prompt. Similar code: repository files are split into sliding-window snippets, tokenized with camel/snake-case splitting and stop-word removal, and ranked by Jaccard similarity to the code near the cursor; an index cache with a BFS file-ordering strategy keeps retrieval latency below 80 milliseconds. Symbol definitions: a code knowledge graph built by static analysis stores function signatures and struct definitions; incremental parsing detects function-call, struct-initialization, and function-body completion positions and retrieves the relevant symbol in under 50 milliseconds. The three retrieved blocks are concatenated in the order symbol, similar code, user behavior code, together with the current file's prefix and suffix, in fill-in-the-middle format.","core_discovery":"The paper's central claim is that repository-level context, captured in three forms, materially improves LLM-based code completion. The three forms are user behavior code (snippets from files the developer recently browsed or edited), similar code snippets (retrieved by text similarity from the rest of the repository), and CKG-based symbol definitions (function signatures and struct definitions from a code knowledge graph built by static analysis). On offline datasets for Python, Go, and TypeScript, each context type raises soft exact match and edit similarity; the largest single gain is in struct initialization, where soft exact match rises from 6.17 to 17.41 when symbol definitions are added. In production, the three strategies were rolled out sequentially and produced relative acceptance-rate gains of 7.1%, 6.3%, and 4.9%, with an estimated combined improvement above 15%. The paper argues these gains matter because they come from signals—developer intent and precise symbol knowledge—that the file being edited alone does not contain.","pith_inferences":["If the acceptance gains generalize, repository-level context is a cheap add-on to existing completion systems: no model retraining is required, only retrieval. A natural next step the paper does not test is replacing the fixed rule-based fusion order with a learned ranker over the same three context streams.","The user-behavior results imply that IDE telemetry—cursor position, file visits, and edit history—carries predictive signal for what a developer will type next. A testable extension is whether on-device, privacy-preserving summarization of that history retains most of the gain.","The large struct-initialization improvement points to a concrete failure mode: hallucinated object members when the type definition is absent. This suggests a focused benchmark: struct and object initialization completions with and without type definitions."],"forward_implications":["Adding user behavior code raises soft exact match by 1.65 to 2.55 percentage points and edit similarity by 1.8 to 2.4 points across Python, Go, and TypeScript.","Similar code retrieval with split tokens raises soft exact match from 49.77 to 56.55 in Python at a 4k token window, and to 59.1 at an 8k window with more snippets.","CKG-based symbol definitions lift struct initialization soft exact match from 6.17 to 17.41, indicating the context reduces hallucinated member names.","The production rollouts of user behavior code, similar code, and symbol definitions each produced a relative acceptance-rate improvement, with an estimated combined improvement above 15%.","Latency constraints are met: similar code retrieval stays below 80 milliseconds and CKG symbol retrieval below 50 milliseconds in production."],"supporting_citations":[{"why":"Supplies the base code model used for all offline experiments.","marker":"[5]"},{"why":"Defines the fill-in-the-middle prompt format that the retrieved context is injected into.","marker":"[6]"},{"why":"The Language Server Protocol approach that ContextModule replaces for symbol retrieval.","marker":"[4]"},{"why":"Provides the embedding-based retrieval baseline compared against token-based similar-code retrieval.","marker":"[8]"},{"why":"Supplies the natural-language stop-word list used in the cut-token feature extraction strategy.","marker":"[7]"},{"why":"Defines the Jaccard similarity score used to rank retrieved similar code snippets.","marker":"[9]"},{"why":"Repository-level retrieval baseline that motivates the need for user-behavior and symbol-level context.","marker":"[34]"},{"why":"Repository representation and expand-and-refine retrieval approach contrasted with ContextModule's design.","marker":"[35]"}],"fun_headline_variants":["Repo context lifts code acceptance by 15%","Code completion gains from your editing history","Three repo signals beat file-only code completion","ContextModule: symbols and history sharpen code AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The offline evaluation assumes that the soft exact match metric and the pre-filtered user-behavior dataset measure completion quality independently of whether the retrieved context was actually needed, so part of the reported gain could be baked into how correctness is counted.","fun_headline_variants_meta":{"raw":{"variants":["Repo context lifts code acceptance by 15%","Code completion gains from your editing history","Three repo signals beat file-only code completion","ContextModule: symbols and history sharpen code AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000184,"raw_usage":{"total_tokens":1313,"prompt_tokens":933,"completion_tokens":380,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":324}},"tokens_in":549,"tokens_out":380,"duration_ms":4569,"temperature":1.0,"reasoning_tokens":324,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:15:29.844264+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a production A/B test that toggles exactly one of the three context strategies at a time while holding the model and prompt template fixed; if removing user behavior code, similar code, or symbol definitions does not lower user acceptance, the central claim fails. A complementary offline check is to recompute the metrics with a strict exact-match or edit-distance criterion that does not count 'retrieved context helped' as a correct answer and see whether the improvements persist.","supporting_citations":[{"cited_title":"Official page for language server protocol,","cited_arxiv_id":null,"evidence_quote":"The Language Server Protocol approach that ContextModule replaces for symbol retrieval."},{"cited_title":"Nltk :: Natural language toolkit,","cited_arxiv_id":null,"evidence_quote":"Supplies the natural-language stop-word list used in the cut-token feature extraction strategy."},{"cited_title":"An efficient recommendation generation using relevant jaccard similarity,","cited_arxiv_id":null,"evidence_quote":"Defines the Jaccard similarity score used to rank retrieved similar code snippets."}],"review_version":1}