{"id":"92a3be87-ca9d-4df4-983e-0cae7518fa8a","arxiv_id":"2512.05242","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Repository-grounded RAG assistants reduce—but do not eliminate—code-integration errors for student game projects, with a small 24B model matching much larger ones.","lead":"A university study of 38 students found they rely heavily on ChatGPT during coding phases but get poor answers about project-specific code, so the authors built a local AI assistant that reads the project's source code and docs. The assistant mostly helps, but even the best models still invent code and miss simple resource-file updates.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central mitigation claim lacks a same-protocol baseline; repository grounding is not shown to cause the improvement.","rationale":"The paper's central claim is comparative, but the only comparison is to a prior self-report user study, not to a matched control. That is the most load-bearing weakness: without a same-protocol baseline, the observed outputs of the grounded system cannot be attributed to repository grounding. The single-run design and manual coding are secondary but compound the issue. The paper is transparent about many limitations (Sect. 8.3) and the reader already assigned CONDITIONAL, so this concern does not move the verdict; it reinforces it. The proposed test — a same-protocol non-grounded baseline — is feasible and would settle the attribution question directly.","tokens_in":21862,"tokens_out":3017,"duration_ms":31040,"concrete_test":"Run Prompt A1 and B1/B2 on the same six LLMs without document RAG and repository tools (or via ChatGPT with the same prompts) using the same single-run logging and defect-catalog coding. Compare defect counts and integration success. If the non-grounded baseline matches or beats the grounded system, the Sect. 9 mitigation claim is unsupported; if the grounded system is clearly better, the claim gains support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Sect. 9) that the integration weakness is 'substantially mitigated' rests on contrasting the case-study results (Sect. 8) with the earlier user study (Sect. 5). That comparison is not controlled: the user study measured student self-reports about ChatGPT across all project phases, while the case study measured defect counts for a grounded system on two selected Preparation tasks. Task difficulty, prompt formulation, and model family all differ; none of the case-study runs include a non-grounded control (same models, same prompts, but without repository RAG/tools). So the observed 'usable, repository-aligned outputs' cannot be attributed to repository grounding rather than to task selection or model capability. The paper's own data weaken 'substantially mitigated': hallucinations appeared in 14/15 sampling runs (Table 4) and missing-resource-entry recurred across almost all models (Table 5), with the authors describing only a 'retrieval-to-action gap.' Furthermore, Sect. 7.1 reports that the slider subtask was excluded after pilot failures, which removes exactly the kind of integration failure the claim addresses. The single-run protocol (Sect. 8.3) compounds the problem: even the internal defect counts are uncharacterized stochastically.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a two-stage empirical investigation of generative AI in a university software project course. Stage 1 is a survey of 38 students (78% of 49 enrolled) about their use of ChatGPT/Copilot across four waterfall phases, yielding descriptive findings: high adoption in Coding & Testing, lower in Design, text-heavy tasks succeed but code comprehension and integration suffer from missing project context. Stage 2 is a technical case study of a locally deployed, repository-aware LLM assistant using document-level RAG plus repository-lookup tools. The authors evaluate six models and 15 sampling configurations on two Preparation-phase tasks, using an internally defined 12-category defect catalog. They report that most outputs were usable and repository-aligned, and conclude that the integration weakness of general-purpose AI is 'substantially mitigated' (Sect. 9). The paper's main contributions are the phase-specific adoption/helpfulness data and a transparent architectural description of the RAG/tool pipeline, plus a candid threats-to-validity section.","tokens_in":22098,"tokens_out":5512,"duration_ms":54745,"significance":"If the mitigation claim were supported, the paper would provide evidence for repository-grounded local LLM assistants in software engineering education, with practical value for institutions that need local deployment for data-protection reasons. The user-study phase separately offers a useful descriptive baseline for curriculum design. The paper is strengthened by IRB approval, written consent, a detailed defect catalog, full logging of prompts and tool calls, version-pinned container images, and explicit discussion of limitations. These are real strengths. However, the central comparative claim is currently supported only by an uncontrolled contrast between a self-report survey and a single-run technical evaluation. As it stands, the paper's value is primarily as an exploratory case study and an architecture description, not as a demonstration that repository grounding mitigates integration failures. The claims need to be recalibrated or the experiments need to be strengthened.","major_comments":[{"comment":"The statement 'This integration weakness ... is substantially mitigated in our repository-aware setup' is the paper's central claim, but the design does not support it. The comparison is between student self-reports about generic ChatGPT across all project phases (Sect. 5) and defect counts for the repository-aware assistant on two selected Preparation tasks (Sect. 8). Task difficulty, prompt wording, model family, phase, and outcome measure all differ, and no run included the same prompts with a non-grounded baseline (e.g., the same models without repository RAG/tools). The observed 'usable, repository-aligned outputs' cannot therefore be attributed to repository grounding rather than to task selection or model capability. The paper's own results also weaken the wording: hallucinations appear in 14/15 sampling runs (Table 4) and missing resource entries recur across nearly all models (T","section":"§9 and §8 (opening)"},{"comment":"The exclusion of 'early but self-corrected' hallucinations from the defect counts is problematic and potentially inflates the apparent quality of the grounded system. The text states that in the initial prompt models often identified the correct class but hallucinated its content, and that once instructed to retrieve the file the hallucinations did not reappear; these cases were then not categorized. If the evaluation is about the system's ability to produce contextually grounded answers, an initial hallucination that is corrected only after an explicit user follow-up is still an observed failure. The paper should either include these events in the counts (and report how many were excluded) or justify the exclusion on explicit, pre-specified criteria. As written, the comparison in Table 5 undercounts the very defect category that is central to the mitigation claim.","section":"§8.2"},{"comment":"The slider subtask of Task 2 was removed after pilot runs 'consistently failed to produce a functional slider implementation' because it would not affect the comparative conclusions. This is a post hoc exclusion of an integration-relevant feature: a UI/state/audio integration task is exactly the kind of repository-aware integration the paper claims is mitigated. Excluding it weakens the scope of the evidence; the conclusion is then projected onto integration broadly. Report the pilot runs and include the slider subtask, or explicitly restrict the claim to the two operationalized subtasks. Otherwise the selective task removal makes the mitigation claim difficult to falsify.","section":"§7.1 and §9"},{"comment":"The single-run protocol (exactly one execution per configuration–task combination, no re-runs) is a load-bearing limitation for every quantitative statement in Tables 4 and 5. Statements such as 'T≈0.5 ... yielded the fewest defects' and 'hallucinations clustered most prominently at T=1.0' are made from one sample per cell in a stochastic generation process. Defect categories that occur once or twice cannot be distinguished from sampling noise. The threats-to-validity paragraph acknowledges the issue, but the results section still draws parameter-sensitivity and model-comparison conclusions from it. Multiple replications per cell (even 3–5), or explicit removal of all comparative claims, are needed before these statements can stand.","section":"§8.3 and §8.1"},{"comment":"The defect catalog is internally defined and the coding is manual, with no inter-rater reliability check, no second coder, and no validation against an external benchmark. Since Tables 4/5 are the entire quantitative basis for the case-study evaluation, coder subjectivity directly affects the central narrative. The paper should report a coding protocol, provide the annotated outputs, and include inter-rater statistics (e.g., Cohen's κ) on a subset. This is particularly important because several categories (integration omission, wrapper-only method, insufficient robustness) require judgment about the codebase.","section":"§7.3 and §8.3"}],"minor_comments":[{"comment":"The text says four participants did not use AI, then adds 'In addition, one participant stated that they were unaware of the AI tools available.' With 34 of 38 users, this implies five non-users; please clarify whether the five reasons refer to the same four non-users or whether there was a fifth non-user.","section":"§5"},{"comment":"The stacked horizontal bars show diverging Likert scales, but the figures do not indicate the 0% split or the scale direction. Add a center line and counts or percentages in each segment so the visual does not mislead.","section":"Figures 3 and 5"},{"comment":"The row labeled 'Standard (no change)' is followed by 'Default 1.0 1.0 0.0'; it is unclear whether these are two different configurations or one. The table header 'temp top p min p' is not defined in the text; define the abbreviations.","section":"Table 3"},{"comment":"The statement 'available from the corresponding authors' is not a persistent, verifiable form of data availability. For a study that emphasizes auditability and reproducibility, deposit logs, prompts, and annotated outputs in a public repository or archive.","section":"Data availability"}],"recommendation":"major_revision","confidential_remarks":"The user-study part is a solid descriptive contribution for a software-engineering-education venue, but the journal-level claim of mitigation goes beyond the evidence. The case study is exploratory and contains several methodological decisions (post hoc task exclusion, self-correction exclusion, single runs, manual coding) that all bias toward the claimed conclusion. The paper would be acceptable after substantial revision if the authors either provide a same-protocol baseline with non-grounded models or explicitly reframe the contribution as an exploratory architecture description with carefully bounded conclusions. I also note that the motivation leans heavily on the authors' own prior conference paper (Borghoff et al. 2025); independent support for the broader claim would strengthen the framing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nWorth knowing: this is two papers in one. The first is a repeat of the authors' ECSEE 2025 user study on how students use ChatGPT/Copilot in a project course; the second is a new case study of a locally deployed, RAG-grounded assistant in the same course. The new material is the case study: six models, 15 sampling configs, two tasks, a 12-category defect catalog, and honest observational data. That is genuinely new empirical material, and the defect catalog is a useful artifact for anyone building or evaluating repo-aware coding assistants.\n\nWhat the paper does well: it is transparent. The system design and deployment are described in enough detail to replicate the setup; the prompts are given; the threat-to-validity section is unusually candid. The authors acknowledge single-run protocol, manual coding, absence of a standardized benchmark, narrow task scope, and a simplistic retrieval path. That is more honest than most.\n\nThe soft spots are real and centered on one sentence in the conclusions: 'This integration weakness ... is substantially mitigated in our repository-aware setup.' That claim is not supported by the evidence. There is no same-protocol baseline: the user study measured student self-reports about generic tools across all phases, while the case study measured defect counts for the grounded system on two Preparation tasks. Task difficulty, prompt style, and model family all differ. And within the case study itself, hallucinations appeared in 14 of 15 sampling runs, missing-resource-entry recurred across almost all models, and the slider subtask—where integration was the point—was excluded after pilot failures. So the mitigation claim is an inference from a cross-study comparison, not a same-protocol result. If the authors had run the same two tasks with a generic ChatGPT baseline, the claim would have had legs; as it stands, it needs to be tempered.\n\nThe single-run design and manual coding without inter-rater reliability are exactly what the limitations section says they are, so those threats are acknowledged, but they still limit what can be concluded. The survey portion is descriptive and internally consistent, though it repeats the prior conference paper.\n\nBottom line: this is a solid exploratory case study and a useful resource for people building repository-aware AI tutors for SE education. It deserves a serious referee, but the referee should push for a baseline condition, repeated runs, and a softened claim in the conclusions. I'd cite the defect catalog and the qualitative model comparison, but not the mitigation claim.","headline":"Useful exploratory case study of a repository-aware LLM assistant, but the headline claim about mitigating integration failures is not supported by the evidence as presented.","tokens_in":22580,"tokens_out":2234,"would_cite":true,"duration_ms":21834,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A repository-aware, retrieval-augmented assistant can ground LLM answers in a project's real code and docs, substantially mitigating the code-integration failures students hit with generic AI tools.","keywords":["software engineering education","generative AI","retrieval-augmented generation","repository-aware LLM","programming project course","user study","code comprehension","qualitative defect analysis"],"falsifier":"Re-run the two tasks under identical prompts and sampling parameters with the repository tools disabled, and have the outputs blindly scored with the same defect catalog; if defect rates and integration failures are indistinguishable from the grounded runs, the central mitigation claim collapses.","tokens_in":21742,"feed_emoji":"🎓","tokens_out":4008,"duration_ms":39871,"temperature":0.7,"pith_summary":"The paper tries to establish that the main reason generic AI assistants fail students in software projects is missing project context, and that a locally deployed, repository-aware assistant using retrieval-augmented generation closes much of that gap. It combines a user study (38 students, 89% adoption) showing AI helped text-heavy tasks but stumbled on code comprehension and integration, with a case study of 42 runs across sampling configurations and six model sizes on two real coding tasks from the course's sample game. The paper claims that grounding responses in retrievable documentation and read-only code lookup made outputs repository-aligned and usable after limited corrections, with hallucinations and missing resource entries as the main remaining defects. A sympathetic reader would care because, if correct, small local models with repository access could serve as practical, privacy-preserving AI tutors instead of generic cloud chatbots.","feed_headline":"Project-aware AI beats generic chatbots on code integration","feed_subtitle":"Local retrieval-augmented assistant anchored outputs in real project code across 42 runs, pointing to small models as viable course tutors.","key_machinery":"The central mechanism is a prompt-to-retrieval-to-generation pipeline with two grounding paths: document-level RAG, which embeds project documentation for semantic retrieval, and code-level lookup, in which the model can call three repository tools (a class-path finder, a file-content getter, and a method inspector) against a read-only versioned repository. These tools are orchestrated by a chat interface and an OpenAI-compatible model server, with every prompt, tool call, and response logged. The pipeline's job is to anchor each answer in actual project artifacts before generation; the logs also make the model's behavior inspectable, which is what turns the evaluation into a qualitative def","core_discovery":"The central claim is that repository grounding, not model size or prompt engineering alone, is what lets a coding assistant produce answers that fit an existing code base. In the case study, the assistant retrieved relevant documentation and, when a prompt mentioned code, resolved class paths, pulled file contents, and inspected methods before answering; across all runs the outputs were consistently usable and could be integrated into the project with limited follow-up corrections. The paper interprets this as substantially mitigating the integration weakness previously seen with generic tools. It also reports that decoding temperature around 0.5 with top-p 0.0 was the most stable configurat","pith_inferences":["A same-protocol comparison—identical prompts, sampling, and coding rubric, with repository grounding switched on versus off—would convert the qualitative mitigation claim into a measurable effect size; the paper did not run that control, but its logged infrastructure makes it straightforward.","The retrieval-to-action gap suggests a testable design fix: constrain the model to quote or synthesize the retrieved snippets before proposing code, which may reduce both hallucination and missing-resource-entry defects.","If grounding generalizes beyond this course, onboarding novices into any unfamiliar codebase—open-source contributions, internships, legacy systems—could adopt the same retrieve-then-answer pattern, turning the assistant into a codebase-comprehension tutor rather than a code generator.","The study's phase-wise findings imply a hybrid curriculum: keep general AI for text-heavy artifacts such as documentation, requirements, and tests, but require repository-aware tools for code changes; that split matches where students reported success and failure."],"forward_implications":["If the mitigation claim holds, a coding-course assistant no longer needs the largest cloud models; a moderately sized locally hosted model with repository access can handle code-comprehension and integration questions, lowering latency, cost, and privacy exposure.","The approach points to a concrete curriculum change: teach students to use repository-aware tools that retrieve and cite project files, rather than prompting generic chatbots in isolation.","For course organizers, the same architecture can be reused year after year with a fixed repository snapshot, making AI assistance reproducible and auditable in project-based teaching.","The observed defect profile suggests the next bottleneck is not model choice but tool discipline: most failures were hallucinations and missing resource entries even when relevant context had been retrieved, so better retrieval-consumption mechanisms should be prioritized.","Because small models were viable, the researchers' plan to run many students simultaneously on one in-house GPU server is technically plausible."],"fun_headline_variants":["Grounding AI in your repo beats generic coding chatbots","Small models can tutor coding when grounded in repo context","Coding assistant quality hinges on repo context, not model size","Study: AI coding help that reads your project integrates better","Repo-aware AI helps code integration better than one-size-fits-all tools"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The conclusion that repository grounding—rather than task difficulty or model capability—caused the improvement rests on one run per configuration, two hand-picked tasks, manual defect coding, and no same-protocol comparison with generic AI tools.","fun_headline_variants_meta":{"raw":{"variants":["Grounding AI in your repo beats generic coding chatbots","Small models can tutor coding when grounded in repo context","Coding assistant quality hinges on repo context, not model size","Study: AI coding help that reads your project integrates better","Repo-aware AI helps code integration better than one-size-fits-all tools"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002006,"raw_usage":{"total_tokens":7645,"prompt_tokens":707,"completion_tokens":6938,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":451,"completion_tokens_details":{"reasoning_tokens":6855}},"tokens_in":451,"tokens_out":6938,"duration_ms":45437,"temperature":1.0,"reasoning_tokens":6855,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T18:24:55.418402+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the two tasks under identical prompts and sampling parameters with the repository tools disabled, and have the outputs blindly scored with the same defect catalog; if defect rates and integration failures are indistinguishable from the grounded runs, the central mitigation claim collapses.","supporting_citations":[],"review_version":1}