Pith. sign in

REVIEW 5 major objections 4 minor 16 references

Learning to Code with Context: A Study-Based Approach

T0 review · 5 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A repository-aware, retrieval-augmented assistant can ground LLM answers in a project's real code and docs, substantially mitigating the code-integration failures students hit with generic AI tools.

desk verdict Useful exploratory case study of a repository-aware LLM assistant, but the headline claim about mitigating integration failures is not supported by the evidence as presented. read the letter →

arxiv 2512.05242 v1 pith:VREUZ7UV submitted 2025-12-04 cs.SE cs.AI

Uwe M. Borghoff , Mark Minas , Jannis Schopp This is my paper · ORCID
classification cs.SEcs.AI
keywords softwareengineeringeducationgenerativeAIretrieval-augmentedgenerationrepository-awareLLMprogrammingprojectcourseuserstudycodecomprehensionqualitativedefectanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the main reason generic AI assistants fail students in software projects is missing project context, and that a locally deployed, repository-aware assistant using retrieval-augmented generation closes much of that gap. It combines a user study (38 students, 89% adoption) showing AI helped text-heavy tasks but stumbled on code comprehension and integration, with a case study of 42 runs across sampling configurations and six model sizes on two real coding tasks from the course's sample game. The paper claims that grounding responses in retrievable documentation and read-only code lookup made outputs repository-aligned and usable after limited corrections, with hallucinations and missing resource entries as the main remaining defects. A sympathetic reader would care because, if correct, small local models with repository access could serve as practical, privacy-preserving AI tutors instead of generic cloud chatbots.

What carries the argument

The central mechanism is a prompt-to-retrieval-to-generation pipeline with two grounding paths: document-level RAG, which embeds project documentation for semantic retrieval, and code-level lookup, in which the model can call three repository tools (a class-path finder, a file-content getter, and a method inspector) against a read-only versioned repository. These tools are orchestrated by a chat interface and an OpenAI-compatible model server, with every prompt, tool call, and response logged. The pipeline's job is to anchor each answer in actual project artifacts before generation; the logs also make the model's behavior inspectable, which is what turns the evaluation into a qualitative def

What would settle it

Re-run the two tasks under identical prompts and sampling parameters with the repository tools disabled, and have the outputs blindly scored with the same defect catalog; if defect rates and integration failures are indistinguishable from the grounded runs, the central mitigation claim collapses.

Watch

Extended reading notes

Core claim

The central claim is that repository grounding, not model size or prompt engineering alone, is what lets a coding assistant produce answers that fit an existing code base. In the case study, the assistant retrieved relevant documentation and, when a prompt mentioned code, resolved class paths, pulled file contents, and inspected methods before answering; across all runs the outputs were consistently usable and could be integrated into the project with limited follow-up corrections. The paper interprets this as substantially mitigating the integration weakness previously seen with generic tools. It also reports that decoding temperature around 0.5 with top-p 0.0 was the most stable configurat

Load-bearing premise

The conclusion that repository grounding—rather than task difficulty or model capability—caused the improvement rests on one run per configuration, two hand-picked tasks, manual defect coding, and no same-protocol comparison with generic AI tools.

Editorial extensions

If this is right

  • If the mitigation claim holds, a coding-course assistant no longer needs the largest cloud models; a moderately sized locally hosted model with repository access can handle code-comprehension and integration questions, lowering latency, cost, and privacy exposure.
  • The approach points to a concrete curriculum change: teach students to use repository-aware tools that retrieve and cite project files, rather than prompting generic chatbots in isolation.
  • For course organizers, the same architecture can be reused year after year with a fixed repository snapshot, making AI assistance reproducible and auditable in project-based teaching.
  • The observed defect profile suggests the next bottleneck is not model choice but tool discipline: most failures were hallucinations and missing resource entries even when relevant context had been retrieved, so better retrieval-consumption mechanisms should be prioritized.
  • Because small models were viable, the researchers' plan to run many students simultaneously on one in-house GPU server is technically plausible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A same-protocol comparison—identical prompts, sampling, and coding rubric, with repository grounding switched on versus off—would convert the qualitative mitigation claim into a measurable effect size; the paper did not run that control, but its logged infrastructure makes it straightforward.
  • The retrieval-to-action gap suggests a testable design fix: constrain the model to quote or synthesize the retrieved snippets before proposing code, which may reduce both hallucination and missing-resource-entry defects.
  • If grounding generalizes beyond this course, onboarding novices into any unfamiliar codebase—open-source contributions, internships, legacy systems—could adopt the same retrieve-then-answer pattern, turning the assistant into a codebase-comprehension tutor rather than a code generator.
  • The study's phase-wise findings imply a hybrid curriculum: keep general AI for text-heavy artifacts such as documentation, requirements, and tests, but require repository-aware tools for code changes; that split matches where students reported success and failure.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper reports a two-stage empirical investigation of generative AI in a university software project course. Stage 1 is a survey of 38 students (78% of 49 enrolled) about their use of ChatGPT/Copilot across four waterfall phases, yielding descriptive findings: high adoption in Coding & Testing, lower in Design, text-heavy tasks succeed but code comprehension and integration suffer from missing project context. Stage 2 is a technical case study of a locally deployed, repository-aware LLM assistant using document-level RAG plus repository-lookup tools. The authors evaluate six models and 15 sampling configurations on two Preparation-phase tasks, using an internally defined 12-category defect catalog. They report that most outputs were usable and repository-aligned, and conclude that the integration weakness of general-purpose AI is 'substantially mitigated' (Sect. 9). The paper's main contributions are the phase-specific adoption/helpfulness data and a transparent architectural description of the RAG/tool pipeline, plus a candid threats-to-validity section.

Significance. If the mitigation claim were supported, the paper would provide evidence for repository-grounded local LLM assistants in software engineering education, with practical value for institutions that need local deployment for data-protection reasons. The user-study phase separately offers a useful descriptive baseline for curriculum design. The paper is strengthened by IRB approval, written consent, a detailed defect catalog, full logging of prompts and tool calls, version-pinned container images, and explicit discussion of limitations. These are real strengths. However, the central comparative claim is currently supported only by an uncontrolled contrast between a self-report survey and a single-run technical evaluation. As it stands, the paper's value is primarily as an exploratory case study and an architecture description, not as a demonstration that repository grounding mitigates integration failures. The claims need to be recalibrated or the experiments need to be strengthened.

major comments (5)
  1. [§9 and §8 (opening)] The statement 'This integration weakness ... is substantially mitigated in our repository-aware setup' is the paper's central claim, but the design does not support it. The comparison is between student self-reports about generic ChatGPT across all project phases (Sect. 5) and defect counts for the repository-aware assistant on two selected Preparation tasks (Sect. 8). Task difficulty, prompt wording, model family, phase, and outcome measure all differ, and no run included the same prompts with a non-grounded baseline (e.g., the same models without repository RAG/tools). The observed 'usable, repository-aligned outputs' cannot therefore be attributed to repository grounding rather than to task selection or model capability. The paper's own results also weaken the wording: hallucinations appear in 14/15 sampling runs (Table 4) and missing resource entries recur across nearly all models (T
  2. [§8.2] The exclusion of 'early but self-corrected' hallucinations from the defect counts is problematic and potentially inflates the apparent quality of the grounded system. The text states that in the initial prompt models often identified the correct class but hallucinated its content, and that once instructed to retrieve the file the hallucinations did not reappear; these cases were then not categorized. If the evaluation is about the system's ability to produce contextually grounded answers, an initial hallucination that is corrected only after an explicit user follow-up is still an observed failure. The paper should either include these events in the counts (and report how many were excluded) or justify the exclusion on explicit, pre-specified criteria. As written, the comparison in Table 5 undercounts the very defect category that is central to the mitigation claim.
  3. [§7.1 and §9] The slider subtask of Task 2 was removed after pilot runs 'consistently failed to produce a functional slider implementation' because it would not affect the comparative conclusions. This is a post hoc exclusion of an integration-relevant feature: a UI/state/audio integration task is exactly the kind of repository-aware integration the paper claims is mitigated. Excluding it weakens the scope of the evidence; the conclusion is then projected onto integration broadly. Report the pilot runs and include the slider subtask, or explicitly restrict the claim to the two operationalized subtasks. Otherwise the selective task removal makes the mitigation claim difficult to falsify.
  4. [§8.3 and §8.1] The single-run protocol (exactly one execution per configuration–task combination, no re-runs) is a load-bearing limitation for every quantitative statement in Tables 4 and 5. Statements such as 'T≈0.5 ... yielded the fewest defects' and 'hallucinations clustered most prominently at T=1.0' are made from one sample per cell in a stochastic generation process. Defect categories that occur once or twice cannot be distinguished from sampling noise. The threats-to-validity paragraph acknowledges the issue, but the results section still draws parameter-sensitivity and model-comparison conclusions from it. Multiple replications per cell (even 3–5), or explicit removal of all comparative claims, are needed before these statements can stand.
  5. [§7.3 and §8.3] The defect catalog is internally defined and the coding is manual, with no inter-rater reliability check, no second coder, and no validation against an external benchmark. Since Tables 4/5 are the entire quantitative basis for the case-study evaluation, coder subjectivity directly affects the central narrative. The paper should report a coding protocol, provide the annotated outputs, and include inter-rater statistics (e.g., Cohen's κ) on a subset. This is particularly important because several categories (integration omission, wrapper-only method, insufficient robustness) require judgment about the codebase.
minor comments (4)
  1. [§5] The text says four participants did not use AI, then adds 'In addition, one participant stated that they were unaware of the AI tools available.' With 34 of 38 users, this implies five non-users; please clarify whether the five reasons refer to the same four non-users or whether there was a fifth non-user.
  2. [Figures 3 and 5] The stacked horizontal bars show diverging Likert scales, but the figures do not indicate the 0% split or the scale direction. Add a center line and counts or percentages in each segment so the visual does not mislead.
  3. [Table 3] The row labeled 'Standard (no change)' is followed by 'Default 1.0 1.0 0.0'; it is unclear whether these are two different configurations or one. The table header 'temp top p min p' is not defined in the text; define the abbreviations.
  4. [Data availability] The statement 'available from the corresponding authors' is not a persistent, verifiable form of data availability. For a study that emphasizes auditability and reproducibility, deposit logs, prompts, and annotated outputs in a public repository or archive.

Circularity Check

0 steps flagged · score 2.0 of 10

No definitional or fitted-input circularity; central claim rests on an uncontrolled comparison, and the one self-citation is not load-bearing.

full rationale

The paper's chain is empirical rather than definitional: a user study (Sects. 4–5) identifies context/integration weaknesses of generic AI tools; a repository-aware RAG assistant is then built (Sect. 6) and evaluated in a case study (Sects. 7–8); Sect. 9 concludes that the weakness is 'substantially mitigated'. No equation, fitted constant, or parameter is reused as a prediction: the defect catalog (Sect. 7.3) is a coding instrument, not a fitted model, and the sampling/model sweeps are direct evaluations, not predictions. The only explicit self-citation in the central claim is 'previously identified by Borghoff et al. (2025)' (Sect. 9), but the same user study is reported in this paper (Sects. 4–5), so the citation is corroborative rather than load-bearing. The main weakness is inferential, not circular: the mitigation conclusion contrasts case-study defect counts with user-study self-reports under different protocols, with no non-grounded control, and the paper itself acknowledges single-run protocols, manual coding, narrow task scope, and the excluded slider subtask (Sects. 7.1, 8.3). Those are validity/correctness threats, not reductions of the conclusion to its inputs by construction. Accordingly, the score is 2, reflecting the minor self-citation framing and the self-referential motivation, while no concrete circular step is present.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No equations or fitted constants; the load-bearing assumptions are methodological: self-reports are valid, single-run outputs are interpretable, two tasks are representative, manual defect coding is reliable, and grounding—rather than task or model—causes the observed outputs. Every one is acknowledged or implicit in the evaluation design.

assumptions (5)
  • domain assumption Self-reported survey answers accurately reflect actual AI tool use and perceived time savings.
    The user-study results (Sect. 5) treat Likert-scale and open-ended self-reports as measures of adoption, helpfulness, and time saved, without independent logging or verification.
  • domain assumption Manual defect coding using the paper's defect catalog is reliable enough for comparative conclusions.
    Section 7.3 defines the catalog; Sect. 8 reports counts as evidence. Section 8.3 admits subjectivity and possible coder bias and gives no inter-rater reliability measure.
  • domain assumption Two Preparation-phase tasks (3D ship models, background music) are representative of the project tasks for which the assistant is intended.
    Section 7.1 selects two tasks and uses them to generalize about hallucination, robustness, and model comparisons.
  • domain assumption Single-run outputs can support qualitative statements about model behavior and parameter sensitivity.
    Section 7.2 fixes one run per configuration; Sect. 8 draws patterns from sparse defect counts; Sect. 8.3 acknowledges this limits stability.
  • domain assumption Integration improvements observed in the case study are attributable to repository grounding rather than task selection or model choice.
    No generic-assistant baseline under the same tasks; the mitigation claim in Sect. 9 compares across different studies (Sect. 5 vs Sect. 8).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning to Code with Context: A Study-Based Approach." pith.science (2026). https://pith.science/paper/VREUZ7UV

@misc{pith2026251205242,
  author       = {Pith},
  title        = {Pith review of: Learning to Code with Context: A Study-Based Approach},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VREUZ7UV}},
  note         = {Machine review of arXiv:2512.05242}
}
read the original abstract

The rapid emergence of generative AI tools is transforming the way software is developed. Consequently, software engineering education must adapt to ensure that students not only learn traditional development methods but also understand how to meaningfully and responsibly use these new technologies. In particular, project-based courses offer an effective environment to explore and evaluate the integration of AI assistance into real-world development practices. This paper presents our approach and a user study conducted within a university programming project in which students collaboratively developed computer games. The study investigates how participants used generative AI tools throughout different phases of the software development process, identifies the types of tasks where such tools were most effective, and analyzes the challenges students encountered. Building on these insights, we further examine a repository-aware, locally deployed large language model (LLM) assistant designed to provide project-contextualized support. The system employs Retrieval-Augmented Generation (RAG) to ground responses in relevant documentation and source code, enabling qualitative analysis of model behavior, parameter sensitivity, and common failure modes. The findings deepen our understanding of context-aware AI support in educational software projects and inform future integration of AI-based assistance into software engineering curricula.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 1 canonical work pages

  1. [1]

    Comput Sci Educ 15(2):83–102

    Ala-Mutka K (2005) A survey of automated assessment approaches for program- ming assignments. Comput Sci Educ 15(2):83–102. URL https://doi.org/10.1080/ 08993400500150747 AlOmar EA (2025) Nurturing code quality: Leveraging static analysis and large language models for software quality in education. ACM Trans Comput Educ 25(2):1–36. URL https://doi.org/10....

    arXiv 2005

  2. [4]

    MIS Quarterly 13(3):319–340

    ACM, pp 110–116, URL https://doi.org/10.1145/ 3587102.3588815 Davis F (1989) Perceived usefulness, perceived ease of use, and user acceptance of infor- mation technology. MIS Quarterly 13(3):319–340. URL https://doi.org/10.2307/ 249008 Douce C, Livingstone D, Orwell J (2005) Automatic test-based assessment of pro- gramming: A review. ACM J Educ Resour Com...

    arXiv 1989

  3. [7]

    In: 2013 12th International Conference on Machine Learning and Applications, Miami, FL, USA, pp 85–92, URL https://doi.org/10.1109/ICMLA

    URL https://doi.org/10.3389/FRAI.2024.1436350 Srikant S, Aggarwal V (2013) Automatic grading of computer programs: A machine learning approach. In: 2013 12th International Conference on Machine Learning and Applications, Miami, FL, USA, pp 85–92, URL https://doi.org/10.1109/ICMLA. 2013.22 Waseem M, Das T, Ahmad A, et al (2024) Chatgpt as a software develo...

    arXiv 2024

  4. [8]

    Inf 15(6):354

    ACM, pp 455:1–455:23, URL https://doi.org/10.1145/3544548.3580919 Kokol P (2024) The use of AI in software engineering: A synthetic knowledge syn- thesis of the recent research literature. Inf 15(6):354. URL https://doi.org/10.3390/ INFO15060354 Krusche S, Seitz A (2018) Artemis: An automatic assessment management system for interactive learning. In: Barn...

    arXiv 2024

  5. [10]

    SN Comput Sci 6(1):63

    IEEE, pp 2212–2217, URL https: //doi.org/10.1109/COMPSAC65507.2025.00310 Martinovic B, Rozic R (2025) Perceived impact of ai-based tooling on software development code quality. SN Comput Sci 6(1):63. URL https://doi.org/10.1007/ S42979-024-03608-4 Mastropaolo A, et al (2023) On the robustness of code generation techniques: An empirical study on github cop...

    arXiv 2025

  6. [11]

    In: Proc

    IEEE/ACM, pp 2149–2160, URL https://doi.org/10.1109/ICSE48619.2023.00181 Moroz EA, Grizkevich VO, Novozhilov IM (2022) The potential of artificial intelligence as a method of software developer’s productivity improvement. In: Proc. 2022 Conf. of Russian Young Researchers in Electrical and Electronic Engineering (ElConRus), IEEE, pp 386–390, URL https://do...

    arXiv 2023

  7. [13]

    3702168 Pudari R, Ernst NA (2023) From copilot to pilot: Towards AI supported soft- ware development

    ACM, pp 33–38, URL https://doi.org/10.1145/3702163. 3702168 Pudari R, Ernst NA (2023) From copilot to pilot: Towards AI supported soft- ware development. arXiv abs-2303.04142. URL https://doi.org/10.48550/arXiv. 2303.04142 Raman A, Kumar V (2022) Programming pedagogy and assessment in the era of AI/ML: A position paper. In: Choppella V, Karkare A, Babu C,...

    arXiv doi:10.48550/arxiv.2303.04142 2023

  8. [14]

    ACM, pp 29–34, URL https: //doi.org/10.1145/3561833.3561843 Rane N, Choudhary S, Rane J (2023) Education 4.0 and 5.0: Integrating artificial intelligence (AI) for personalized and adaptive learning. Available at SSRN 4638365 URL https://doi.org/10.2139/ssrn.4638365 Rapaka A, Dharmadhikari SC, Kasat K, et al (2025) Revolutionizing learning - A jour- ney in...

    arXiv 2023

Show all 16 references
  1. [16]

    ACM Comput Surv 54(10s):206:1–206:73

    SCITEPRESS, pp 406–413, URL https://doi.org/10.5220/0012631600003687 Yang Y, Xia X, Lo D, et al (2022) A survey on deep learning for software engineering. ACM Comput Surv 54(10s):206:1–206:73. URL https://doi.org/10.1145/3505243 36

    2022 doi

  2. [2018]

    Australasian Journal of Educational Technology 41(3)

    ACM, pp 284–289, URL https://doi.org/10.1145/3159450.3159602 Li S, Liu J, Dong Q (2025) Generative artificial intelligence-supported programming education: Effects on learning performance, self-efficacy and processes. Australasian Journal of Educational Technology 41(3). URL h...

    2025

  3. [2019]

    In: Schmidt A, et al (eds) Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI 2023, Hamburg, Germany, April 23-28,

    ACM, pp 27–52, URL https://doi.org/ 10.1145/3344429.3372501 Kazemitabaar M, Chow J, Ma CKT, et al (2023) Studying the effect of AI code gen- erators on supporting novice learners in introductory programming. In: Schmidt A, et al (eds) Proceedings of the 2023 CHI Conference on ...

    2023

  4. [2020]

    In: Sheard J, Denny P (eds) ACE ’22: Australasian Computing Education Conference, Virtual Event, Australia, February 14 - 18,

    Association for Computational Linguistics, pp 1536–1547, URL https://doi.org/10.18653/v1/2020.findings-emnlp.139 Finnie-Ansley J, Denny P, Becker BA, et al (2022) The robots are coming: Exploring the implications of OpenAI Codex on introductory programming. In: Sheard J, Denny...

    2020 doi

  5. [2022]

    1145/3511861.3511863 Gao Y, Xiong Y, Gao X, et al (2024) Retrieval-augmented generation for large language models: A survey

    ACM, pp 10–19, URL https://doi.org/10. 1145/3511861.3511863 Gao Y, Xiong Y, Gao X, et al (2024) Retrieval-augmented generation for large language models: A survey. arXiv abs-2312.10997. URL https://doi.org/10.48550/ 33 arXiv.2312.10997 Garousi V, Jafarov Z, Movsumova A, et al ...

    2024

  6. [2023]

    3593669 Borghoff UM, Minas M, M¨ onch K (2024) Automatic program assessment, grading and code generation: Possible AI-support in a software development course

    ACM, pp 22–30, URL https://doi.org/10.1145/3593663. 3593669 Borghoff UM, Minas M, M¨ onch K (2024) Automatic program assessment, grading and code generation: Possible AI-support in a software development course. In: Rutkowski L, et al (eds) Artificial Intelligence and Soft Com...

    2024 doi

  7. [2024]

    15 Piscitelli A, Costagliola G, Rosa MD, et al (2024) Influence of large language models on programming assignments - A user study

    Association for Computational Linguistics, pp 237–250, URL https://doi.org/10.18653/V1/2024.EMNLP-MAIN. 15 Piscitelli A, Costagliola G, Rosa MD, et al (2024) Influence of large language models on programming assignments - A user study. In: Proceedings of the 16th International...

    2024 doi

  8. [2025]

    IEEE Softw 41(6):34–37

    ACM, pp 161–170, URL https://doi.org/10.1145/3723010.3723012 32 Carleton AD, Falessi D, Zhang H, et al (2024) Generative AI: redefining the future of software engineering. IEEE Softw 41(6):34–37. URL https://doi.org/10.1109/MS. 2024.3441889 Choudhuri R, Liu D, Steinmacher I, e...

    2024

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.