{"id":"a1a54956-1272-483a-a630-97be3c4b83f0","arxiv_id":"2502.01273","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Analysis of student-LLM conversations and code in a 13-week software engineering course finds ChatGPT preferred over Copilot and conversational prompting yields lower-complexity code.","lead":"This paper analyzes how 126 computer science students used ChatGPT and GitHub Copilot during a 13-week software engineering project. It reports that students preferred ChatGPT, that ChatGPT code was less complex, and that conversational refinement improved code quality.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Copilot and ChatGPT code are measured at different stages (post-edit vs raw output), so the complexity comparison in Fig. 2 is biased; the paper needs to test whether the gap survives on unmodified snippets.","rationale":"The reader's weakest assumption identifies exactly the same issue: the ChatGPT vs. Copilot complexity comparison is confounded by the different measurement stages. This is the most load-bearing concern because the abstract's central technical claim (ChatGPT produces lower computational complexity) depends entirely on this comparison. The concern is concrete, acknowledged by the authors, and potentially fatal to that claim unless addressed. A level-0 subset analysis is a straightforward way to test the robustness of the finding using data already in hand. Other concerns (e.g., lack of statistical testing, small sample, self-reported tagging) are secondary; even with significance tests, the asymmetry would still undermine the comparison. The paper does have merit as a descriptive study of student-LLM interaction, so conditional acceptance with a requirement to reframe the claim is appropriate, which is what the reader recommended. Our analysis does not alter that verdict.","tokens_in":10825,"tokens_out":3414,"duration_ms":31923,"concrete_test":"Subset the complexity comparison to snippets tagged with intervention level 0 (no human modification) for both tools. Recompute Figure 2 (distributions and means of LOC, cyclomatic complexity, CFG depth, Halstead effort) using only level-0 snippets. If ChatGPT remains significantly lower on these metrics, the measurement asymmetry alone cannot explain the result. If the gap vanishes or reverses, the headline claim must be reframed as a comparison of integrated code, not raw tool outputs. This test is feasible with the existing dataset because intervention levels are already collected for every tagged snippet.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that ChatGPT generates lower-complexity code than Copilot (Abstract, Section III.B) rests on an asymmetric comparison. ChatGPT snippets are tracked as exact model outputs, whereas Copilot snippets are assessed after students have modified and integrated them into the repository (Section III.B explicitly acknowledges this difference). Because Figure 4 shows that repository code tends to be more complex than the original ChatGPT output, student edits typically increase complexity. Thus, Copilot's post-edit complexity is likely inflated relative to its raw suggestions, biasing the comparison in ChatGPT's favor. The paper does not correct for this asymmetry or quantify its direction; it simply notes it. This is load-bearing because the headline 'lower computational complexity' could be an artifact of measurement timing rather than a property of the tools. The same asymmetry also weakens the derived narrative that conversational interaction inherently produces more concise code, since ChatGPT's measured advantage is partly built into the way its outputs are extracted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an observational study of 126 undergraduate students using ChatGPT and GitHub Copilot during a 13-week software engineering project. The authors collected LLM-generated code snippets, student-annotated intervention levels, ChatGPT conversation logs, and repository code, then analyzed tool usage, code complexity (LOC, cyclomatic complexity, CFG depth, Halstead effort), code integration patterns, prompt evolution, and sentiment. The main reported findings are that students prefer ChatGPT over Copilot, ChatGPT-generated code has lower computational complexity than Copilot-generated code, conversational interaction supports iterative code refinement, students' prompting skills improve over the semester, and conversations end on a positive note. The paper also discusses pedagogical implications and design considerations for AI-assisted SE courses.","tokens_in":10967,"tokens_out":2555,"duration_ms":26661,"significance":"If the central claims held, this would be a valuable contribution to the emerging literature on LLM use in software engineering education: it provides a real classroom dataset with 730 tagged code snippets, 62 ChatGPT conversations, and longitudinal data across three milestones, and it explicitly connects code complexity metrics with conversational interaction. The paper is transparent about many of its limitations, including self-report bias and the known difficulty of sentiment analysis on technical text, and it makes its data collection methodology (tagging, intervention levels, conversation linking) reproducible. However, the headline comparison between ChatGPT and Copilot is currently undermined by a measurement asymmetry, and several key claims are made without statistical support. With the load-bearing issues addressed, the dataset and qualitative insights could be of genuine use to educators and researchers.","major_comments":[{"comment":"The comparison of integration effort between ChatGPT and Copilot rests on self-reported intervention levels without any statistical testing. The paper reports that 53.6% of Copilot snippets required level 1 intervention and 15.2% required none, versus 26% of ChatGPT snippets requiring no modification, but it does not test whether these differences are significant or account for per-team clustering. Given that Section IV acknowledges that self-reports may be underreported or selectively disclosed, a formal test (e.g., mixed-effects model or at minimum a chi-square test with team-level clustering) and effect sizes are needed before claiming that ChatGPT code 'generally met project requirements with minimal refinement.'","section":"III.B"},{"comment":"The abstract's claim that 'students prefer ChatGPT over Copilot' is not directly supported by the data presented. Section III.A reports counts of accepted snippets per tool (223 ChatGPT vs. 507 Copilot), which reflects aggregate usage, not preference: higher Copilot usage could indicate convenience or default tool availability, and the paper does not report user surveys or explicit preference ratings. The claim should be softened to 'students used ChatGPT and Copilot in different ways' or supported with a direct preference measure.","section":"III.B"}],"minor_comments":[{"comment":"The sentence 'Additionally, the study's analysis of GPT-generated code is more precise due to the ability to track exact model outputs' is important but appears only as an aside; it should be moved into the methodological design and discussed as a limitation with potential mitigation.","section":"III.B"},{"comment":"The Threats to Validity section is brief and omits a discussion of how the measurement asymmetry between ChatGPT and Copilot affects every complexity comparison; this should be elevated to a primary threat.","section":"IV"},{"comment":"There are several typographical and grammatical errors, including 'Furthre analyzing' (Introduction), 'prevelance' (Summary), and 'the student teams heavily on AI assistants' (Section III.A). A proofreading pass is needed.","section":"Throughout"},{"comment":"The reuse analysis reports counts (95 instances used once, 23 twice, etc.) but does not specify the total number of instances considered; the denominator is needed to interpret these numbers.","section":"III.C"},{"comment":"The description of Figure 7's construction is dense and somewhat ambiguous: the 'prioritized the first occurrence' rule for ties should be justified, since it may bias the reported 'mean index' downward.","section":"III.D"},{"comment":"Several references are incomplete or non-standard, including missing venue details for [11], [13], [14], [17], [19], [20], and [22]; the bibliography should be checked against a consistent style.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is built on a unique and apparently carefully collected dataset, and the authors are transparent about many limitations. The main reason for major revision is the unaddressed measurement asymmetry between ChatGPT and Copilot complexity measurements, which directly bears on the abstract's headline claim. If the authors can provide a corrected comparison (e.g., level-0 snippets only, or raw Copilot suggestions) and add basic statistical significance testing, the paper could become a solid empirical contribution. The sentiment analysis, as acknowledged by the authors, is too weak to support the current narrative; it should be downgraded or validated. The 'students prefer ChatGPT' claim should be removed or re-supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful descriptive dataset paper, but the headline comparison between ChatGPT and Copilot is built on an apples-to-oranges measurement, so treat the complexity claims as suggestive, not established.\n\nWhat's new: a 13-week longitudinal trace of 126 students with both ChatGPT conversation logs and tagged LLM-generated code, plus prompt and sentiment analysis over milestones. That's genuinely rare; most studies use short lab tasks. The observation that students' prompts get more specific and that integrated code increasingly resembles ChatGPT outputs over the semester is a real, pedagogically interesting pattern. The authors are also candid: they flag the self-report bias and the VADER mismatch with technical language in Section IV.\n\nThe soft spots, in order of severity. First, the central ChatGPT-vs-Copilot complexity comparison is confounded. ChatGPT snippets are exact model outputs; Copilot snippets are post-edit repository code. Figure 4 shows students typically increase complexity when integrating ChatGPT code, so the same bias likely inflates Copilot's measured complexity. The paper acknowledges this in one sentence but does not quantify or correct it, and the abstract still asserts lower complexity as a finding. Second, the claim that 'students prefer ChatGPT over Copilot' sits oddly next to Table I, which shows roughly twice as many Copilot snippets accepted (507 vs 223). If preference is measured through a survey or interview, that instrument isn't described. Third, there are no statistical tests anywhere; claims of 'significant decline' or 'trends' are eyeballed from density plots. For a paper with 730 snippets, a few Wilcoxon tests would be cheap. Fourth, the self-reported intervention levels are coarse and subjective, though the authors own that.\n\nIf the authors reframe the contribution as 'how students actually interact with these tools' rather than 'which tool is better,' most of these issues dissolve. The conversation-level similarity analysis and prompt evolution are solid descriptive contributions on their own.\n\nVerdict: worth a serious referee, and I'd accept it conditionally. The data collection alone is a step up from typical LLM4SE student studies. But the authors need to either fix the comparison basis or moderate the claims. I'd bring this to a reading group for the dataset and method discussion, though I wouldn't cite the tool comparison in my own work.","headline":"Useful longitudinal dataset on student-LLM interaction, but the ChatGPT-vs-Copilot complexity comparison is confounded by measuring raw ChatGPT output against post-edit Copilot code, so the headline claim should be treated as suggestive, not established.","tokens_in":11430,"tokens_out":2140,"would_cite":false,"duration_ms":20068,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Conversational interaction with ChatGPT produces simpler, better-fitting code than Copilot's auto-completions, and students' prompting visibly improves over one semester.","keywords":["LLM for code generation","LLM for learning","AI for software engineering","software engineering education","ChatGPT","GitHub Copilot","code complexity","prompt engineering"],"falsifier":"Record Copilot's verbatim suggestion text before any student edit, run the same four complexity metrics on it, and compare it with ChatGPT's raw responses to identical prompts; if Copilot's raw code is no longer or higher-effort, the reported advantage is a measurement artifact.","tokens_in":10662,"feed_emoji":"💬","tokens_out":7066,"duration_ms":59550,"temperature":0.7,"pith_summary":"Across a 13-week software-engineering course, 126 students used ChatGPT and GitHub Copilot to build a static program analyzer, and the researchers studied the conversations, the code that was generated, and the code that was actually integrated. The paper argues that conversational interaction with an LLM produces code that is shorter, less complex, and better matched to what a student needs than the auto-completed code Copilot offers. It also claims that students' prompting improves measurably over the semester, with later prompts producing code that enters the repository almost unchanged. If true, this matters for teaching because it shifts the skill students need from writing code to steering an AI through dialogue, and it suggests curricula should explicitly train prompt refinement.","feed_headline":"ChatGPT conversations yield simpler code than Copilot auto-complete","feed_subtitle":"A 13-week study of 126 students finds dialogue-driven code is shorter, lower-effort, and better matched to needs.","key_machinery":"The argument runs on three measurement devices working together. The first is a four-metric complexity battery — lines of code, cyclomatic complexity, maximum control-flow depth, and Halstead effort — applied to every tagged LLM-generated snippet. The second is a conversation-to-repository similarity pipeline that matches ChatGPT outputs inside a dialogue to code in the final repository using syntax-tree token sets and Jaccard similarity, with long common subsequences treated as equivalent above 90%. The third is sentiment scoring of prompts using a rule-based lexicon, which yields the positive-to-frustrated-to-resolved arc. The tagging scheme records which tool generated each snippet and how many manual edits were needed (none, under 10% of lines, or over 10%), and this metadata carries the headline comparison.","core_discovery":"The central discovery is a workflow-level result: when students interact with ChatGPT through back-and-forth prompts, the generated code's complexity falls as the conversation proceeds, and the final version is more concise and lower in Halstead effort than Copilot's auto-completed output. Copilot-generated code is longer, has higher cyclomatic complexity, and produces many more outliers across all four metrics. Students reported less manual intervention for ChatGPT code, with 26% of ChatGPT snippets requiring no changes versus 15.2% for Copilot. The similarity between ChatGPT output and the final repository code rises across the three milestones as prompts become more specific and context-rich, and sentiment in conversations follows a pattern of initial optimism, mid-conversation frustration, and final resolution.","pith_inferences":["One implication the authors leave implicit is that the complexity gap between ChatGPT and Copilot may partly reflect a measurement asymmetry: ChatGPT outputs were captured verbatim, while Copilot outputs were measured after student editing, so a controlled study that records Copilot's raw suggestions would be needed to separate workflow effects from model effects.","The observed prompt improvement over a single semester suggests prompt engineering is trainable enough to be a formal curriculum component, and this paper's milestone-by-milestone similarity data could serve as a baseline for measuring the effect of such instruction.","Because integrated repository code is often more complex than the raw ChatGPT output, students are evidently adding their own logic rather than blindly accepting AI code; that pattern could be used to design exercises that make the critical-editing step explicit and assessable."],"forward_implications":["ChatGPT's conversational mode yields more concise, lower-effort code than Copilot's auto-completion, so tool choice can measurably affect code maintainability in student projects.","Students' prompts become more specific and context-rich over the semester, indicating that prompt engineering is a skill that develops through project experience.","AI-generated code is reused with increasing similarity to the repository over milestones, showing that iterative dialogue brings LLM output progressively closer to project needs.","Most conversations end with a positive sentiment recovery, suggesting that back-and-forth refinement helps students reach a resolution rather than leaving them stuck.","LLM usage is heavy early in the project and declines later, which points to AI assistants being most useful during the initial design and scaffolding phases."],"supporting_citations":[{"why":"Supplies the project design and the tagging scheme for LLM-generated code, including the human-intervention levels (0, 1, 2) that the study reuses.","marker":"[3]"},{"why":"Provides the rule-based sentiment lexicon used to score the emotional tone of student prompts.","marker":"[10]"},{"why":"Supplies a benchmark of novice-user prompts that contextualizes the study's analysis of students' evolving prompting skill.","marker":"[15]"}],"fun_headline_variants":["ChatGPT chats beat Copilot autocomplete for simpler code","Study: Conversational AI yields shorter, lower-effort code","Students get simpler code from ChatGPT than Copilot","Dialogue-driven LLM code outperforms auto-complete in study","126-student study: ChatGPT code simpler than Copilot's"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparison assumes that ChatGPT's raw outputs and Copilot's accepted outputs are measured on the same footing, but Copilot's snippets were measured after students edited them, so the complexity gap could come from how the tools were tracked rather than from the tools themselves.","fun_headline_variants_meta":{"raw":{"variants":["ChatGPT chats beat Copilot autocomplete for simpler code","Study: Conversational AI yields shorter, lower-effort code","Students get simpler code from ChatGPT than Copilot","Dialogue-driven LLM code outperforms auto-complete in study","126-student study: ChatGPT code simpler than Copilot's"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000163,"raw_usage":{"total_tokens":1220,"prompt_tokens":900,"completion_tokens":320,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":237}},"tokens_in":516,"tokens_out":320,"duration_ms":3591,"temperature":1.0,"reasoning_tokens":237,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T15:48:57.128431+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record Copilot's verbatim suggestion text before any student edit, run the same four complexity metrics on it, and compare it with ChatGPT's raw responses to identical prompts; if Copilot's raw code is no longer or higher-effort, the reported advantage is a measurement artifact.","supporting_citations":[{"cited_title":"Vader: A parsimonious rule-based model for sentiment analysis of social media text,","cited_arxiv_id":null,"evidence_quote":"Provides the rule-based sentiment lexicon used to score the emotional tone of student prompts."},{"cited_title":"Studenteval: A benchmark for evaluating llms on novice user prompts in educational settings,","cited_arxiv_id":null,"evidence_quote":"Supplies a benchmark of novice-user prompts that contextualizes the study's analysis of students' evolving prompting skill."}],"review_version":1}