Pith. sign in

REVIEW 1 major objections 6 minor 13 references

DebugTracker: Lightweight Process Evidence for Classroom Debugging

T0 review · 1 major / 6 minor · reviewed 2026-07-08 · glm-5.2

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VS Code extension captures how students debug, not just what they submit

desk verdict A well-engineered classroom debugging-trace tool with a structural gap in its most important assessment signal read the letter →

arxiv 2607.05871 v1 pith:GNSNXMMR submitted 2026-07-07 cs.SE cs.CY

Jiatong Liu , Xue Yao , Zehua Zhang , Yongqiang Tian This is my paper · ORCID
classification cs.SEcs.CY
keywords evidencecodedebugtrackerdebuggingclassroomdebuggerextensionhuman
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents DebugTracker, a Visual Studio Code extension designed to close a gap in classroom debugging assessment: instructors typically see only the final code and pass/fail result, losing visibility into how a student reproduced a failure, formed a hypothesis, edited code, and verified the fix. The tool records lightweight, structured process evidence — test commands, editor navigation, debugger lifecycle events, student-authored checkpoints, and source snapshots — as an append-only JSONL event log, then exports a timeline and Markdown report for human review. A central design choice is the separation of uncoached Evaluation Mode traces from coached Training Mode traces, so assessment evidence stays distinct from practice sessions where students receive process prompts and optional AI feedback. The authors validate the prototype through 16 automated checks and an 11-case manual trial matrix covering packaged installation on Windows, macOS, and Linux, with debugging tasks in Python, TypeScript, and Java that share a single bug to confirm the tool works through one language-agnostic event model rather than language-specific tooling.

What carries the argument

The central mechanism is a thin set of listeners over standard VS Code events (editor open/edit, debugger start/stop/step/breakpoint/continue, terminal commands) feeding a single append-only JSONL writer. Each event carries a session id, timestamp, mode (evaluation or training), source (session/editor/debugger/terminal/test/student/human/ai), event type, optional file location, a short summary, and a structured payload. Test runs are detected by matching terminal commands against built-in and instructor-configured patterns; debugger activity is observed through the Debug Adapter Protocol; student checkpoints and human labels enter through explicit command-palette actions. Timelines and Mark

What would settle it

Deploy DebugTracker in a real classroom, collect traces from students solving identical debugging tasks, and ask multiple instructors to independently classify each session as systematic or non-systematic debugging using only the exported Markdown reports. If instructor classifications show low inter-rater agreement, or if they correlate poorly with classifications made from full screen recordings of the same sessions, the core claim that lightweight metadata is sufficient for process assessment would be undermined.

Watch

Extended reading notes

Core claim

DebugTracker demonstrates that the standard VS Code extension API — editor events, the Debug Adapter Protocol, and shell-integrated terminal commands — is sufficient to capture a structured debugging-process trace across multiple programming languages without full screen recording or keystroke logging. The append-only JSONL event schema, with its separation of student, instructor, and AI sources, preserves a single ordered evidence stream that can be re-rendered into timelines and reports without mutating the underlying data. The two-mode design (Evaluation vs. Training) keeps coached and uncoached sessions on separate, clearly labeled tracks, addressing the risk that practice scaffolding污染s

Load-bearing premise

The paper assumes that capturing task-relevant metadata — test commands, editor navigation, debugger lifecycle events, and student checkpoints — is sufficient for an instructor to reconstruct a meaningful debugging narrative, without recording full screen video or keystrokes. No empirical evidence is provided that instructors can actually distinguish systematic debugging from guessing using the exported reports alone.

Editorial extensions

If this is right

  • If the captured events are sufficient to reconstruct a meaningful debugging narrative, instructors could assess debugging competence rather than just code correctness, enabling rubric-based process grading at classroom scale.
  • The append-only, source-attributed event model could serve as a portable format for debugging-process data, analogous to how ProgSnap2 standardizes programming-process snapshots, but scoped specifically to debugging episodes.
  • The explicit separation of coached and uncoached traces could be adopted by other educational tools to prevent scaffolding contamination in assessment, particularly as AI-assisted feedback becomes more common in IDEs.
  • If deployed at scale, the structured traces could enable automated detection of debugging anti-patterns (e.g., edit-test cycles without hypothesis formation) for early pedagogical intervention.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The tool's value proposition depends on an untested assumption: that instructors reading exported Markdown reports can reliably distinguish systematic debugging from guessing using the captured event granularity. A controlled study comparing instructor assessments from DebugTracker traces against expert video review would test this directly.
  • The language-agnostic claim rests on VS Code's Debug Adapter Protocol coverage; languages with less mature VS Code debugger extensions would produce thinner traces, potentially creating unequal assessment conditions across programming courses.
  • The test-command detection mechanism relies on shell integration and pattern matching, which may fail for non-standard test runners or custom scripts, creating silent gaps in the evidence record that instructors would not detect from the report alone.
  • The privacy-conscious capture policy (no full terminal output, no keystrokes) means the trace may miss debugging-relevant context such as intermediate print-statement output or exploratory code edits that were reverted before a snapshot.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 6 minor

Summary. The paper presents DebugTracker, a VS Code extension that records lightweight debugging-process evidence for classroom tasks. The tool separates uncoached Evaluation Mode traces from coached Training Mode traces, stores append-only JSONL events, and exports timeline and Markdown reports for instructor review. The prototype is validated with debugging tasks in Python, TypeScript, and Java, 16 automated checks, and an 11-case manual trial matrix across three operating systems. The paper is submitted as a tool demonstration to ISSTA '26.

Significance. The paper addresses a genuine gap in debugging education: final-code-only assessment cannot distinguish systematic debugging from lucky guessing. DebugTracker's design is well-motivated and the engineering validation is solid for a tool demonstration. Strengths include the append-only JSONL event schema (Table 2), the explicit mode separation between coached and uncoached traces, the language-agnostic capture design using standard VS Code mechanisms, and the reproducible artifact package (VSIX, sample tasks, demo script, archived Zenodo snapshot). The 16 automated checks and 11-case manual trial matrix provide reasonable engineering evidence that the tool works as described across languages and operating systems. The paper is appropriately scoped as an artifact validation rather than a pedagogical study.

major comments (1)
  1. Table 1, 'Record hypothesis' row, Instructor check 'Hypothesis before edit?': This check is presented as something the captured evidence already supports. However, as described in Section 2 and Section 4, in Evaluation Mode process prompts are disabled and student checkpoints (including hypotheses) enter only through 'explicit command-palette actions.' There is no enforcement or fallback mechanism: if a student does not voluntarily record a hypothesis, the trace shows an edit with no preceding hypothesis event, which is ambiguous between 'student guessed' and 'student hypothesized but did not record it.' This is a structural property of the design, not a missing empirical validation. The paper should either (a) qualify Table 1's 'Hypothesis before edit?' check to note that it is contingent on voluntary student compliance in Evaluation Mode, or (b) describe a fallback mechanism (e.g., a轻量
minor comments (6)
  1. Section 5: the 11-case manual trial matrix would benefit from a table listing each case, the configuration tested, and the result (pass/fail). Currently the matrix is described only in prose, making it difficult to assess coverage.
  2. Section 4, paragraph on language-agnostic design: the claim that built-in patterns recognize 'common JavaScript, Python, Java, and C/C++ test commands when shell integration is available' is not validated in the trial matrix, which covers only Python, TypeScript, and Java. Consider noting this distinction.
  3. Figure 2: the JSONL excerpts are difficult to read in the rendered figure. Consider reformatting or providing a wider listing in an appendix.
  4. Section 2: the phrase 'the same debugging path' is used loosely. Consider clarifying that this is a typical or expected path, not an enforced sequence.
  5. Section 7: the CC BY-NC-SA 4.0 license may conflict with some institutions' open-source or artifact expectations. Consider noting the rationale for the NonCommercial restriction.
  6. Reference [6] is dated 2026 with an access date of 2026-06-24. Verify this is the intended citation date.

Simulated Author's Rebuttal

1 responses · 0 unresolved

The referee recommends minor revision and identifies one major comment regarding the 'Hypothesis before edit?' instructor check in Table 1. The referee correctly observes that in Evaluation Mode, hypothesis recording depends on voluntary student compliance via command-palette actions, creating ambiguity when no hypothesis event precedes an edit. We agree this is a structural property of the design and will revise the manuscript to qualify Table 1 accordingly.

read point-by-point responses

  1. Referee: Table 1, 'Record hypothesis' row, Instructor check 'Hypothesis before edit?': This check is presented as something the captured evidence already supports. However, as described in Section 2 and Section 4, in Evaluation Mode process prompts are disabled and student checkpoints (including hypotheses) enter only through 'explicit command-palette actions.' There is no enforcement or fallback mechanism: if a student does not voluntarily record a hypothesis, the trace shows an edit with no preceding hypothesis event, which is ambiguous between 'student guessed' and 'student hypothesized but did not record it.' This is a structural property of the design, not a missing empirical validation. The paper should either (a) qualify Table 1's 'Hypothesis before edit?' check to note that it is contingent on voluntary student compliance in Evaluation Mode, or (b) describe a fallback mechanism.

    Authors: The referee is correct on both the factual description and the design implication. In Evaluation Mode, process prompts are disabled and hypothesis checkpoints enter only through explicit command-palette actions, so an absent hypothesis event is genuinely ambiguous between 'no hypothesis was formed' and 'a hypothesis was formed but not recorded.' This is a structural property of the design, not a validation gap. We accept option (a): we will qualify Table 1's 'Hypothesis before edit?' check to state explicitly that it is contingent on voluntary student compliance in Evaluation Mode, and we will add a brief note in Section 2 explaining that the absence of a hypothesis event in an Evaluation trace is not itself evidence that no hypothesis was formed. We considered option (b) but decided against adding a fallback mechanism (such as a pre-edit prompt or soft gate) in Evaluation Mode, because any such mechanism would constitute coaching and would undermine the mode separation that the tool is designed to maintain. The qualification is the honest representation of what the evidence supports. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: tool demonstration paper with self-contained engineering validation

full rationale

This is a tool demonstration paper, not a derivation or prediction paper. DebugTracker's claims are about what the extension does (records events, exports reports, separates modes), and the validation in Section 5 checks that the implemented tool does those things via 16 automated tests and an 11-case manual matrix. No parameters are fitted to data and then re-predicted. No self-citations appear in the reference list — all 12 references are to external work (McCauley et al., Fitzgerald et al., Zeller, Parnin & Orso, etc.). No uniqueness theorem or prior ansatz from the authors is invoked. The paper explicitly scopes its validation: 'It validates the artifact rather than the pedagogy' (Section 5), which is an honest boundary statement, not a circular deflection. The skeptic's concern about the hypothesis checkpoint depending on voluntary student action in Evaluation Mode is a legitimate design limitation and correctness risk, but it is not circularity: the paper does not claim to derive the checkpoint's diagnostic value from its own inputs by construction. The derivation chain from design (Section 4) to implementation to validation (Section 5) is straightforward engineering, not a self-referential loop.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

DebugTracker is a software tool, not a theoretical derivation. It introduces no new mathematical constants, fitted parameters, or postulated physical entities. The 'axioms' are domain assumptions about the tool's operating environment and its intended use case.

assumptions (3)
  • domain assumption VS Code extension APIs (editor, debugger, terminal) are stable and sufficient to capture meaningful debugging process evidence.
    The tool's design depends entirely on the availability and reliability of these standard VS Code mechanisms (Section 4).
  • domain assumption The captured metadata (test commands, editor navigation, debugger lifecycle, student checkpoints) is sufficient to reconstruct a meaningful debugging narrative without recording full screen or keystrokes.
    The paper states it 'records task-relevant metadata rather than full keystroke replay' (Section 4) and assumes this is enough for instructor review.
  • domain assumption Instructors can reliably distinguish systematic debugging from guessing using the exported timeline and Markdown reports.
    The paper claims the trace lets an instructor answer questions like 'Hypothesis before edit?' (Table 1) but provides no empirical evidence that instructors can actually do this reliably from the reports alone.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DebugTracker: Lightweight Process Evidence for Classroom Debugging." pith.science (2026). https://pith.science/paper/GNSNXMMR

@misc{pith2026260705871,
  author       = {Pith},
  title        = {Pith review of: DebugTracker: Lightweight Process Evidence for Classroom Debugging},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GNSNXMMR}},
  note         = {Machine review of arXiv:2607.05871}
}
read the original abstract

Debugging exercises are often assessed from final code and test outcomes, yet these artifacts hide how students reproduced failures, formed hypotheses, inspected evidence, edited code, and verified fixes. We present DebugTracker, a Visual Studio Code extension that records lightweight debugging-process evidence for classroom tasks. DebugTracker separates uncoached Evaluation Mode traces from coached Training Mode traces, stores append-only JSONL events, and exports timeline and Markdown reports for human review. The prototype records test commands, editor and debugger metadata, student checkpoints, source snapshots, optional image evidence, human labels, and optional AI-assisted practice feedback. DebugTracker is largely language-agnostic: it captures process evidence through standard VS Code mechanisms rather than language-specific tooling, although debugger evidence depends on the relevant VS Code language extension. We validate the prototype with debugging tasks in Python, TypeScript, and Java, 16 automated checks, and an 11-case manual trial matrix spanning packaged VSIX installation and three operating systems.

Figures

Figures reproduced from arXiv: 2607.05871 by the authors.

Figure 1
Figure 1. DebugTracker workflow from session mode to reviewable evidence. A session begins in one of two mutually exclusive [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Exported Markdown report excerpt and syntax [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 13 canonical work pages

  1. [1]

    Neil Christopher Charles Brown, Michael Kölling, Davin McCall, and Ian Utting

    work page

  2. [2]

    In Proceedings of the 45th ACM Technical Symposium on Computer Science Education (SIGCSE ’14)

    Blackbox: A Large Scale Repository of Novice Programmers’ Activity. In Proceedings of the 45th ACM Technical Symposium on Computer Science Education (SIGCSE ’14). ACM, New York, NY, USA, 223–228. doi:10.1145/2538862.2538924

    work page doi:10.1145/2538862.2538924

  3. [3]

    Sue Fitzgerald, Gary Lewandowski, Renee McCauley, Laurie Murphy, Beth Simon, Lynda Thomas, and Carol Zander. 2008. Debugging: Finding, Fixing and Flailing, a Multi-Institutional Study of Novice Debuggers.Computer Science Education18, 2 (2008), 93–116. doi:10.1080/08993400802114508

    work page doi:10.1080/08993400802114508 2008

  4. [4]

    Edwards, Essi Isohanni, Ari Korhonen, Andrew Petersen, Kelly Rivers, Miguel Ángel Rubio, Judy Sheard, Bronius Skupas, Jaime Spacco, Claudia Szabo, and Daniel Toll

    Petri Ihantola, Arto Vihavainen, Alireza Ahadi, Matthew Butler, Jürgen Börstler, Stephen H. Edwards, Essi Isohanni, Ari Korhonen, Andrew Petersen, Kelly Rivers, Miguel Ángel Rubio, Judy Sheard, Bronius Skupas, Jaime Spacco, Claudia Szabo, and Daniel Toll. 2015. Educational Data Mining and Learning Analytics in Programming: Literature Review and Case Studi...

    work page doi:10.1145/2858796.2858798 2015

  5. [5]

    Ko and Brad A

    Andrew J. Ko and Brad A. Myers. 2004. Designing the Whyline: A Debugging Interface for Asking Questions about Program Behavior. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’04). ACM, New York, NY, USA, 151–158. doi:10.1145/985692.985712

    work page doi:10.1145/985692.985712 2004

  6. [6]

    Renee McCauley, Sue Fitzgerald, Gary Lewandowski, Laurie Murphy, Beth Simon, Lynda Thomas, and Carol Zander. 2008. Debugging: A Review of the Literature from an Educational Perspective.Computer Science Education18, 2 (2008), 67–92. doi:10.1080/08993400802114581

    work page doi:10.1080/08993400802114581 2008

  7. [7]

    Microsoft. 2026. Visual Studio Code Extension API. https://code.visualstudio. com/api. Accessed 2026-06-24

    work page 2026

  8. [8]

    Chris Parnin and Alessandro Orso. 2011. Are Automated Debugging Techniques Actually Helping Programmers?. InProceedings of the 2011 International Sym- posium on Software Testing and Analysis (ISSTA ’11). ACM, New York, NY, USA, 199–209. doi:10.1145/2001420.2001445

    work page doi:10.1145/2001420.2001445 2011

Show all 13 references
  1. [9]

    Price, David Hovemeyer, Kelly Rivers, Ge Gao, Austin Cory Bart, Ayaan M

    Thomas W. Price, David Hovemeyer, Kelly Rivers, Ge Gao, Austin Cory Bart, Ayaan M. Kazerouni, Brett A. Becker, Andrew Petersen, Luke Gusukuma, Stephen H. Edwards, and David Babcock. 2020. ProgSnap2: A Flexible For- mat for Programming Process Data. InProceedings of the 2020 AC...

    2020 doi

  2. [10]

    Murphy, and Kris De Volder

    Jonathan Sillito, Gail C. Murphy, and Kris De Volder. 2006. Questions Program- mers Ask during Software Evolution Tasks. InProceedings of the 14th ACM SIG- SOFT International Symposium on Foundations of Software Engineering (SIGSOFT FSE ’06). ACM, New York, NY, USA, 23–34. doi...

    2006 doi

  3. [11]

    Hollingsworth, and Nelson Padua-Perez

    Jaime Spacco, David Hovemeyer, William Pugh, Fawzi Emad, Jeffrey K. Hollingsworth, and Nelson Padua-Perez. 2006. Experiences with Marmoset: Designing and Using an Advanced Submission and Testing System for Program- ming Courses. InProceedings of the 11th Annual SIGCSE Conferen...

    2006 doi

  4. [12]

    Eric Wong, Ruizhi Gao, Yihao Li, Rui Abreu, and Franz Wotawa

    W. Eric Wong, Ruizhi Gao, Yihao Li, Rui Abreu, and Franz Wotawa. 2016. A Survey on Software Fault Localization.IEEE Transactions on Software Engineering 42, 8 (2016), 707–740. doi:10.1109/TSE.2016.2521368

    2016 doi

  5. [13]

    2009.Why Programs Fail: A Guide to Systematic Debugging(2 ed.)

    Andreas Zeller. 2009.Why Programs Fail: A Guide to Systematic Debugging(2 ed.). Morgan Kaufmann, Burlington, MA, USA. https://www.sciencedirect.com/ book/9780123745156/why-programs-fail

    2009

Pith tools

Reviewed July 8, 2026 · model on record in the stance chip above.