{"id":"31318d39-5855-4626-8a87-88066f40111e","arxiv_id":"2506.21266","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"KOALA provides a configurable, open-source tool for collecting detailed IDE activity data from students, including novel data like hotkey usage, and converts it to a standard format.","lead":"KOALA is a configurable IDE plugin that records code snapshots and student interactions while they solve programming tasks in JetBrains IDEs. It is designed to give computing education researchers flexible, standardized data collection without writing custom plugins.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Event capture is asserted but not validated: no ground-truth check for hotkey/focus events, and source code is only promised, so the novel data types at the center of the claim are unverified.","rationale":"The reader's conditional verdict centers on the absence of validation of data collection accuracy and non-intrusiveness. My stress-test converges on the same point: the novel contributions named in the abstract—employed hotkeys and focus switches—are precisely the data types most dependent on correct event capture, and no evidence is supplied that capture is complete or that the plugin does not distort student behavior. I am not claiming the implementation is wrong; the IntelliJ Platform does provide action and editor event APIs, so such a plugin is feasible. The issue is that the paper does not demonstrate that this particular implementation uses those APIs reliably, and its single case study only reports aggregate counts. A scripted ground-truth test would settle the question directly, and if the tool passes it, the conditional verdict can be upgraded. The manuscript's self-stated 'upon acceptance' open-sourcing further blocks current independent audit, reinforcing the condition rather than overturning the verdict.","tokens_in":11166,"tokens_out":4957,"duration_ms":63791,"concrete_test":"Use the IntelliJ test framework to run a scripted session: execute a fixed sequence of, say, 50 IDE actions, each triggered once by keyboard shortcut, once by menu, and once by toolbar; switch editor focus among three files; and type known text in a task file. Run KOALA with the §4.1 default configuration and compare its activity_data.csv and fileeditor_data.csv to ground truth. Report recall and precision for hotkey and focus events, plus end-to-end latency overhead. If hotkey recall is not 100% or focus events are missing under split-view or refactoring operations, the §5 dataset cannot be treated as a complete record of student behavior.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that KOALA faithfully records code snapshots and IDE feature usage, with employed hotkeys and focus switches as novel data types. For that claim to hold, the IntelliJ Platform SDK must deliver every relevant action trigger and editor-focus transition, and the plugin must not miss or misattribute them. Section 4.1 describes what is captured but provides no accuracy check. In particular, hotkey detection depends on inferring the trigger from an action event's InputEvent, and focus changes must be reconstructed from editor/file listeners; both are vulnerable to systematic gaps (non-standard keymaps, popups, programmatic action invocations, split views, tool-window focus). Section 5 reports counts such as 21 thousand hotkey usages, but there is no ground-truth comparison, no precision/recall measurement, and no overhead measurement. The case study therefore cannot establish that the dataset is complete or that the plugin is non-intrusive. Additionally, the manuscript promises the plugin will be open-sourced only 'upon acceptance', despite a GitHub footnote, so the capture logic cannot currently be audited independently.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents KOALA, a tool for collecting fine-grained IDE interaction data from students solving programming tasks in JetBrains IDEs. KOALA consists of an IDE plugin, a data-collection server, a visualization dashboard, and a converter to the ProgSnap2 format. The plugin is configured through YAML files that control task content, IDE settings, surveys, code-snapshot granularity, and activity-event frequency. The authors report a case study with 28 students completing tasks in two JetBrains Academy in-IDE courses, yielding over 585 thousand activity events and 127 thousand code snapshots, and they illustrate the data with a ranking of the most-used hotkeys in a refactoring course.","tokens_in":11396,"tokens_out":3804,"duration_ms":44282,"significance":"If the tool performs as described, KOALA would be a valuable contribution to computing-education research by providing a configurable, standardized way to collect code snapshots and IDE activity data across JetBrains IDEs, including data types such as hotkey usage and file-focus changes that are not commonly available in prior tools. The paper's strengths include a clear architecture, a detailed configuration taxonomy, built-in privacy controls, a dashboard, and ProgSnap2 compatibility, as well as a non-trivial dataset from 28 students. The prior use of KOALA in a published study is also a positive indicator. However, the central claim of accurate and non-intrusive event capture is not yet validated, and the source code and dataset are only promised upon acceptance.","major_comments":[{"comment":"The central claim that KOALA completely and accurately records IDE actions, hotkeys, and focus changes is not supported by any validation. Hotkey detection appears to depend on inferring the trigger from an action event's InputEvent, and focus changes are reconstructed from editor/file listeners; both are vulnerable to systematic gaps from non-default keymaps, programmatic action invocations, popups, split editors, and tool-window focus. No precision/recall measurement, ground-truth comparison, or event-loss analysis is reported for any of the collected event types. Please add an accuracy evaluation, for example by replaying scripted action sequences in a controlled IDE session and measuring detection rates, and document the assumptions and failure modes of the event handlers.","section":"Section 4.1, 'Collected data'"},{"comment":"The case study reports collecting 127 thousand code snapshots at 'the most granular setting possible' (after each keystroke) and 585 thousand activity events, but it provides no measurement of the plugin's performance overhead or its effect on student behavior. Since the paper motivates the tool partly by the need for non-intrusive data collection, the absence of CPU, memory, latency, or user-experience measurements leaves the non-intrusiveness claim untested. Please include at least basic overhead measurements or a qualitative comparison of IDE responsiveness with and without the plugin.","section":"Section 5, 'Study setup' and 'Collected data'"},{"comment":"The manuscript is internally inconsistent about artifact availability: footnote 1 lists a GitHub repository for KOALA, but Section 4.1 states that 'upon acceptance, the plugin will be fully open-sourced,' and Section 5 states that the full dataset will be shared 'upon acceptance.' For a tool paper, the event-capture logic and the raw data are necessary to independently assess the main claims. Please make the repository and dataset available at review time, or provide a sufficiently detailed technical appendix covering the event-handler implementations, the CSV schema, and the anonymization procedure.","section":"Sections 4.1 and 5, 'Output format'"},{"comment":"The illustrative analysis in Figure 4 shows counts of hotkey usage, but it does not demonstrate that the newly collected data types (hotkeys and focus switches) are accurate or that they enable analyses beyond what existing tools' logs already provide. The claim that these data are 'not collected in prior works' is plausible, but it would be substantially strengthened by a concrete comparison with data obtainable from a prior tool such as TaskTracker, or by showing that the focus/hotkey data yield a non-trivial behavioral insight rather than a simple ranking.","section":"Section 5, 'Collected data' and 'Possible applications'"}],"minor_comments":[{"comment":"The CSV output schema is not described in the text; readers are left to infer it from Figure 2. A table or appendix listing each data type, its fields, and the triggering condition would make the tool easier to adopt and verify.","section":"Section 4.1, 'Collected data'"},{"comment":"The hotkey labels in Figure 4 appear garbled in the provided version, making it difficult to read which shortcuts are being counted; please ensure the figure renders clearly in the final PDF.","section":"Figure 4"},{"comment":"The description of Test My Code states that the server side is 'actively supported' while the IDE plugins are no longer updated; please clarify whether this is a fully maintained option or a legacy system, since Table 1 lists it as a current comparison point.","section":"Section 2.1, 'Data Collection Tools'"},{"comment":"The privacy section says students explicitly consent to collection of personal data such as name and email, while Section 5 says the distributed data is 'without students' personal information'; please clarify whether personal information is never sent to the server or is removed during the export step.","section":"Section 4.1, 'Privacy policy'"},{"comment":"The step numbering in Figure 1 appears to be inconsistent with the accompanying textual list (steps are labeled 1, 2, 3, 4, 6, 5); please check the figure labels and the narrative order.","section":"Section 3, 'KOALA: Usage Pipeline'"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for the venue and reports a useful tool. The main concern is that the paper's central claims about complete and non-intrusive event capture are not yet validated; this is fixable with an additional validation study and artifact availability. I would encourage the editor to treat the 'upon acceptance' availability of source code and data as a condition for final acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Karol et al. present KOALA, a JetBrains IDE plugin for collecting code snapshots and activity data from students. The tool fills a genuine gap: TaskTracker is unmaintained and requires extra plugins, Test My Code is effectively dead on the IDE side, and none of the existing tools capture hotkey usage or focus switches as first-class data types. KOALA's YAML configuration is a real step forward—researchers can control task order, IDE settings, snapshot granularity, surveys, and even third-party log files without touching plugin code. The architecture is straightforward and the ProgSnap2 converter plus dashboard make the tool practical. The paper is clearly written, the comparison table is informative, and the case study gives a concrete sense of the data volume and format.\n\nThe soft spots are real. The central claim that the plugin reliably captures hotkeys and focus events is not validated. Section 4.1 says what is collected, but there is no check against ground truth, no precision/recall for action or focus events, no measurement of plugin overhead, and no discussion of known failure modes (custom keymaps, programmatic action triggers, focus changes via tool windows). The 21k hotkey usages in the case study are just counts. For a tool whose selling point is fine-grained IDE interaction data, that is a meaningful gap, though not a fatal one—it can be fixed with a small validation study.\n\nI also note that the manuscript says the plugin will be fully open-sourced \"upon acceptance\" and the dataset shared \"upon acceptance\", despite a GitHub link in the introduction. That mismatch should be cleaned up; ideally the code and data are available at submission so reviewers can audit the capture logic. The claim that the tool is \"convenient\" is not empirically supported—no usability data, just configuration examples—but that is a minor wording issue.\n\nThe circularity burden is low: the tool is not being fitted to explain its own data, and the previous use in Birillo et al. is an external application, not a validation. The case study is limited but appropriately described as a showcase. The dataset, once released, will be a contribution on its own.\n\nWho is this for? Computing education researchers who need a configurable JetBrains-based data collection pipeline, and tool builders looking for a reference architecture. A serious referee should engage with it, mainly to push on the validation and availability points. I would support sending it to review, with the expectation of a major revision that adds event-capture validation and makes code/data available.","headline":"A useful, well-described IDE data collection tool whose novelty is real but whose event-capture accuracy is asserted, not yet demonstrated.","tokens_in":11902,"tokens_out":2034,"would_cite":false,"duration_ms":23728,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"KOALA is a configurable plugin for JetBrains IDEs that records students' code, actions, and hotkeys during programming tasks, with output convertible to the standard ProgSnap2 format.","keywords":["data collection","IDE interaction data","code snapshots","JetBrains IDEs","ProgSnap2","computing education","activity tracking","in-IDE learning"],"falsifier":"Record a session in a JetBrains IDE on video or with an independent keylogger while KOALA runs at its finest granularity, then compare the logs event by event; any unlogged keystroke, hotkey, or focus switch, or any phantom event, would show that the capture is not complete.","tokens_in":11026,"feed_emoji":"⌨️","tokens_out":8758,"duration_ms":84825,"temperature":0.7,"pith_summary":"KOALA is an open-source plugin-and-server system for JetBrains IDEs that records what students do while solving programming tasks: code snapshots taken at a configurable frequency (down to every keystroke), IDE actions such as running and debugging, hotkey usage, file-focus switches, tool-window changes, and survey answers. The paper's claim is that this tool overcomes the two main limitations of earlier data-collection tools: it is configurable through YAML files without touching source code, and it captures event types (hotkeys, focus changes) that previous JetBrains tools did not record. The collected data is stored on a bundled server and can be converted to the standard ProgSnap2 format, so researchers can reuse existing analysis tools. To show it works, the paper reports a case study with 28 students solving tasks in two in-IDE courses, collecting over 127,000 code snapshots and 585,000 activity events. A sympathetic reader would take the central contribution to be a ready-to-use, standardized measurement instrument for computing-education research.","feed_headline":"Configurable JetBrains IDE plugin logs code, actions, and hotkeys","feed_subtitle":"A configurable plugin records code, hotkeys, and IDE events, then exports to ProgSnap2 for analysis.","key_machinery":"Central to the design is the YAML configuration layer, which controls the plugin's behavior through several files—scenario, task content, IDE settings and inspections, survey, activity data, code tracking, and research metadata. A plugin built on the IntelliJ Platform Plugin SDK reads these files and subscribes to IDE events, so a single codebase can capture data across all JetBrains IDEs without recompilation. The same plugin-server architecture used by earlier JetBrains trackers is reused, with the addition of a converter that exports the raw CSV records into the ProgSnap2 format, making the tool's output interoperable with existing analysis tools.","core_discovery":"KOALA is a plugin for all JetBrains IDEs, built on the IntelliJ Platform SDK, that lets a researcher or educator define a whole study through YAML configuration files: the sequence of tasks and surveys, which files the student will work in, which IDE settings and inspections to enable or disable (e.g., code completion), how often code snapshots should be taken, and which activities to track. During a session, the plugin records the student's code at the configured granularity—in the case study, the full file after every keystroke—together with all IDE actions, run/debug invocations, keyboard shortcuts, file-editor focus changes, and tool-window openings, writing them to CSV files. The data is uploaded to a Ktor-based server (provided as a Docker image) when the student submits or pauses, and can then be inspected on an interactive dashboard and converted into ProgSnap2, the standard format for programming-process data. The paper demonstrates the pipeline on 28 students from two courses using the JetBrains Academy in-IDE learning plugin, obtaining tens of thousands of snapshots and hundreds of thousands of events, and uses the newly available hotkey data to show which refactoring shortcuts students favored. The tool's stated purpose is to make fine-grained IDE interaction data easy to collect, standardize, and share for computing-education research.","pith_inferences":["If event capture is validated against ground truth, such as screen recordings or an independent keylogger, KOALA could become a shared measurement benchmark that lets different research groups compare IDE interaction patterns quantitatively.","The YAML-driven design points toward a natural extension in which IDE settings are toggled per student cohort, enabling causal experiments on how features like code completion affect learning outcomes.","Because only IntelliJ-platform events are logged, anything happening outside the IDE—browser searches, notes, OS-level distractions—remains invisible, so a complete picture would require supplementing KOALA with other sensors.","The dataset's long-term value depends on the promised public release of both the anonymized data and the plugin source code, since reproducibility of the instrumentation is what would let other groups build on the measurements."],"forward_implications":["Researchers can run the same configuration across IntelliJ IDEA, PyCharm, and CLion, so a single study design can capture Java, Python, and C++ problem solving without rewriting the tool.","Studies can now measure IDE feature adoption directly—hotkeys, run/debug invocations, focus switches—rather than inferring it from code snapshots alone.","Because data converts to ProgSnap2, datasets collected with KOALA can be combined with existing shared datasets and analyzed with tools already built for that format.","The configurable snapshot granularity lets a study trade data volume against detail, from full-file-per-keystroke histories to coarse summaries such as function names on save.","The case study's hotkey ranking in a refactoring course (Reformat, Inline, Move, Introduce constant) gives a concrete example of insights that were not obtainable from earlier JetBrains trackers."],"supporting_citations":[{"why":"DevEventTracker, the Eclipse-based baseline whose save-only snapshot granularity and lack of customization KOALA positions against.","marker":"[34]"},{"why":"TaskTracker, the earlier JetBrains tool whose plugin-server architecture KOALA follows and whose configuration limits and extra-plugin requirements it claims to remove.","marker":"[41]"},{"why":"Test My Code, the widely used NetBeans and JetBrains plugin set that lacks per-task IDE customization and serves as a comparison baseline.","marker":"[46]"},{"why":"ProgSnap2, the standardized format KOALA exports to, making its data usable with existing analysis tools.","marker":"[47]"},{"why":"The IntelliJ Platform SDK paper, the framework that lets a single plugin capture IDE events across all JetBrains IDEs.","marker":"[37]"},{"why":"The in-IDE learning courses used in the case study, defining the setting KOALA is designed for.","marker":"[25]"},{"why":"A prior study that already used KOALA to collect interaction data in JetBrains IDEs, cited as evidence of practical usefulness.","marker":"[24]"}],"fun_headline_variants":["KOALA plugin tracks every keystroke and IDE action for education research","Configurable IDE tool captures code, keystrokes, and focus changes","New plugin makes live IDE data collection for students easy","KOALA: open-source tool for fine-grained IDE process data","Study student coding with KOALA's full IDE activity tracking"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The plugin's event capture is complete and does not disturb students, meaning the IDE delivers every relevant keystroke, hotkey, action, and focus change to the plugin, and the plugin's presence does not alter how students solve tasks.","fun_headline_variants_meta":{"raw":{"variants":["KOALA plugin tracks every keystroke and IDE action for education research","Configurable IDE tool captures code, keystrokes, and focus changes","New plugin makes live IDE data collection for students easy","KOALA: open-source tool for fine-grained IDE process data","Study student coding with KOALA's full IDE activity tracking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000245,"raw_usage":{"total_tokens":1590,"prompt_tokens":1054,"completion_tokens":536,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":670,"completion_tokens_details":{"reasoning_tokens":448}},"tokens_in":670,"tokens_out":536,"duration_ms":6016,"temperature":1.0,"reasoning_tokens":448,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:27:52.692154+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record a session in a JetBrains IDE on video or with an independent keylogger while KOALA runs at its finest granularity, then compare the logs event by event; any unlogged keystroke, hotkey, or focus switch, or any phantom event, would show that the capture is not complete.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DevEventTracker, the Eclipse-based baseline whose save-only snapshot granularity and lack of customization KOALA positions against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TaskTracker, the earlier JetBrains tool whose plugin-server architecture KOALA follows and whose configuration limits and extra-plugin requirements it claims to remove."},{"cited_title":"In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education","cited_arxiv_id":null,"evidence_quote":"Test My Code, the widely used NetBeans and JetBrains plugin set that lacks per-task IDE customization and serves as a comparison baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ProgSnap2, the standardized format KOALA exports to, making its data usable with existing analysis tools."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The IntelliJ Platform SDK paper, the framework that lets a single plugin capture IDE events across all JetBrains IDEs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The in-IDE learning courses used in the case study, defining the setting KOALA is designed for."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A prior study that already used KOALA to collect interaction data in JetBrains IDEs, cited as evidence of practical usefulness."}],"review_version":1}