{"id":"e6c009d1-954b-4461-b14f-cf3258fd10c7","arxiv_id":"2507.15831","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A fine-grained logging toolset and a 100-hour, 9,207-execution Jupyter notebook dataset show that developers mostly re-execute cells for iterative code fixes, not exploration.","lead":"The paper presents a plugin, server, and scripts that log every cell edit and execution in Jupyter notebooks, and uses them to record over 100 hours of work from 20 developers. It releases a new dataset and reports that most notebook re-executions are small code fixes and tweaks rather than open-ended exploration, suggesting Jupyter needs better debugging and inspection tools.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central debugging-versus-exploration claim rests on GPT-4o purpose labels whose only reported validation is a 0.55 overlap with human labels; per-label error rates are not reported, so the headline 72.2% vs 14.1% split is not yet shown to be robust.","rationale":"The reader identified the same weakest assumption: GPT-4o purpose-label accuracy is the load-bearing premise for RQ2 and for the paper's headline interpretation. The paper's own threat section flags that the annotations cannot be fully trusted, but it does not test how label error would affect the reported proportions. This is a genuine, concrete measurement-validity concern, not a merely stylistic objection. The dataset and tooling contributions are real and independently useful, and the paper is honest about many limitations, so the appropriate outcome remains CONDITIONAL as the reader concluded. My stress-test pass does not change that verdict: the paper should add per-label label-quality analysis and uncertainty bounds around the RQ2 percentages, reconcile the inconsistent 25.6% statistic, and provide a commit-pinned repository. No additional fatal flaw was found in the logging tooling or in the RQ1 transition-count analysis.","tokens_in":20034,"tokens_out":2810,"duration_ms":34213,"concrete_test":"On the existing 400-transition gold sample, compute the full per-label precision, recall, and confusion matrix for GPT-4o relative to the human ground truth. Then apply a noise-correction (e.g., matrix-inversion or bootstrap) to the 3,573 GPT-4o labels and report corrected proportions and confidence intervals for code iteration versus exploration. If the corrected exploration share is not significantly below the corrected code-iteration share, the debugging-over-exploration conclusion does not follow. An additional check would be to have two human annotators label a new random sample of 300 previously unlabeled self-transitions and compare the resulting proportions with the GPT-4o-based Figure 5.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's main conclusion—that notebooks are used more for debugging and code iteration than for exploration—comes almost entirely from RQ2, where GPT-4o labeled all 3,573 self-transitions (Section 7.2). The only reported accuracy check is that GPT-4o and the human ground truth share at least one label on 0.55 of the 400 manually annotated transitions (Section 7.2, Section 8.1). This is a lenient overlap metric, not a per-label precision/recall or confusion matrix. Because the label distribution is highly unbalanced—'edit code' alone is 45.1% of labels while 'explore variable' is 11.4%—a model biased toward 'edit code' would simultaneously inflate the code-iteration category (72.2%) and deflate exploration (14.1%). The paper acknowledges in Section 8.1 that it 'cannot fully trust' the GPT-4o annotations, but the subsequent assertion that the effects are 'strong statistical effects' is not a substitute for quantifying the label noise. If GPT-4o systematically misclassifies exploration-oriented edits as 'edit code' or 'fix', the central claim that tooling should prioritize debugging over exploration is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops and releases a toolchain for recording fine-grained Jupyter notebook interactions (a browser plugin, a Flask server, post-processing scripts, and a notebook re-executor), uses it to collect a new dataset, JuNE, from 20 developers working on data-analysis and machine-learning tasks for more than 100 hours, and analyzes the resulting 9,207 execution events. It defines self-transitions and inter-transitions between cell executions, characterizes change sizes and output types (RQ1), labels self-transition change purposes using manual open coding plus GPT-4o (RQ2), and labels data science steps using a DASWOW-derived CatBoost classifier (RQ3). The central empirical conclusion is that notebook development is highly non-linear and that most self-transition work is code iteration/debugging rather than exploration, which the authors use to motivate better debugging tooling for Jupyter.","tokens_in":20241,"tokens_out":6093,"duration_ms":60287,"significance":"If the RQ2 and RQ3 labelings can be validated, the paper would make a substantial empirical contribution: it provides the research community with a reusable instrumentation pipeline, a new fine-grained dataset of realistic notebook development, and a concrete, testable claim about tooling priorities (debugging over exploration). The release of the dataset, code, and intermediate results is a clear strength, as is the honest discussion of the annotation models' limitations. The current manuscript, however, does not yet demonstrate that the central proportions are robust to annotation noise, so the study's headline conclusion is not yet fully supported.","major_comments":[{"comment":"The validation of the GPT-4o purpose labels is not sufficient to carry the central RQ2 claim. The only reported check is a 0.55 overlap on 400 transitions, defined as sharing at least one label, and the ground truth for those 400 transitions was itself produced by comparing the human open-coding labels with GPT-4o's labels (Section 6.2), so the benchmark is not an independent human judgment. No per-label precision/recall, no confusion matrix, and no uncertainty intervals are reported for the 72.2% code-iteration and 14.1% exploration proportions. Because 'edit code' alone is 45.1% of labels and the class distribution is highly unbalanced, a systematic bias toward code-iteration labels could easily produce these aggregate numbers even if the true exploration share is higher. The statement in Section 8.1 that the conclusions rest on 'strong statistical effects' is not a substitute for quantifying the label noise. Please report per-label agreement metrics, a confusion matrix against the human labels, bootstrap confidence intervals for the headline proportions, and a sensitivity analysis under alternative labelings.","section":"Section 7.2 and Section 8.1"},{"comment":"The RQ3 transition analysis relies on the CatBoost classifier with F1=0.72, which is validated only on the DASWOW corpus and then applied to the new JuNE execution logs. DASWOW consists of standalone notebook cells from GitHub, whereas JuNE cells are development-time executions, so distribution shift is a real risk that is not assessed. Moreover, the diagonal dominance reported in Section 7.3.2 is at least partly an artifact of using a single deterministic classifier to label both endpoints of every transition; the statement that this dominance 'underscores ... the stability of our machine learning annotation model' is circular. Please validate the data-science-step labels on a sample of JuNE cells and report transition probabilities with confidence intervals, or otherwise show that the F1 measured on DASWOW transfers.","section":"Section 6.3 and Section 7.3"},{"comment":"The paper does not specify whether the reported percentages are computed over labels or over transitions. The Figure 5 caption says 'some code transitions may include more than one change purpose,' yet the text and abstract move interchangeably between '72.2% of the change purposes in self-transitions' and '14.1% of transitions.' If a transition can carry multiple labels, label-based proportions overcount transitions that received several purposes and do not support statements about the fraction of transitions. Please state the denominator explicitly and report both label-based and transition-based proportions.","section":"Section 6.2 and Section 7.2"}],"minor_comments":[{"comment":"The body text says '5% of the cells with at least one re-execution account for 25.6% of all re-executions,' but the caption of Figure 4a says 'cells with only one re-execution account for 25.6% of all re-executions.' These are different claims; please correct one of them.","section":"Section 7.1 and Figure 4a"},{"comment":"The caption of Figure 4c says error outputs represent 'only one third' of outputs before re-execution, while the body text says 'more than 20% of outputs prior to re-execution are error messages.' Please align these numbers or explain why they refer to different quantities.","section":"Section 7.1 and Figure 4c"},{"comment":"The discussion of the 327 GPT-4o 'no change' labels versus the 306 string-identical transitions should state whether the 8.8% proportion in Figure 5 uses GPT-4o labels or the corrected count; this affects the interpretation of the 'no change' category.","section":"Section 7.2"},{"comment":"Reference [18] is incomplete: it gives a journal name and year but no article title or author. Please complete the citation.","section":"References"},{"comment":"The open-coding step is described as reaching 'full agreement' after discussion, but no inter-annotator agreement statistic (e.g., Cohen's kappa) is reported for the two annotators before reconciliation. Please add it or explain why it is omitted.","section":"Section 6.2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe main reason to read this paper is the artifact: a working plugin, server, post-processing stack, and replay tool for fine-grained Jupyter execution logs, plus the JuNE dataset (20 developers, 29 notebooks, about 9.2k executions, over 100 hours). That fills a real gap—there is no other released corpus at this resolution. The engineering is careful and the dataset deserves to be used.\n\nThe paper also does solid qualitative work: the open coding of 400 self-transitions and the 11-category purpose taxonomy are sensible, and RQ1's descriptive findings (39% self-transitions, small edit distances, error outputs before re-execution) are reproducible from the logs. I trust those.\n\nThe soft spot is exactly where the stress test points. The central claim—that notebooks are leveraged for debugging rather than exploration—rests on the 72.2% vs 14.1% split from GPT-4o labels on all 3,573 self-transitions. The only validation is 0.55 overlap with human labels on 400 transitions, and that is a lenient 'share at least one label' metric, not per-label precision or a confusion matrix. With an unbalanced distribution (edit code alone is 45.1%), a model biased toward the majority class would inflate code iteration and deflate exploration. The paper admits in Sec 8.1 that it cannot fully trust the annotations, then asserts the effects are strong without quantifying label noise. That assertion is doing a lot of work. The fix is straightforward: report a confusion matrix for GPT-4o versus the 400 ground-truth transitions, per-label precision/recall, and a noise-aware bound on the proportions. Until then, treat the debugging-vs-exploration conclusion as suggestive.\n\nTwo smaller issues. First, the text says '5% of the cells with at least one re-execution account for 25.6% of all re-executions,' while the Figure 4a caption says 'cells with only one re-execution account for 25.6%'—these are incompatible. Second, RQ3 relies on the CatBoost classifier at F1=0.72; the paper acknowledges this, but the diagonal dominance of the transition matrices is partly a label-persistence artifact and should not be over-interpreted.\n\nThe self-citations are heavy but mostly legitimate since the data-science taxonomy comes from the authors' own prior work. A commit-pinned repository would help reproducibility.\n\nWho is this for: anyone building notebook tooling or studying data-science workflows empirically. The dataset justifies refereeing. I would send it to review with a request for major revision: fix the statistic, add uncertainty to the label-based proportions, and soften the central claim accordingly.","headline":"Valuable dataset and tooling for fine-grained Jupyter logs; the headline debugging-vs-exploration claim is not yet backed by the label validation.","tokens_in":20871,"tokens_out":3206,"would_cite":true,"duration_ms":31357,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper uses fine-grained execution logs from 20 developers to argue that Jupyter notebook development is dominated by small iterative code-fixing cycles (39% self-transitions; 72.2% code iteration vs 14.1% exploration), so notebook…","keywords":["Jupyter notebooks","fine-grained logs","execution traces","code evolution","self-transitions","change purposes","data science workflow","empirical study"],"falsifier":"Have expert annotators label a random sample of the 3,573 self-transitions (or all of them) and compute the code-iteration versus exploration shares; if the 72.2% figure falls toward the 14.1% exploration share after correction, the conclusion that notebooks are used mainly for debugging collapses.","tokens_in":19759,"feed_emoji":"📓","tokens_out":5058,"duration_ms":50071,"temperature":0.7,"pith_summary":"Using logs of every cell execution from 20 developers working more than 100 hours on data-analysis and machine-learning tasks, the paper shows that notebook development is dominated by small, repeated changes to the same cell rather than by exploration of rich outputs. Re-executing the same cell accounts for 39% of all transitions, and among those re-executions 72.2% of the change purposes are code iteration—fixing, debugging, cleaning, or readability edits—while only 14.1% are exploration. The authors conclude that the interactive nature of Jupyter is leveraged mainly for debugging and inspection, and argue that notebook tooling should therefore prioritize IDE-like debugging support. The contribution is a reusable data-collection toolset and the JuNE dataset of 2,655 cells and 9,207 executions, offered to enable further study of notebook development.","feed_headline":"72% of notebook re-runs are code iteration, not exploration","feed_subtitle":"Fine-grained logs of 100+ hours of Jupyter work suggest tooling should focus on debugging and inspection.","key_machinery":"The machinery is the transition model: the log of each notebook is converted into a series of transitions from one executed cell to the next, split into inter-transitions (moving to another cell) and self-transitions (re-executing the same cell). Self-transitions carry the change analysis: edit distance between consecutive versions measures change size, output types before re-execution indicate triggers, and an 11-label purpose taxonomy (obtained by manual open coding of 400 transitions, then applied at scale with GPT-4o) classifies why the code changed. Data science steps come from a single-label CatBoost/CodeBERT classifier trained on the DASWOW taxonomy, enabling transition matrices between workflow steps.","core_discovery":"The central claim is that during real notebook development, users iterate on single cells in a cyclical, small-step manner: about 39% of execution transitions are self-transitions (re-running the same cell), the average change between consecutive runs is about 13% of the cell, and the purpose labels attached to those self-transitions show code iteration (fix, debug, edit, clean, readability, comment/uncomment) at 72.2% versus exploration (explore variable, visualize data) at 14.1%. The paper interprets this as evidence that notebooks are used less as an exploratory 'rich output' medium and more as a debugging and inspection environment, and that current tooling fails to support that reality. It further finds that data science steps such as data preprocessing and exploration stay stable within a cell across self-transitions yet recur throughout the whole development session, pointing to a non-linear, entangled workflow.","pith_inferences":["A direct test of the debugging-vs-exploration interpretation would be to give a similar set of participants a notebook environment with built-in variable inspectors and debuggers and measure whether the share of self-transitions drops; the current data cannot determine this.","The participants worked on two day-long, scored tasks, so competition and time pressure may inflate code-iteration behavior; logging routine professional notebook use over weeks would show whether the pattern generalizes.","The boundary between 'exploration' and 'debugging' is not clean—printing a variable to inspect it can serve both—so future work should measure whether observable behavior (e.g., using a debugger or inspector) changes the distribution of purposes."],"forward_implications":["Self-transitions dominate enough (39%) that tools which only track notebook snapshots miss a large share of actual development work.","The high share of 'fix' (16.8%) and 'edit code' (45.1%) purposes implies that IDE-like autocomplete, syntax checking, and variable inspection could remove a large portion of re-executions.","Because data preprocessing and exploration recur throughout the session, tooling for these steps should be continuously available rather than triggered once at the start.","If the claim holds, improving debugging tools in Jupyter may matter more for user productivity than improving output rendering and visualization features."],"supporting_citations":[{"why":"Establishes the research premise that fine-grained development logs reveal changes invisible to version-control snapshots, motivating the tooling.","marker":"[1]"},{"why":"Documents the reproducibility problems of notebooks and motivates studying execution order during development.","marker":"[7]"},{"why":"Provides the baseline for notebook code quality compared to scripts; the new logs extend this to the development process.","marker":"[8]"},{"why":"Shows notebooks change substantially between versions, supporting the need for fine-grained rather than snapshot-based analysis.","marker":"[10]"},{"why":"Prior fine-grained Jupyter execution-log study of errors; the paper extends this to wider change purposes.","marker":"[11]"},{"why":"Supplies the DASWOW data science steps taxonomy and dataset used to build the classifier behind RQ3.","marker":"[13]"},{"why":"The released tooling, dataset JuNE, and materials that make the study reproducible and the data reusable.","marker":"[12]"}],"fun_headline_variants":["Jupyter study: 72% of re-runs are code iteration","Notebook re-runs: code iteration beats exploration 72–14","20 devs, 100+ hours: Jupyter re-runs mostly code iteration","Fine-grained Jupyter logs show 72% code iteration on re-runs","Notebooks as debugging tools: 72% of re-runs are code iteration"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the GPT-4o labels of change purpose are accurate enough that the 72.2%-versus-14.1% split reflects reality; the paper reports only a 0.55 agreement with human labels on 400 sampled transitions and states that it cannot fully trust the annotation.","fun_headline_variants_meta":{"raw":{"variants":["Jupyter study: 72% of re-runs are code iteration","Notebook re-runs: code iteration beats exploration 72–14","20 devs, 100+ hours: Jupyter re-runs mostly code iteration","Fine-grained Jupyter logs show 72% code iteration on re-runs","Notebooks as debugging tools: 72% of re-runs are code iteration"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000404,"raw_usage":{"total_tokens":2096,"prompt_tokens":929,"completion_tokens":1167,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":1064}},"tokens_in":545,"tokens_out":1167,"duration_ms":11038,"temperature":1.0,"reasoning_tokens":1064,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:22:27.788900+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have expert annotators label a random sample of the 3,573 self-transitions (or all of them) and compute the code-iteration versus exploration shares; if the 72.2% figure falls toward the 14.1% exploration share after correction, the conclusion that notebooks are used mainly for debugging collapses.","supporting_citations":[{"cited_title":"Negara, M","cited_arxiv_id":null,"evidence_quote":"Establishes the research premise that fine-grained development logs reveal changes invisible to version-control snapshots, motivating the tooling."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the reproducibility problems of notebooks and motivates studying execution order during development."},{"cited_title":"Grotov, S","cited_arxiv_id":null,"evidence_quote":"Provides the baseline for notebook code quality compared to scripts; the new logs extend this to the development process."},{"cited_title":"Raghunandan, A","cited_arxiv_id":null,"evidence_quote":"Shows notebooks change substantially between versions, supporting the need for fine-grained rather than snapshot-based analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior fine-grained Jupyter execution-log study of errors; the paper extends this to wider change purposes."},{"cited_title":"Ramasamy, C","cited_arxiv_id":null,"evidence_quote":"Supplies the DASWOW data science steps taxonomy and dataset used to build the classifier behind RQ3."},{"cited_title":"Titov, K","cited_arxiv_id":null,"evidence_quote":"The released tooling, dataset JuNE, and materials that make the study reproducible and the data reusable."}],"review_version":1}