Pith. sign in

REVIEW 3 major objections 5 minor 31 references

Observing Fine-Grained Changes in Jupyter Notebooks During Development Time

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper uses fine-grained execution logs from 20 developers to argue that Jupyter notebook development is dominated by small iterative code-fixing cycles (39% self-transitions; 72.2% code iteration vs 14.1% exploration), so notebook…

desk verdict Valuable dataset and tooling for fine-grained Jupyter logs; the headline debugging-vs-exploration claim is not yet backed by the label validation. read the letter →

arxiv 2507.15831 v3 pith:MYDB6XXF submitted 2025-07-21 cs.SE

Sergey Titov , Konstantin Grotov , Cristina Sarasua , Yaroslav Golubev , Dhivyabharathi Ramasamy , Alberto Bacchelli , Abraham Bernstein , Timofey Bryksin This is my paper · ORCID
classification cs.SE
keywords Jupyternotebooksfine-grainedlogsexecutiontracescodeevolutionself-transitionschangepurposesdatascienceworkflowempiricalstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Using logs of every cell execution from 20 developers working more than 100 hours on data-analysis and machine-learning tasks, the paper shows that notebook development is dominated by small, repeated changes to the same cell rather than by exploration of rich outputs. Re-executing the same cell accounts for 39% of all transitions, and among those re-executions 72.2% of the change purposes are code iteration—fixing, debugging, cleaning, or readability edits—while only 14.1% are exploration. The authors conclude that the interactive nature of Jupyter is leveraged mainly for debugging and inspection, and argue that notebook tooling should therefore prioritize IDE-like debugging support. The contribution is a reusable data-collection toolset and the JuNE dataset of 2,655 cells and 9,207 executions, offered to enable further study of notebook development.

What carries the argument

The machinery is the transition model: the log of each notebook is converted into a series of transitions from one executed cell to the next, split into inter-transitions (moving to another cell) and self-transitions (re-executing the same cell). Self-transitions carry the change analysis: edit distance between consecutive versions measures change size, output types before re-execution indicate triggers, and an 11-label purpose taxonomy (obtained by manual open coding of 400 transitions, then applied at scale with GPT-4o) classifies why the code changed. Data science steps come from a single-label CatBoost/CodeBERT classifier trained on the DASWOW taxonomy, enabling transition matrices between workflow steps.

What would settle it

Have expert annotators label a random sample of the 3,573 self-transitions (or all of them) and compute the code-iteration versus exploration shares; if the 72.2% figure falls toward the 14.1% exploration share after correction, the conclusion that notebooks are used mainly for debugging collapses.

Watch

Extended reading notes

Core claim

The central claim is that during real notebook development, users iterate on single cells in a cyclical, small-step manner: about 39% of execution transitions are self-transitions (re-running the same cell), the average change between consecutive runs is about 13% of the cell, and the purpose labels attached to those self-transitions show code iteration (fix, debug, edit, clean, readability, comment/uncomment) at 72.2% versus exploration (explore variable, visualize data) at 14.1%. The paper interprets this as evidence that notebooks are used less as an exploratory 'rich output' medium and more as a debugging and inspection environment, and that current tooling fails to support that reality. It further finds that data science steps such as data preprocessing and exploration stay stable within a cell across self-transitions yet recur throughout the whole development session, pointing to a non-linear, entangled workflow.

Load-bearing premise

The load-bearing premise is that the GPT-4o labels of change purpose are accurate enough that the 72.2%-versus-14.1% split reflects reality; the paper reports only a 0.55 agreement with human labels on 400 sampled transitions and states that it cannot fully trust the annotation.

Editorial extensions

If this is right

  • Self-transitions dominate enough (39%) that tools which only track notebook snapshots miss a large share of actual development work.
  • The high share of 'fix' (16.8%) and 'edit code' (45.1%) purposes implies that IDE-like autocomplete, syntax checking, and variable inspection could remove a large portion of re-executions.
  • Because data preprocessing and exploration recur throughout the session, tooling for these steps should be continuously available rather than triggered once at the start.
  • If the claim holds, improving debugging tools in Jupyter may matter more for user productivity than improving output rendering and visualization features.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the debugging-vs-exploration interpretation would be to give a similar set of participants a notebook environment with built-in variable inspectors and debuggers and measure whether the share of self-transitions drops; the current data cannot determine this.
  • The participants worked on two day-long, scored tasks, so competition and time pressure may inflate code-iteration behavior; logging routine professional notebook use over weeks would show whether the pattern generalizes.
  • The boundary between 'exploration' and 'debugging' is not clean—printing a variable to inspect it can serve both—so future work should measure whether observable behavior (e.g., using a debugger or inspector) changes the distribution of purposes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper develops and releases a toolchain for recording fine-grained Jupyter notebook interactions (a browser plugin, a Flask server, post-processing scripts, and a notebook re-executor), uses it to collect a new dataset, JuNE, from 20 developers working on data-analysis and machine-learning tasks for more than 100 hours, and analyzes the resulting 9,207 execution events. It defines self-transitions and inter-transitions between cell executions, characterizes change sizes and output types (RQ1), labels self-transition change purposes using manual open coding plus GPT-4o (RQ2), and labels data science steps using a DASWOW-derived CatBoost classifier (RQ3). The central empirical conclusion is that notebook development is highly non-linear and that most self-transition work is code iteration/debugging rather than exploration, which the authors use to motivate better debugging tooling for Jupyter.

Significance. If the RQ2 and RQ3 labelings can be validated, the paper would make a substantial empirical contribution: it provides the research community with a reusable instrumentation pipeline, a new fine-grained dataset of realistic notebook development, and a concrete, testable claim about tooling priorities (debugging over exploration). The release of the dataset, code, and intermediate results is a clear strength, as is the honest discussion of the annotation models' limitations. The current manuscript, however, does not yet demonstrate that the central proportions are robust to annotation noise, so the study's headline conclusion is not yet fully supported.

major comments (3)
  1. [Section 7.2 and Section 8.1] The validation of the GPT-4o purpose labels is not sufficient to carry the central RQ2 claim. The only reported check is a 0.55 overlap on 400 transitions, defined as sharing at least one label, and the ground truth for those 400 transitions was itself produced by comparing the human open-coding labels with GPT-4o's labels (Section 6.2), so the benchmark is not an independent human judgment. No per-label precision/recall, no confusion matrix, and no uncertainty intervals are reported for the 72.2% code-iteration and 14.1% exploration proportions. Because 'edit code' alone is 45.1% of labels and the class distribution is highly unbalanced, a systematic bias toward code-iteration labels could easily produce these aggregate numbers even if the true exploration share is higher. The statement in Section 8.1 that the conclusions rest on 'strong statistical effects' is not a substitute for quantifying the label noise. Please report per-label agreement metrics, a confusion matrix against the human labels, bootstrap confidence intervals for the headline proportions, and a sensitivity analysis under alternative labelings.
  2. [Section 6.3 and Section 7.3] The RQ3 transition analysis relies on the CatBoost classifier with F1=0.72, which is validated only on the DASWOW corpus and then applied to the new JuNE execution logs. DASWOW consists of standalone notebook cells from GitHub, whereas JuNE cells are development-time executions, so distribution shift is a real risk that is not assessed. Moreover, the diagonal dominance reported in Section 7.3.2 is at least partly an artifact of using a single deterministic classifier to label both endpoints of every transition; the statement that this dominance 'underscores ... the stability of our machine learning annotation model' is circular. Please validate the data-science-step labels on a sample of JuNE cells and report transition probabilities with confidence intervals, or otherwise show that the F1 measured on DASWOW transfers.
  3. [Section 6.2 and Section 7.2] The paper does not specify whether the reported percentages are computed over labels or over transitions. The Figure 5 caption says 'some code transitions may include more than one change purpose,' yet the text and abstract move interchangeably between '72.2% of the change purposes in self-transitions' and '14.1% of transitions.' If a transition can carry multiple labels, label-based proportions overcount transitions that received several purposes and do not support statements about the fraction of transitions. Please state the denominator explicitly and report both label-based and transition-based proportions.
minor comments (5)
  1. [Section 7.1 and Figure 4a] The body text says '5% of the cells with at least one re-execution account for 25.6% of all re-executions,' but the caption of Figure 4a says 'cells with only one re-execution account for 25.6% of all re-executions.' These are different claims; please correct one of them.
  2. [Section 7.1 and Figure 4c] The caption of Figure 4c says error outputs represent 'only one third' of outputs before re-execution, while the body text says 'more than 20% of outputs prior to re-execution are error messages.' Please align these numbers or explain why they refer to different quantities.
  3. [Section 7.2] The discussion of the 327 GPT-4o 'no change' labels versus the 306 string-identical transitions should state whether the 8.8% proportion in Figure 5 uses GPT-4o labels or the corrected count; this affects the interpretation of the 'no change' category.
  4. [References] Reference [18] is incomplete: it gives a journal name and year but no article title or author. Please complete the citation.
  5. [Section 6.2] The open-coding step is described as reaching 'full agreement' after discussion, but no inter-annotator agreement statistic (e.g., Cohen's kappa) is reported for the two annotators before reconciliation. Please add it or explain why it is omitted.

Circularity Check

1 steps flagged · score 4.0 of 10

GPT-4o validation loop: the 'ground truth' for RQ2 was revised using GPT-4o's own labels, so the 0.55 overlap is partly self-agreement and cannot validate the headline debugging-versus-exploration percentages.

  1. self definitional [Section 6.2 (RQ2 methodology) and Section 7.2 (validation of GPT-4o labels)]
    "Finally, as a third step, we revised the labels from the open coding together with GPT-4o's labels, to define the ground truth labels for the sample of 400 transitions. ... When we revised the sample of 400 transitions to come up with a final 'ground truth' label, we established that GPT-4o showed a 0.55 overlap with the ground truth over these 400 transitions, in our 11-label task."

    The reference 'ground truth' for the 400-transition sample is not independent: the paper explicitly revised the human open-coding labels together with GPT-4o's labels before fixing the final ground-truth label. The reported 0.55 overlap therefore measures GPT-4o against a standard that already incorporates GPT-4o's outputs, so it is partly self-agreement rather than an unbiased accuracy estimate. This overlap is the only quantitative validation for the GPT-4o annotations that produce the headline RQ2 split (72.2% code iteration vs 14.1% exploration) and the paper's conclusion that notebooks are used more for debugging than exploration. Section 8.1's caveat that the authors 'cannot fully trust' the GPT-4o annotations does not break the loop.

full rationale

The derivation chain is mostly observational: RQ1's transition counts, edit distances, and output types are computed directly from the logged events; RQ3 labels cells with a CatBoost classifier trained on the externally published DASWOW dataset (F1=0.721 on the DASWOW test set) and the transition matrices are then literal counts, so no fitted parameter is renamed as a finding. The paper's reliance on the same-authors DASWOW taxonomy [13] is a self-citation dependency, but it is not a derivation loop: the taxonomy is a published artifact with its own test-set evaluation, and the paper justifies its choice against an alternative taxonomy. The one concrete circularity is in RQ2's validation: the 'ground truth' for the 400-transition sample was produced by revising the human open-coding labels together with GPT-4o's labels, so the reported 0.55 agreement is partly GPT-4o agreeing with itself. Because this overlap is the only quantitative support for the GPT-4o annotations that generate the headline 72.2% code-iteration vs 14.1% exploration split, the circularity is load-bearing for the paper's central debugging-versus-exploration narrative, even though the percentages themselves are not fitted to the ground truth. Section 8.1 candidly states that the authors 'cannot fully trust' the GPT-4o annotations, but this admission does not repair the validation loop. Score 4 reflects one genuine circular validation step while the main empirical observations retain independent content.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on two learned annotation pipelines rather than on fitted equations. The free parameters are the training choices for the data science step classifier, the manual sample used to create the purpose taxonomy, and the commercial GPT-4o annotation model. All are chosen by the authors and carry label noise into the reported percentages. No new physical or theoretical entities are introduced.

free parameters (3)
  • CatBoost classifier hyperparameters = iterations=400, learning_rate=0.2
    Chosen to optimize weighted F1 for the single-label data science step classifier used in RQ3; no sensitivity analysis is reported, so transition matrices depend on this choice.
  • Manual open-coding sample size = 400 self-transitions (200 per task)
    The 11-category change-purpose taxonomy is derived from this sample and used as ground truth to evaluate GPT-4o; a different sample could change the taxonomy and the reported proportions.
  • GPT-4o annotation model = GPT-4o snapshot as of 2025-03-10
    All 3,573 self-transition purpose labels are generated by this model; its agreement with the 400-sample human ground truth is only 0.55, so the model choice directly affects every RQ2 percentage.
assumptions (4)
  • domain assumption The DASWOW taxonomy of data science steps is a complete and appropriate labeling scheme for the new JuNE logs.
    RQ3 classifies every cell with the ten-step scheme from Ramasamy et al. [13]; the paper itself later argues the taxonomy 'does not accurately capture' the workflow, which would undermine the classification.
  • ad hoc to paper GPT-4o labels are reliable enough for aggregate proportions despite 0.55 overlap with human labels.
    Section 7.2 reports the overlap and Section 8.1 says the model 'cannot fully trust', yet the 72.2% code-iteration finding is computed from these labels without a correction or sensitivity analysis.
  • domain assumption Self-transitions are the relevant unit for analyzing code evolution; inter-transitions are excluded.
    Section 6 defines the two transition types and states a 'conservative approach' focusing only on self-transitions; this assumes cross-cell changes are not meaningfully interpretable for purpose analysis.
  • domain assumption The single-label CatBoost classifier trained on DASWOW generalizes to the hackathon notebooks in JuNE.
    The classifier achieves F1=0.72 on the DASWOW test set, which is a different corpus from the new execution logs; label noise propagates into the RQ3 transition matrices, acknowledged in Section 8.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Observing Fine-Grained Changes in Jupyter Notebooks During Development Time." pith.science (2026). https://pith.science/paper/MYDB6XXF

@misc{pith2026250715831,
  author       = {Pith},
  title        = {Pith review of: Observing Fine-Grained Changes in Jupyter Notebooks During Development Time},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MYDB6XXF}},
  note         = {Machine review of arXiv:2507.15831}
}
read the original abstract

In software engineering research, the analysis of fine-grained logs led to significant innovations in areas such as refactoring, security, and code completion. However, even though computational notebooks are a staple of data science and an important tool in machine learning, few similar studies have been conducted in this area. To help bridge this research gap, this paper makes three scientific contributions. (1) We introduce a toolset for collecting code changes in Jupyter notebooks during development time. (2) We use it to collect more than 100 hours of work related to a data analysis task and a machine learning task (carried out by 20 developers with different levels of expertise), resulting in a dataset containing 2,655 cells and 9,207 cell executions. (3) Finally, we use this dataset to investigate the dynamic nature of the notebook development process and the changes that take place in the notebooks. In our analysis of the collected data, we classified the changes made to the cells between executions and found that a significant number of these changes constituted code iteration modifications. We report a number of other insights and propose detailed future research directions on the novel data.

Figures

Figures reproduced from arXiv: 2507.15831 by the authors.

Figure 1
Figure 1. The pipeline of the developed tooling. main parts, visualized in [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. The overview of our empirical analysis. granular but logically finished steps. Then, we transformed the dataset to represent notebook development as a series of transitions from one cell to another. From the development log, we reconstructed the data as follows: for each execution, we recorded information about the state of the executed cell, including the time of the previous execution, the source code, and the sav… view at source ↗
Figure 3
Figure 3. Example of structure of the evolution graph. Red boxes highlight examples of [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: (a) Distribution of the number of consecutive re-executions. Note the exponen [PITH_FULL_IMAGE:figures/full_fig_p021_4.png]
Figure 5
Figure 5. Figure 5: Proportion of labels provided by GPT-4o (as of the 10th of March, 2025) when [PITH_FULL_IMAGE:figures/full_fig_p023_5.png]
Figure 6
Figure 6. Figure 6: The presence of different data science steps depending on the quantile of the [PITH_FULL_IMAGE:figures/full_fig_p025_6.png]
Figure 7
Figure 7. Figure 7: The transitional matrices for data science steps — only inter-transitions. Note [PITH_FULL_IMAGE:figures/full_fig_p026_7.png]
Figure 8
Figure 8. Figure 8: The evolution of the Mean number of objects metric in time for individual notebooks (grey lines) and their average curve (red line). effects that require deeper investigation. One can pursue studying what errors frequently occur in the notebooks, what objects are most …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 31 canonical work pages

  1. [1]

    Negara, M

    S. Negara, M. Vakilian, N. Chen, R. E. Johnson, D. Dig, Is it dangerous to use version control histories to study source code evolution?, in: Eu- ropean Conference on Object-Oriented Programming, Springer, 2012, pp. 79–103

    work page 2012

  2. [2]

    C. Ko, M. Ruschitzka, K. Levitt, Execution monitoring of security- critical programs in distributed systems: A specification-based ap- proach, in: Proceedings. 1997 IEEE symposium on security and privacy (Cat. No. 97CB36097), IEEE, 1997, pp. 175–187

    work page 1997

  3. [3]

    M. D. Syer, Z. M. Jiang, M. Nagappan, A. E. Hassan, M. Nasser, P. Flora, Leveraging performance counters and execution logs to di- agnose memory-related performance issues, in: 2013 IEEE international conference on software maintenance, IEEE, 2013, pp. 110–119

    work page 2013

  4. [4]

    Bibaev, A

    V. Bibaev, A. Kalina, V. Lomshakov, Y. Golubev, A. Bezzubov, N. Po- varov, T. Bryksin, All you need is logs: Improving code completion by learning from anonymous IDE usage logs, in: Proceedings of the 30th 32 ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022, pp. 1269–1279

    work page 2022

  5. [5]

    Lyulina, A

    E. Lyulina, A. Birillo, V. Kovalenko, T. Bryksin, TaskTracker-tool: A toolkit for tracking of code snapshots and activity data during solu- tion of programming tasks, in: Proceedings of the 52nd ACM Technical Symposium on Computer Science Education, 2021, pp. 495–501

    work page 2021

  6. [6]

    Kasatskii, A

    V. Kasatskii, A. Sergeyuk, A. Serova, S. Titov, T. Bryksin, The effect of perceptual load on performance within IDE in people with ADHD symptoms, in: International Conference on Human-Computer Interac- tion, Springer, 2023, pp. 122–141

    work page 2023

  7. [7]

    J. F. Pimentel, L. Murta, V. Braganholo, J. Freire, Understanding and improving the quality and reproducibility of Jupyter notebooks, Empir- ical Software Engineering 26 (4) (2021) 65

    work page 2021

  8. [8]

    Grotov, S

    K. Grotov, S. Titov, V. Sotnikov, Y. Golubev, T. Bryksin, A large- scale comparison of Python code in Jupyter notebooks and scripts, in: Proceedings of the 19th International Conference on Mining Software Repositories, 2022, pp. 353–364

    work page 2022

Show all 31 references
  1. [9]

    D. E. Knuth, Literate programming, The computer journal 27 (2) (1984) 97–111

    1984

  2. [10]

    Raghunandan, A

    D. Raghunandan, A. Roy, S. Shi, N. Elmqvist, L. Battle, Code code evo- lution: Understanding how people change data science notebooks over time, in: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 2023, pp. 1–12

    2023

  3. [11]

    W.-H. Chen, J. L. Cheoh, M. Keim, S. Brunswicker, T. Zhang, Towards understanding fine-grained programming mistakes and fixing patterns in data science, Proceedings of the ACM on Software Engineering 2 (FSE) (2025) 1824–1846

    2025

  4. [12]

    Titov, K

    S. Titov, K. Grotov, C. Sarasua, Y. Golubev, D. Ramasamy, A. Bac- chelli, A. Bernstein, T. Bryksin, Supplementary materials,https:// doi.org/10.5281/zenodo.16098734, [Online. Accessed 12-June-2026]. 33

    2026 doi

  5. [13]

    Ramasamy, C

    D. Ramasamy, C. Sarasua, A. Bacchelli, A. Bernstein, Workflow analysis of data science code in public GitHub repositories, Empirical Software Engineering 28 (1) (2023) 7

    2023

  6. [14]

    Negara, N

    S. Negara, N. Chen, M. Vakilian, R. E. Johnson, D. Dig, A compar- ative study of manual and automated refactorings, in: ECOOP 2013– Object-Oriented Programming: 27th European Conference, Montpel- lier, France, July 1-5, 2013. Proceedings 27, Springer, 2013, pp. 552–576

    2013

  7. [15]

    Y. S. Yoon, B. A. Myers, A longitudinal study of programmers’ back- tracking, in: 2014 IEEE Symposium on Visual Languages and Human- Centric Computing (VL/HCC), IEEE, 2014, pp. 101–108

    2014

  8. [16]

    Negara, M

    S. Negara, M. Codoban, D. Dig, R. E. Johnson, Mining fine-grained code changes to detect unknown change patterns, in: Proceedings of the 36th International Conference on Software Engineering, 2014, pp. 803–813

    2014

  9. [17]

    Hilton, N

    M. Hilton, N. Nelson, H. McDonald, S. McDonald, R. Metoyer, D. Dig, TDDViz: Using software changes to understand conformance to test driven development, in: Agile Processes, in Software Engineering, and Extreme Programming: 17th International Conference, XP 2016, Ed- inburgh, ...

    2016

  10. [18]

    van der Wolk, The (im)possibilities of scientific research under the GDPR, Cybersecurity Law Report (2019)

    A. van der Wolk, The (im)possibilities of scientific research under the GDPR, Cybersecurity Law Report (2019)

    2019

  11. [19]

    Titov, Y

    S. Titov, Y. Golubev, T. Bryksin, ReSplit: Improving the structure of Jupyternotebooksbyre-splittingtheircells, in: 2022IEEEinternational conference on software analysis, evolution and reengineering (SANER), IEEE, 2022, pp. 492–496

    2022

  12. [20]

    J. Wang, L. Li, A. Zeller, Better code, better sharing: On the need of analyzing Jupyter notebooks, in: Proceedings of the ACM/IEEE 42nd international conference on software engineering: new ideas and emerg- ing results, 2020, pp. 53–56

    2020

  13. [21]

    J. F. Pimentel, L. Murta, V. Braganholo, J. Freire, A large-scale study about quality and reproducibility of Jupyter notebooks, in: 2019 34 IEEE/ACM 16th international conference on mining software reposito- ries (MSR), IEEE, 2019, pp. 507–517

    2019

  14. [22]

    Källén, T

    M. Källén, T. Wrigstad, Jupyter notebooks on GitHub: characteristics and code clones, arXiv preprint arXiv:2007.10146 (2020)

    2020 arXiv

  15. [23]

    D. Wang, J. D. Weisz, M. Muller, P. Ram, W. Geyer, C. Dugan, Y. Tausczik, H. Samulowitz, A. Gray, Human-AI collaboration in data science: Exploring data scientists’ perceptions of automated ai, Pro- ceedings of the ACM on human-computer interaction 3 (CSCW) (2019) 1–24

    2019

  16. [24]

    O’Reilly Media, Inc

    M. Grinberg, Flask Web development, “O’Reilly Media, Inc.", 2018

    2018

  17. [25]

    Owens, G

    M. Owens, G. Allen, SQLite, Apress LP New York, 2010

    2010

  18. [26]

    C. S. Bojer, J. P. Meldgaard, Kaggle forecasting competitions: An over- looked learning opportunity, International Journal of Forecasting 37 (2) (2021) 587–603

    2021

  19. [27]

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, M. Zhou, CodeBERT: A pre-trained model for pro- gramming and natural languages, in: Findings of the Association for Computational Linguistics: EMNLP 2020, 2020, pp. 1536–1547

    2020

  20. [28]

    Prokhorenkova, G

    L. Prokhorenkova, G. Gusev, A. Vorobev, A. V. Dorogush, A. Gulin, CatBoost: Unbiased boosting with categorical features, Advances in neural information processing systems 31 (2018)

    2018

  21. [29]

    Patra, M

    J. Patra, M. Pradel, Nalin: Learning from runtime behavior to find name-value inconsistencies in Jupyter notebooks, in: Proceedings of the 44th International Conference on Software Engineering, 2022, pp. 1469– 1481

    2022

  22. [30]

    Grotov, A

    K. Grotov, A. Borzilov, M. Krivobok, T. Bryksin, Y. Zharov, Debug smarter, not harder: AI agents for error resolution in computational notebooks, in: Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing: System Demonstrations, 2024, pp. 363–371. 35

    2024

  23. [31]

    C. Yang, R. A. Brower-Sinning, G. Lewis, C. Kästner, Data leakage in notebooks: Static detection and better processes, in: Proceedings of the 37th IEEE/ACM international conference on automated software engineering, 2022, pp. 1–12. 36

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.