{"id":"df987393-9b3a-400f-b45f-e887291948f3","arxiv_id":"2501.09745","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new dataset of 48,398 real Jupyter notebook editing events from GitHub shows that LLMs predict code edits poorly, with improved but still limited performance after fine-tuning.","lead":"This paper mines 48,398 real Jupyter notebook edits from 792 machine learning repositories on GitHub and tests whether LLMs can predict those edits. It shows that current models score low even after fine-tuning, and that giving the models more surrounding code helps the most.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated diff extraction may count notebook output/metadata changes as code edits; the dataset's core claim and the LLM evaluation both hinge on this.","rationale":"The reader identified exactly this unvalidated extraction assumption as the weakest point, and I agree. The paper deserves credit for releasing the dataset and scripts, but release does not establish that what was extracted corresponds to developer code edits. The proposed sample audit is cheap, decisive, and directly tests the central contribution. If the audit shows low contamination, the conditional concern is resolved. If it shows high contamination, the dataset and all evaluation conclusions are unreliable. Thus the reader's CONDITIONAL verdict should stand unchanged rather than being upgraded or rejected without data.","tokens_in":8143,"tokens_out":4686,"duration_ms":48347,"concrete_test":"Randomly sample 200 entries from the released 48,398 file-level modifications. For each, re-fetch the parent/child commits, parse the notebook JSON with nbformat, extract only code cell sources, and produce a diff with nbdime or an equivalent notebook-aware aligner. Independently label each entry as genuine code-source change versus output/metadata/ordering artifact, using two annotators and reporting inter-annotator agreement. If more than 10% of sampled entries are artifacts, or if the notebook-aware diff disagrees with the paper's SequenceMatcher output on more than 5% of changed cells, regenerate the dataset with a notebook-aware differ and rerun the evaluation before the central claim is accepted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the 48,398 entries are genuine Jupyter notebook code edits made by developers, and that LLM performance on them is low. Section II steps 3-5 extract diffs by running git show and then applying difflib.SequenceMatcher to 'the content of the code cells' before and after each commit. The paper does not state that notebook JSON fields other than code cell source (execution counts, outputs, metadata, cell ordering) are stripped before comparison, nor does it report manual validation on a sample. If such fields are included, or if SequenceMatcher misaligns cells after insertion/deletion/reordering, entries can represent output churn or serialization artifacts rather than developer code edits. The claimed average of 166 edited lines per revision and the reported BLEU/CodeBLEU/EditSim/RougeL scores are then not measurements of code-edit prediction but of a mixed noisy signal. This is a correctness risk in dataset construction, not a disagreement with consensus, because every downstream finding inherits it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a dataset of 48,398 Jupyter notebook edits mined from 20,095 commits across 792 GitHub repositories tagged with machine-learning and Jupyter-notebook topics. The dataset records cell-level and line-level differences, commit messages, and repository metadata. The authors use the dataset to evaluate DeepSeek-Coder models (1.3B and 6.7B parameters) on predicting the edited code at both cell and file granularity, under few-shot prompting, post-processing, and supervised fine-tuning. The main findings are that edits are localized (claimed average of 166 lines per revision), that providing more context improves file-level predictions, and that all tested models perform poorly, with fine-tuning yielding the largest gains.","tokens_in":8328,"tokens_out":3896,"duration_ms":41090,"significance":"If the dataset genuinely captures developer code edits in Jupyter notebooks, it fills a clear gap: existing notebook datasets focus on code generation or comprehension, not on editing. The paper also provides a reproducible collection pipeline and an open dataset, which is valuable for the SE4AI community. However, the central significance claim rests on the correctness of the diff-extraction process. No manual validation is reported, so the possibility of output-churn or serialization artifacts being counted as code edits is a real threat to every downstream statistic and evaluation. The evaluation itself is standard but lacks error bars or significance testing, and at least one reported comparison is internally inconsistent. With additional validation and corrections, the dataset could be a useful resource; in its current form, the core claim is not yet established.","major_comments":[{"comment":"The dataset construction extracts diffs by running git show and then applying difflib.SequenceMatcher to 'the content of the code cells' before and after a commit. The paper never states whether non-code JSON fields of the notebook—such as outputs, execution counts, cell metadata, or cell ordering—are stripped before comparison. If they are not stripped, or if SequenceMatcher misaligns cells after insertion, deletion, or reordering, then some of the 48,398 entries may represent output churn or serialization artifacts rather than developer code edits. Because every downstream statistic, including the 166-line average and all BLEU/CodeBLEU/EditSim/RougeL scores, is computed over these entries, this is a load-bearing validity risk. Please specify exactly what git show returns and what is fed to SequenceMatcher, and report a manual validation study on a random sample of edits (e.g., 100-200 entries) with inter-annotator agreement.","section":"Section II, steps 3-5"},{"comment":"The abstract and introduction state that 'changes average only 166 lines of code in repositories' and that 'the majority of developer edits target specific portions of notebooks, with an average of 166 lines modified per revision.' Table I reports token counts, not line counts, and Table II reports only total LOC for the whole dataset. There is no definition of what constitutes a changed line in this statistic, and no table or computation that yields the number 166. This claim is presented as a key finding (Finding 2) and must be either backed by a matching table entry and a precise definition or removed.","section":"Abstract and Section III"},{"comment":"The text claims that the 6.7B model 'consistently outperforms' the 1.3B model on file-level edits, but Table IV shows a CodeBLEU of 15.39 for the 1.3B model and 15.34 for the 6.7B model, i.e., the smaller model is slightly higher. Similar small-margin comparisons appear elsewhere (e.g., file-level 1.3B BLEU 10.75 vs 6.7B BLEU 11.76). Without significance tests or error bars, such claims are fragile. Please either correct the 'consistently outperforms' wording or provide statistical support for the claimed ordering.","section":"Section IV-E and Table IV"},{"comment":"The fine-tuning subsection states that 'we fine-tune three DeepSeek-Coder instructed models, each with 1.3 billion parameters' and that results are compared against 'the average results of the three fine-tuned models,' yet Tables III and IV report a single '1.3B-finetune' row. It is also unclear how 'approximate 13K revisions' relates to the 12,874 commits in the training set reported in Table II. Please clarify the experimental protocol: how many models were trained, what hyperparameters were used, how the averages were computed, and what the variance across runs was.","section":"Section IV-D"}],"minor_comments":[{"comment":"There are several typos: 'the using LLMs' should be 'using LLMs', and 'four-folds' should be 'fourfold'.","section":"Abstract"},{"comment":"'Juypter' is misspelled consistently; it should be 'Jupyter'. The text also uses 'Github' and 'GitHub' inconsistently.","section":"Section II and Figure 1"},{"comment":"The phrase 'only the modification type rather than additions or deletions' is unclear; it seems to mean that only lines whose type is 'replace' are further diffed line-by-line, but this should be stated explicitly.","section":"Section II, step 5"},{"comment":"The filter that excludes commit messages with two or fewer words and removes duplicate entries is not accompanied by any sensitivity analysis. Please report how many entries would remain under different thresholds or whether this threshold was chosen based on qualitative examination.","section":"Section II, step 6"},{"comment":"The metric RougeL is sometimes written as 'ROUGEL' or 'RougeL' in the text and tables; please standardize the notation.","section":"Section IV-C"},{"comment":"The statement that low BLEU scores 'both below 19' apply only to the few-shot results, but the fine-tuned file-level BLEU is 27.46; please rephrase to avoid confusion.","section":"Section IV-E"},{"comment":"Reference [1] lacks author names, and several references are missing publisher or venue details (e.g., [4], [16]). Please ensure all entries are complete.","section":"References"},{"comment":"The paper contains no limitations section. Given that the dataset construction relies on heuristics, an explicit discussion of potential diff-alignment errors, the star-based repository selection, and the commit-message word filter would improve the paper's transparency.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The main blocking issue is the lack of validation of the diff-extraction pipeline. If the authors add a manual validation sample and correct the internal inconsistencies around the 166-line statistic and the fine-tuning description, the paper could become a solid dataset contribution for the SE4AI community. I would also encourage the editor to check that the Zenodo link and replication scripts are indeed publicly accessible at the time of final decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing is the dataset: 48,398 Jupyter notebook edits mined from 20,095 commits across 792 ML repos, with cell-level and line-level granularity. As far as I know this is the first public benchmark of real developer edits in Jupyter notebooks, and it fills a real gap—previous notebook datasets are about generation or comprehension, not maintenance. The LLM evaluation is straightforward few-shot and fine-tuning on DeepSeek-Coder, and the finding that all models score low even after fine-tuning is plausible and honest. That alone makes the paper a useful baseline for SE4AI work.\n\nThe paper documents the construction pipeline decently, but there's a soft spot that matters. Section II steps 3-5 use git show and SequenceMatcher to compare the content of code cells before and after each commit. It's never explicitly stated that notebook fields like outputs, execution counts, and metadata are stripped before comparison, and there's no manual validation sample. If output cells are being counted as code edits, the dataset is measuring noise, and every downstream metric inherits that. This is a correctness risk in the central artifact, not a stylistic quibble. The fix is easy—state the stripping clearly and hand-validate a couple hundred edits—but until that's done I'd be cautious about the dataset's fidelity claims.\n\nTwo smaller issues: repository selection uses the top 1000 by stars, which biases toward popular, well-maintained projects; that's fine as a limitation but deserves a sentence in the paper. And there are no error bars or significance tests on the model scores; for a first benchmark that's acceptable, but it would help to know whether the 6.7B vs 1.3B differences are real.\n\nThe stress-test note about SequenceMatcher misalignment on cell reordering is a legitimate sub-concern, but I think it's minor compared to the outputs/metadata issue, and it should be caught by the same validation pass.\n\nOverall, this is a solid, useful contribution with one load-bearing question mark. The math and evaluation are straightforward; the citation pattern looks fine. I'd send it to peer review with a request for validation of the dataset extraction. If the authors show that the edits are genuinely code changes, I'd be happy to use the dataset and cite the paper.","headline":"First Jupyter edit dataset, but the diff extraction needs validation before the numbers carry weight.","tokens_in":8812,"tokens_out":2789,"would_cite":true,"duration_ms":28948,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces a new dataset of 48,398 Jupyter notebook edits mined from 20,095 revisions of 792 machine learning repositories, then shows that large language models predict these edits poorly even after fine-tuning.","keywords":["Jupyter notebooks","code edit prediction","large language models","machine learning maintenance","benchmark dataset","GitHub repositories","fine-tuning","code metrics"],"falsifier":"Manually inspect a random sample of, say, 200 dataset instances and count how many cell-diff pairs are genuine code edits rather than notebook serialization changes such as output updates, metadata changes, or reordered cells; if a large fraction are artifacts, the conclusion that LLMs are poor at real maintenance edits would not follow from this benchmark.","tokens_in":7964,"feed_emoji":"📓","tokens_out":7225,"duration_ms":68542,"temperature":0.7,"pith_summary":"The paper introduces the first large-scale dataset of code edits made by developers to Jupyter notebooks in machine learning repositories, drawn from 20,095 revisions across 792 projects. Using this dataset, it tests whether large language models can predict the edits a developer will make next, given the commit message and the code before the edit. Across cell-level and file-level tasks, all tested models, from 1.3B to 6.7B parameters, with few-shot prompting and with fine-tuning, score low on standard similarity metrics, although fine-tuning roughly doubles or triples BLEU scores. The authors read the low accuracy as evidence that real-world machine-learning notebook maintenance is harder than the generation-style tasks these models are usually benchmarked on.","feed_headline":"48,398 real notebook edits expose LLM code-edit limits","feed_subtitle":"Benchmark from ML repositories shows fine-tuned models still score far below practical accuracy.","key_machinery":"The central object is the dataset itself: each instance is a commit that modifies a Jupyter notebook, with the repository, commit hash and message, the code cells before and after, and the cell- and line-level diffs. It is assembled by selecting top-starred repositories tagged with Jupyter and machine-learning topics, cloning them, running git log and git show, and aligning pre- and post-commit cell contents with Python's difflib SequenceMatcher. The evaluation machinery consists of five-shot prompting and fine-tuning (LoRA, PiSSA, FlashAttention-v2) of DeepSeek-Coder models, scored by BLEU, CodeBLEU, EditSim, and ROUGE-L.","core_discovery":"The central claim is that a dataset of 48,398 notebook edits, mined from 20,095 commits in 792 machine learning repositories, provides the first benchmark for Jupyter-specific code-edit prediction, and that the evaluated large language models fail to solve it. Edits are highly localized, averaging about 166 changed lines per revision even though repositories average 8,380 lines, and commit messages are short with a mean of 5.66 words. On the test split, the best few-shot model reaches BLEU 13.30 for cell-level edits and 11.76 for file-level edits; after fine-tuning a 1.3B model reaches BLEU 25.86 and 27.46, still far from practical utility. The paper's conclusion is that contextual information and task-specific fine-tuning improve predictions materially, but real-world ML maintenance remains an open problem.","pith_inferences":["A testable extension is to validate the ground truth by having human annotators label a random sample of aligned cell diffs as genuine code edits versus notebook serialization artifacts, which would quantify how much noise output fields, metadata, or cell reordering introduce.","Because EditSim scores are markedly higher than BLEU in the reported results, character-level similarity may overstate success on short cells; future benchmarks could add execution-based validation that runs predicted cells and checks produced outputs.","Retrieval-augmented selection of only the edited cell's surrounding cells, together with the commit message, is a natural baseline the reported context-dependence results predict should outperform full-file prompting."],"forward_implications":["Large models outperform smaller ones on notebook edit prediction, but none approach usable accuracy, so this dataset can serve as a realistic stress test for code-editing LLMs.","Fine-tuning with task-specific data produces the largest gains, multiples of BLEU and ROUGE-L over few-shot baselines, implying that domain-adapted training matters more than model scale in this task.","Edits concentrate in small regions despite large notebooks, so models that exploit surrounding context should focus on local cells rather than whole-file rewriting.","The dataset's cell- and line-level granularity and commit metadata enable future work on suggestions, bug fixes, and feature additions in machine learning notebooks.","Because all models still produce syntax errors after fine-tuning, the task exposes a gap in current LLM code-editing abilities."],"supporting_citations":[{"why":"Supplies the repository selection mechanism, top-starred Jupyter/ML repos, that determines the dataset's scope.","marker":"[8]"},{"why":"Provides the DeepSeek-Coder 1.3B and 6.7B instructed models whose few-shot and fine-tuned predictions are evaluated.","marker":"[9]"},{"why":"Defines the Jupyter notebook format and cell structure that the dataset records.","marker":"[11]"},{"why":"Low-rank adaptation method used in the fine-tuning experiments.","marker":"[10]"},{"why":"Gives the longest-common-subsequence F1 metric used for prediction-reference similarity.","marker":"[13]"},{"why":"Defines the primary n-gram overlap metric used to measure prediction quality.","marker":"[19]"},{"why":"Extends the evaluation with keyword, syntax-tree, and data-flow matching for code.","marker":"[21]"},{"why":"Supplies the character-level Levenshtein similarity metric used to score edits.","marker":"[24]"}],"fun_headline_variants":["LLMs flunk first Jupyter edit benchmark","First notebook-edit dataset shows LLMs fall short","48K Jupyter edits stump code LLMs","Fine-tuned models still weak at notebook code edits","ML notebook edits: LLMs need more context"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that difflib's alignment of notebook cell contents across consecutive commits faithfully captures the developer's intended edit; if outputs, metadata, or reordered cells are counted as code changes, the ground truth is contaminated.","fun_headline_variants_meta":{"raw":{"variants":["LLMs flunk first Jupyter edit benchmark","First notebook-edit dataset shows LLMs fall short","48K Jupyter edits stump code LLMs","Fine-tuned models still weak at notebook code edits","ML notebook edits: LLMs need more context"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000462,"raw_usage":{"total_tokens":2314,"prompt_tokens":952,"completion_tokens":1362,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":1289}},"tokens_in":568,"tokens_out":1362,"duration_ms":9685,"temperature":1.0,"reasoning_tokens":1289,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:40:39.291144+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually inspect a random sample of, say, 200 dataset instances and count how many cell-diff pairs are genuine code edits rather than notebook serialization changes such as output updates, metadata changes, or reordered cells; if a large fraction are artifacts, the conclusion that LLMs are poor at real maintenance edits would not follow from this benchmark.","supporting_citations":[{"cited_title":"Rest api endpoints for repositories,","cited_arxiv_id":null,"evidence_quote":"Supplies the repository selection mechanism, top-starred Jupyter/ML repos, that determines the dataset's scope."},{"cited_title":"Deepseek-coder: When the large language model meets programming – the rise of code intelligence,","cited_arxiv_id":null,"evidence_quote":"Provides the DeepSeek-Coder 1.3B and 6.7B instructed models whose few-shot and fine-tuned predictions are evaluated."},{"cited_title":"Jupyter notebooks - a publishing format for reproducible computational workflows,","cited_arxiv_id":null,"evidence_quote":"Defines the Jupyter notebook format and cell structure that the dataset records."},{"cited_title":"Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics,","cited_arxiv_id":null,"evidence_quote":"Gives the longest-common-subsequence F1 metric used for prediction-reference similarity."},{"cited_title":"Bleu: a method for automatic evaluation of machine translation,","cited_arxiv_id":null,"evidence_quote":"Defines the primary n-gram overlap metric used to measure prediction quality."},{"cited_title":"Intellicode compose: code generation using transformer,","cited_arxiv_id":null,"evidence_quote":"Supplies the character-level Levenshtein similarity metric used to score edits."}],"review_version":1}