{"id":"f3e3ec44-f493-453a-99e1-0a2c50516ca8","arxiv_id":"2502.04184","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Only about 21% of popular public Jupyter notebooks are truly broken, while most apparent failures stem from missing packages, files, or execution order and are at least partially executable.","lead":"This study ran 42,546 popular public Jupyter notebooks through automated error checks and repairs and found that only about 21 percent are truly broken. Most apparent failures come from missing packages, missing input files, or execution-order issues, and many notebooks still run partway, which changes how notebook reusability should be measured.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 21.3% 'pathologically non-executable' figure is definitional: notebooks with ModuleNotFound/FileNotFound/NameError are labeled restorable regardless of whether restoration actually works, and the paper's own success rates imply restorable could be far lower.","rationale":"The reader identified the same load-bearing weakness: the restorable classification is based on a definitional taxonomy rather than demonstrated restoration success. My stress test confirms this is the central soft spot. The paper's own data contradict the 78.7% restorable figure: full restoration is 5.4% and partial restoration is 12.8%, suggesting the actual restorable fraction, under an outcome-based definition, is far below 78.7%, potentially reversing or substantially weakening the headline. There are also concrete numeric inconsistencies (e.g., Table I percentages sum beyond 100%, abstract says 42.7% while the body says 40.5%, and the introduction claims 4,943 full restorations while the conclusion says 5.4%, which would be about 1,872). These do not destroy the paper's genuine contributions: partial executability is a useful metric, the LLM restoration results are informative despite modest success rates, and the focus on popular notebooks is a reasonable design choice. The central claim, however, needs empirical reclassification to be supported. A CONDITIONAL verdict is appropriate because the paper can be fixed by redefining restorability based on demonstrated outcomes and correcting the arithmetic inconsistencies; the direction of the paper is defensible, but the headline as written is not yet established.","tokens_in":17738,"tokens_out":4798,"duration_ms":48394,"concrete_test":"Recompute RQ2 using the paper's own restoration logs: for every notebook in the 27,272 restorable bucket, classify it as restorable only if at least one restoration attempt (module install, synthetic input generation, or NameError definition) increased the number of successfully executed cells or achieved full execution; classify all others as pathological/unrestored. Then recalculate the 21.3% figure. Also verify Table I by recording exactly one first-error category per notebook and confirming the counts sum to 34,659; the current percentages sum to 107.3%, indicating either multiple-error counting or an arithmetic inconsistency that must be reconciled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central headline, that only 21.3% of non-executable notebooks are truly pathological, rests on the taxonomy in Section IV-A2: any notebook whose first error is FileNotFound, ModuleNotFound, or NameError is classified as restorable, and RQ2 then counts all such notebooks as non-pathological (27,272 of 34,659). But Section V-D reports the actual restoration outcomes, and they do not match this label. Of 23,476 ModuleNotFound notebooks, only 3,760 (16.0%) had their modules successfully installed, while 5,970 had deprecated modules and 13,746 failed for other reasons. Of 4,546 FileNotFound notebooks, synthetic data restored full or partial executability in only 1,378 (30.3%). Of 404 NameError notebooks, 183 (45.3%) improved. The paper's own summary says restoration fully fixed 5.4% and partially fixed 12.8% of all non-executable notebooks, i.e., about 18.2% - not 78.7%. Moreover, since Papermill records only the first error, a notebook whose first error is FileNotFound may still contain a later AttributeError or ValueError that is genuinely pathological, so the first-error taxonomy further overcounts restorable notebooks. The 'restorable' label is thus an assumption, not a measurement, and the 21.3% headline changes substantially if restorability is defined by demonstrated restoration success rather than by error type alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that prior estimates of notebook non-executability (e.g., 76%) are too pessimistic because they use a binary, strict notion of executability. It proposes a distinction between restorable notebooks (those failing only due to missing modules, input files, or undefined names) and pathologically non-executable notebooks (those with intrinsically unresolvable errors). The authors build a dataset of 42,546 popular public GitHub notebooks, execute them with Papermill, and report that 81.5% are non-executable as-is. Using a taxonomy based on the first error, they classify 21.3% of non-executable notebooks as pathological and 78.7% as potentially restorable. They then apply LLM-based restoration strategies for ModuleNotFound, FileNotFound, and NameError, reporting full restoration of 5.4% of non-executable notebooks and partial restoration of 12.8%, with improvements in cell-level executability for the partially restored cases. The paper also introduces a partial-executability metric and shows that even pathologically non-executable notebooks run, on average, 34.1% of their cells.","tokens_in":18003,"tokens_out":5331,"duration_ms":50516,"significance":"If the central claim held, the paper would substantially revise the accepted view that most public notebooks are non-executable, and it would provide a useful framework for separating environmental misconfiguration from intrinsic code breakage. The dataset construction with GitHub-star stratification, the public artifact, and the partial-executability metric are genuine contributions, as is the demonstration that LLM-based restoration can improve execution in a non-trivial fraction of cases. However, the headline number (21.3% pathological) is not an empirical measurement but a definitional consequence of the error taxonomy, and the paper's own restoration results show that the true rate of demonstrated restoration is much lower. The paper's framing therefore overstates the strength of the evidence, although the underlying measurement framework and partial results remain valuable if appropriately qualified.","major_comments":[{"comment":"The headline claim that only 21.3% of non-executable notebooks are pathologically non-executable is an arithmetic consequence of the taxonomy, not of the restoration experiments. In Section IV-A2, all notebooks whose first error is ModuleNotFound, FileNotFound, or NameError are categorized as restorable, and in Section V-B these 27,272 notebooks are counted as '78.7% restorable'. But Section V-D reports that only 3,760 of 23,476 ModuleNotFound notebooks (16.0%) had their modules successfully installed, and only 1,378 of 4,546 FileNotFound notebooks (30.3%) were fully or partially restored. Thus the 'restorable' label is an assumption about environmental misconfiguration, not a measured outcome. The paper should either redefine restorability in terms of demonstrated restoration success or explicitly present 78.7% as an upper bound on potentially restorable notebooks, with the restoration success rates reported separately. Without this change, the title's answer to the question 'Are the Majority ... Pathologically Non-Executable?' is not supported by the paper's own data.","section":"Section IV-A2 and Section V-B"},{"comment":"The classification uses only the first error that halts execution, which overestimates restorability. A notebook whose first error is FileNotFound may, after the input file is generated, fail again with an AttributeError, ValueError, or other error that the taxonomy would classify as pathological. The restoration results in Section V-D demonstrate this: among the 23,476 ModuleNotFound notebooks, 13,746 failed for other reasons after installation attempts, and only a subset of those with valid modules were restored. The paper should report how many notebooks were attempted for each error type and how many of those failed again at later cells, and it should distinguish notebooks that were ultimately fully restored from those that only progressed to a different error. Using first-error categories as a proxy for restorability is only valid if the subsequent-error distribution is reported.","section":"Section IV-A2 (Papermill first-error logging)"},{"comment":"The paper reports inconsistent restoration success numbers. The abstract states that LLM-based methods improve executability by 42.7% for module installation and 28% for synthetic data, but Section V-D1 and the RQ4 takeaway report 40.5% for module installation. The introduction states that the fix 'fully restores full executability of 4,943 of the notebooks' for which the module and synthetic data were found, while the abstract and conclusion state that 5.4% of previously non-executable notebooks (which would be about 1,871 of 34,659) were fully restored. These numbers cannot all be correct. The authors should reconcile the count of fully restored notebooks, the percentage figures, and the improvement percentages, and ensure that the abstract, introduction, and results sections are consistent.","section":"Section V-D vs. Abstract/Introduction"}],"minor_comments":[{"comment":"The takeaway says 'only 21.3% of all notebooks are pathologically non-executable', but 7,387 / 42,546 = 17.4%. The 21.3% figure refers to the percentage of non-executable notebooks, not all notebooks; this should be phrased precisely to avoid confusion.","section":"Section V-B, Takeaway RQ2"},{"comment":"The NameError restoration success rate is reported differently: Section V-D3 says 45.3% of 404 notebooks improved and 8.7% fully restored, while the RQ4 takeaway lists a 9.4% improvement figure for NameError. The relationship between '45.3% improved' and '9.4%' is not explained; please clarify.","section":"Section V-D3 and Abstract"},{"comment":"The description of the 13% sampling step is vague. After collecting approximately 318,000 notebooks, the authors 'take a 13% sample' but do not specify the sampling procedure (uniform random, stratified by repository, or something else). This matters for the generalizability of the 21.3% estimate and should be described precisely.","section":"Section IV-A (sampling)"},{"comment":"The sentence 'the executability of 1,378 (37%) notebooks are fully or partially restored' should specify that 37% is the success rate among the 3,729 notebooks for which input files were generated, not among all 4,546 FileNotFound notebooks. This is stated in the surrounding text but the phrase is ambiguous.","section":"Section V-D2"},{"comment":"The threat that 'an earlier fatal runtime error (that we cannot resolve) may hinder our ability to capture FileNotFound or ModuleNotFound errors' is acknowledged, but the same concern applies in the opposite direction for the restorable category: a first FileNotFound may mask later pathological errors. This should be stated explicitly in the threats section.","section":"Section VI, Threats to validity"},{"comment":"The introduction claims a 95% executability improvement in the case study, and Figure 10 summarizes before/after executable notebook counts, but the figure is not described in the text beyond a one-line caption. Please add a sentence in Section V-D4 explaining what Figure 10 shows and how the before/after counts are derived.","section":"Introduction and Figure 10"}],"recommendation":"major_revision","confidential_remarks":"The paper is built on a solid empirical framework and a valuable dataset, and the partial-executability metric is a genuinely useful addition. However, the central claim that 'only 21.3% of non-executable notebooks are truly pathological' is not supported by the restoration experiments; it is definitional. The actual demonstrated full-restoration rate of 5.4% and partial-restoration rate of 12.8% suggest that the true pathological fraction is considerably higher than 21.3% if restorability is measured by successful restoration. I would encourage the editor to require a major revision in which the authors either rework the taxonomy to be outcome-based or substantially weaken the headline claim and present the 78.7% figure as a ceiling under optimistic assumptions. The inconsistency in reported success percentages (42.7 vs. 40.5, 4,943 vs. 5.4%) also needs to be resolved before the paper can be trusted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the paper has a useful idea, but its headline number—only 21.3% of non-executable notebooks are pathological—is an assumption wearing a measurement's clothes. The actual restoration data tell a different story, and the paper's own numbers don't always line up.\n\nWhat's genuinely new: the notion of partial executability (fraction of cells that run before the first error) is a sensible way to talk about reuse value of broken notebooks, and measuring it on a popularity-filtered sample of 42k GitHub notebooks is a legitimate improvement over prior all-GitHub samples. The LLM-based restoration pipeline—installing modules, generating synthetic input files, synthesizing definitions—is a reasonable, reproducible effort, and they've shipped code and data. The finding that even 'pathological' notebooks average 34.1% executable cells is worth knowing.\n\nThe soft spot is the RQ2 taxonomy. In Section IV, they classify notebooks with ModuleNotFound, FileNotFound, or NameError as 'restorable' and everything else as 'pathologically non-executable.' That's a choice, not a measurement. RQ2 then reports 78.7% restorable because 27,272 notebooks have one of those three error types. But RQ4 shows what actually happens when you try: module installation fully works for 3,760 of 23,476 ModuleNotFound notebooks (16%), synthetic data restores 1,378 of 4,546 FileNotFound notebooks (30%), and NameError fixes work for 183 of 404 (45%). So the real full-restoration rate is 5.4% (or 4,943 notebooks, depending on which sentence you read), not 78.7%. The 'restorable' label is optimistic.\n\nThere are also numeric inconsistencies that need a cleanup pass: the abstract says 42.7% improvement for module install while the body says 40.5%; the conclusion says 9.4% for NameError while the section says 45.3% of notebooks improved; and the error counts in Table I sum to more than 34,659, suggesting overlap or missing categories. The paper also doesn't report variance or confidence intervals, which matters when you're comparing across datasets. Their own threats-to-validity note that earlier fatal errors can mask later ones, which actually strengthens the concern that first-error classification overcounts restorable notebooks.\n\nThat said, the core argument—that binary executability understates reuse potential—is defensible and likely correct. The paper deserves a serious referee, but it needs major revision: re-base the restorability classification on demonstrated restoration attempts, reconcile the numbers, and be more careful about what 'restorable' means. I'd send it to review with a request for heavy revision, and I'd bring it to a reading group as a good example of a definitional claim colliding with empirical results.","headline":"The paper's real contribution is partial executability; its headline '21.3% pathological' number is an assumption, not a measurement, and the actual restoration data are much weaker.","tokens_in":18542,"tokens_out":4143,"would_cite":true,"duration_ms":36156,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Most public notebooks that fail to run are misconfigured, not broken, and can be partially restored.","keywords":["computational notebooks","non-executability","notebook restoration","LLM-based repair","partial executability","misconfiguration","public notebook reuse","Jupyter notebooks"],"falsifier":"Take a random sample of notebooks the paper labels pathologically non-executable (first error something like AttributeError, TypeError, or ValueError) and try to restore them after supplying the repository's complete dependency files and any available local data or execution order. If a substantial share then run to completion, the claim that only 21.3% of non-executable notebooks are pathological is too low. Conversely, if a large share of ModuleNotFound notebooks remain broken even after their modules install successfully, the 'restorable' category is too optimistic.","tokens_in":17481,"feed_emoji":"📓","tokens_out":7636,"duration_ms":68874,"temperature":0.7,"pith_summary":"Prior studies classify a notebook as executable only if it runs end-to-end without error, and have reported that roughly three-quarters of public notebooks fail that test. This paper argues that the binary notion is too rigid: a notebook that stops because an input file is missing or a module is uninstalled is not broken, only misconfigured, and can often be restored. Analyzing 42,546 popular public notebooks, the authors find 81.5% non-executable under the strict definition, but only 21.3% of non-executable notebooks are pathologically non-executable; the rest are executable or potentially restorable. They show that lightweight LLM-driven fixes—installing correct modules, generating synthetic input data, defining undefined names—fully restore 5.4% of previously non-executable notebooks and meaningfully improve partial executability of many others. The claim matters because it reframes notebook reuse: most failures are environmental, and even partially executable notebooks carry value.","feed_headline":"Only 21.3% of 'broken' notebooks are truly unrepairable","feed_subtitle":"Most execution failures in 42,546 popular notebooks are missing files or modules, not bad code—and cheap fixes help.","key_machinery":"The machinery is a two-dimensional view of executability. First, a taxonomy: notebooks are executable, restorable (first error is a missing module, missing input file, or undefined name), or pathologically non-executable (all other unresolvable errors). Second, a degree-of-executability metric defined as the ratio of cells successfully executed before the first error to the total number of cells, turning a binary label into a continuum. The measurement pipeline executes each notebook with an automated notebook-runner in a per-repository virtual environment, records the first error, then iteratively applies targeted LLM-driven repairs—installing inferred module names, generating synthetic input data, or inserting definitions for undefined names—and re-executes to measure improvement. This combination lets the authors separate 'not runnable here' from 'not runnable anywhere' and quantify partial value.","core_discovery":"The paper's central claim is that most public computational notebooks are not intrinsically broken. Under a strict 'as-is' execution, 81.5% of 42,546 popular notebooks fail; but of the 34,659 non-executable notebooks, only 21.3% (7,387) suffer errors the authors classify as pathological—syntax or indentation failures, attribute/type/value errors, and similar problems that require knowing the notebook's intended semantics to repair. The remaining non-executable notebooks fail because of missing modules, missing input files, or undefined names, which the authors treat as environmental or session-state issues rather than code defects. In support, installing the correct modules improves executability by 40.5% on average for notebooks where installation succeeds, LLM-generated synthetic input files fully or partially restore 37% of notebooks with FileNotFound errors, and LLM-generated definitions fix 45.3% of NameError notebooks. The conclusion is that executability should be measured on a continuum—cells executed before the first error divided by total cells—and evaluated within the interactive notebook paradigm, not by traditional atomic software standards.","pith_inferences":["The paper classifies every non-Module/File/Name first error as pathologically non-executable without attempting restoration for most of those error types; if AttributeError and TypeError notebooks were actually fixable, the pathological share could fall below 21.3%.","The 'restorable' label is assigned by error type rather than by demonstrated success; actual repair rates (21.5% for module installation, 37% for synthetic files) suggest that a fix-based definition would put the truly restorable share well below 78.7%.","The partial-executability metric could serve as a practical triage signal: notebooks with high partial execution are the cheapest repair targets, so repair effort should be prioritized there.","A natural testable extension is to run the same error-driven restoration with a different LLM or with multi-shot feedback; the paper's single-shot open-source LLM numbers may be a lower bound."],"forward_implications":["Notebook non-executability is substantially lower than the commonly cited 76% figure when measured with a finer taxonomy: most failures are missing environments, not bad code.","Partial executability—the fraction of cells that run before the first error—should be reported alongside binary executability, since even pathological notebooks average 34.1% executable cells.","Lightweight LLM-driven restoration can recover full or partial execution for a meaningful share of notebooks: 5.4% fully restored, with average improvements of 40.5% from module installation and 28% from synthetic input generation.","Studies of notebook quality should stratify by popularity; NameError and other low-level errors concentrate in low-star notebooks, biasing earlier large-scale datasets.","Executability is a prerequisite for reproducibility, so restoration and partial execution open the door to dynamic analysis tools on notebooks previously dismissed as unusable."],"supporting_citations":[{"why":"Supplies the 76% non-executability baseline and the binary executability definition this paper argues against.","marker":"[3]"},{"why":"Extended empirical study of notebook quality and reproducibility that also treats executability as a binary end-to-end property.","marker":"[5]"},{"why":"Prior tool for restoring notebook execution environments by installing required packages, a method the paper adapts and measures.","marker":"[6]"},{"why":"Prior approach restoring deprecated APIs in notebooks; context for the module and dependency restoration strategy.","marker":"[7]"},{"why":"Restores reproducibility through cell reordering and dependency analysis; motivates the paper's def-use analysis and partial-execution view.","marker":"[9]"},{"why":"Automated notebook execution engine used to run notebooks and log the first encountered error.","marker":"[17]"},{"why":"Open-source large language model that powers the error-driven restoration prompts.","marker":"[20]"},{"why":"Justifies using repository star counts as the popularity criterion for selecting the notebook dataset.","marker":"[11]"}],"fun_headline_variants":["Most broken notebooks aren't broken—just missing modules or files","Only 21% of failed notebooks are truly unrepairable, study finds","Simple fixes—installing modules, generating data—revive most notebooks","Notebook breakage: 4 in 5 failures are environmental, not code","The '76% non-executable' stat is misleading: most are restorable"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a notebook whose first error is a missing module, missing file, or undefined name is automatically 'restorable', while every other first error is 'pathologically non-executable'; the 21.3% headline follows from that classification, not from demonstrated restoration of most notebooks.","fun_headline_variants_meta":{"raw":{"variants":["Most broken notebooks aren't broken—just missing modules or files","Only 21% of failed notebooks are truly unrepairable, study finds","Simple fixes—installing modules, generating data—revive most notebooks","Notebook breakage: 4 in 5 failures are environmental, not code","The '76% non-executable' stat is misleading: most are restorable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00066,"raw_usage":{"total_tokens":3085,"prompt_tokens":1076,"completion_tokens":2009,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":692,"completion_tokens_details":{"reasoning_tokens":1909}},"tokens_in":692,"tokens_out":2009,"duration_ms":13763,"temperature":1.0,"reasoning_tokens":1909,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T23:12:27.854009+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of notebooks the paper labels pathologically non-executable (first error something like AttributeError, TypeError, or ValueError) and try to restore them after supplying the repository's complete dependency files and any available local data or execution order. If a substantial share then run to completion, the claim that only 21.3% of non-executable notebooks are pathological is too low. Conversely, if a large share of ModuleNotFound notebooks remain broken even after their modules install successfully, the 'restorable' category is too optimistic.","supporting_citations":[{"cited_title":"A large-scale study about quality and reproducibility of jupyter notebooks,","cited_arxiv_id":null,"evidence_quote":"Supplies the 76% non-executability baseline and the binary executability definition this paper argues against."},{"cited_title":"Understanding and improving the quality and reproducibility of jupyter notebooks,","cited_arxiv_id":null,"evidence_quote":"Extended empirical study of notebook quality and reproducibility that also treats executability as a binary end-to-end property."},{"cited_title":"Restoring execution environments of jupyter notebooks,","cited_arxiv_id":null,"evidence_quote":"Prior tool for restoring notebook execution environments by installing required packages, a method the paper adapts and measures."},{"cited_title":"Restoring the executability of jupyter notebooks by automatic upgrade of deprecated apis,","cited_arxiv_id":null,"evidence_quote":"Prior approach restoring deprecated APIs in notebooks; context for the module and dependency restoration strategy."},{"cited_title":"Assessing and restoring repro- ducibility of jupyter notebooks,","cited_arxiv_id":null,"evidence_quote":"Restores reproducibility through cell reordering and dependency analysis; motivates the paper's def-use analysis and partial-execution view."},{"cited_title":"Papermill: Parameterize, execute, and analyze note- books,","cited_arxiv_id":null,"evidence_quote":"Automated notebook execution engine used to run notebooks and log the first encountered error."},{"cited_title":"Introducing meta llama 3: The most capable openly available llm to date","cited_arxiv_id":null,"evidence_quote":"Open-source large language model that powers the error-driven restoration prompts."},{"cited_title":"Understanding the factors that impact the popularity of github repositories,","cited_arxiv_id":null,"evidence_quote":"Justifies using repository star counts as the popularity criterion for selecting the notebook dataset."}],"review_version":1}