{"id":"ed25d75e-b30c-42df-bbcb-bd7c99d6a0b7","arxiv_id":"2504.12365","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new benchmark for Jupyter runtime-aware code prediction shows five frontier LLMs get under 19% exact match on output prediction and under 14% on next cell prediction, with runtime context not helping.","lead":"Themisto is a benchmark built from Jupyter notebook execution histories, with two tasks: predict the next code cell a developer will run, and predict a cell's output from earlier cells and runtime state. Baselines on five large language models show low accuracy and no benefit from injecting runtime variables, pointing to what the authors call an understudied gap for code models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Documented 'Runtime' prompts in Appendix A.2 contain no runtime variables, memory, or timing, so the central negative claim rests on a comparison that may be vacuous.","rationale":"The reader's weakest assumption concerned whether the text-serialized runtime snapshot is a fair way to grant models access to runtime context. My reading goes further: the prompt template in Appendix A.2, which is the only documented inference setup, does not include any runtime snapshot at all. The central claim therefore is not merely overgeneralized; the key comparison between 'Runtime' and 'No Runtime' conditions is not supported by the published evidence. The benchmark artifact and no-runtime baselines may still have value, but the paper's headline conclusion that current LLMs cannot leverage runtime context cannot be accepted without seeing the actual runtime-condition prompt and verifying that it contains the features from Table 1. This warrants rejection of the paper in its current form, with a path to revision through prompt-code release and an ablation demonstration that the runtime context is genuinely present and salient.","tokens_in":6410,"tokens_out":4130,"duration_ms":45511,"concrete_test":"Download the released Zenodo artifact and inspect the exact prompt-building code used for the 'Runtime' condition. Verify whether any of Table 1's runtime features (runtime variables, memory bytes, execution time) are inserted into the model context. Then run a small ablation, e.g., GPT-4o-mini on 50 sampled examples, in three conditions: (a) no runtime, (b) the printed A.2 template, and (c) the printed A.2 template plus the 'Runtime Variables:' block from A.4. If (a) and (b) are identical and (c) differs materially, the paper's conclusion is invalid as written.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that adding runtime information does not help LLMs. For that claim to hold, the 'Runtime' condition must actually supply the runtime features described in Table 1. The inference prompts printed in Appendix A.2 do not do this: both Output Prediction and Next Cell Prediction templates list only 'Code: ...' and 'Output: ...' pairs. There is no 'Runtime Variables', 'Execution Time', or 'Memory Usage' field anywhere in the prompt. Thus, as documented, the 'Runtime' and 'No Runtime' settings are the same prompt, which would trivially produce the nearly identical scores in Table 2. The discussion in Section 2.3 ('performance cannot be improved by simply adding all available information') presumes the information was added, but the appendix shows otherwise. This is not a subtle serialization choice such as repr truncation; it is the complete absence of the runtime state. If the real code constructs a different prompt than the one printed, the paper must publish that construction; if it uses the printed prompt, the negative result is vacuous.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Themisto, a benchmark built from re-executed Jupyter notebook trajectories in the JuNE dataset. It defines two tasks: cell output prediction and next cell prediction, with trajectory features including code, output, execution time, memory bytes, runtime variables, and a hash index. The authors evaluate five frontier LLMs under 'No Runtime' and 'Runtime' prompt conditions, report generally low exact-match scores (e.g., 0.16-0.19 for output prediction, 0.06-0.14 for next cell prediction), and find that adding runtime information does not improve scores. They conclude that current LLMs cannot effectively leverage runtime context and call for more research on runtime-aware code models. The paper self-identifies low variability in Section 4 as a main threat to validity.","tokens_in":6623,"tokens_out":5336,"duration_ms":53898,"significance":"If the negative result were fully supported, Themisto would be a useful community asset for a genuinely understudied problem: using live kernel state in notebook assistants. The data is released on Zenodo, the two tasks are concrete and plausible proxies for interactive notebook help, and the metric choices (exact match, ROUGE-L, ChrF) are standard for this area. However, the claimed absence of benefit from runtime context is not currently established, because the documented inference prompts do not contain runtime information, and the small, low-variability sample limits generalizability. The resource itself remains potentially valuable for future runtime-aware models, but the central empirical claim needs to be demonstrated with a correctly documented and statistically grounded comparison.","major_comments":[{"comment":"The two prompt templates printed in Appendix A.2 are identical for the 'Runtime' and 'No Runtime' conditions: both list only 'Code: ...' and 'Output: ...' pairs, with no 'Runtime Variables', 'Execution Time', or 'Memory Usage' fields, even though Table 1 and Appendix A.4 describe these features as part of the trajectory. As documented, the comparison in Table 2 is therefore vacuous for the claim that 'models are not yet able to effectively leverage the runtime context'. The paper must either print the actual prompt construction used for the Runtime condition, including the full serialization of runtime variables, or state clearly that only the code/output pairs were used; without this, the central negative result is unsupported.","section":"Appendix A.2 / Table 2"},{"comment":"The tables report point estimates without confidence intervals, error bars, or significance tests. The observed Runtime-vs-NoRuntime differences are mostly 0.01-0.02 in exact match, and on 200 examples a difference of one or two correct predictions can shift the score materially; the 'drop by half' for Claude-3.5 (0.18 to 0.09) corresponds to a small absolute count. The conclusion that runtime context provides no benefit needs per-example variance estimates (e.g., bootstrap CIs) or a paired significance test before it can be treated as a definitive negative result.","section":"Section 2.3 / Table 2"},{"comment":"The benchmark is derived from only four re-executed notebooks and a 200-example subsample, with exception-containing outputs explicitly removed because 'foundational models struggle with stack traces'. The paper acknowledges low variability in Section 4, but the abstract and Section 2.3 draw a broad conclusion about 'the current generation of LLMs' as a whole. The conclusion should be explicitly restricted to the sampled trajectory distribution, or the evaluation needs to be extended to more notebooks and a larger sample before claims about the general inability to use runtime context can be made.","section":"Section 2.2 / Section 4"}],"minor_comments":[{"comment":"The row labels 'GPT-4oRuntime' and 'GPT-4o-miniRuntime' appear to be missing a space; the intended labels are likely 'GPT-4o Runtime' and 'GPT-4o-mini Runtime'.","section":"Table 3"},{"comment":"The phrase 'these results are compensated by higher scores on ROUGE-L and ChrF' is awkward; the intended meaning is likely that the modest exact-match scores are accompanied by higher lexical-similarity scores.","section":"Section 2.3"},{"comment":"The paper states that four notebooks from the JuNE dataset were re-executed, but it does not explain how those four were selected out of the 29 notebooks available; a brief selection criterion would improve reproducibility.","section":"Section 2.2"},{"comment":"The 'Next Cell Prediction' template ends with 'Predict the next code cell that would logically follow:' without the closing code block marker or a period in the extracted text; please ensure the printed template exactly matches the one used in the experiments.","section":"Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":"The prompt-template issue in Appendix A.2 is critical. If the printed templates reflect the actual experiments, the Runtime condition does not contain any runtime state, and the paper's main negative claim is not supported by the data. I would ask the authors to publish the exact prompts and the code that constructs them, and if the runtime fields were truly omitted, to rerun the evaluation before accepting the central claim. The benchmark resource itself and the idea of measuring runtime-aware notebook assistance remain worthwhile, so a major revision is appropriate rather than rejection. I also note that the benchmark is derived from the authors' own JuNE dataset; this is not inherently disqualifying, but the paper should state the lineage explicitly and avoid presenting the benchmark as fully independent of that dataset."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short note: the new thing here is a benchmark of Jupyter execution trajectories with two tasks, next-cell prediction and output prediction, plus baseline numbers for five models. The data artifact is real, and the authors are transparent about scale and diversity limits.\n\nThe problem is the 'Runtime' condition. Table 2 compares models with and without runtime information, and the paper says the runtime scores show models cannot leverage runtime context. But Appendix A.2, which is the only documented inference setup, gives a prompt that contains only code and output pairs. No runtime variables, no memory, no execution time. The template for the runtime setting is identical to the no-runtime setting. If that is actually what was run, then the negative result is vacuous — you compared a thing to itself. If the real experiments used a different prompt with runtime fields, the paper must publish that construction. Either way, as submitted, the central claim is unsupported.\n\nThe rest of the paper is ordinary. The tasks are close adaptations of CruxEval and REval, the sample is 200 examples from the authors' own JuNE dataset, and there are no error bars or significance tests, which matters because the reported differences between models are small. The exception exclusion and the single naive prompt format are additional limitations the authors partly acknowledge.\n\nOn the positive side, the trajectory data with per-step runtime state is a legitimate new artifact for the Jupyter/notebook-assistant community. The paper is clearly written and the baseline comparisons are useful as a starting point. But the key finding, that adding runtime context doesn't help, cannot be accepted on this evidence.\n\nRecommendation: send to peer review only if the authors can supply the actual runtime prompt and re-run the comparison, or make the current version conditional with a strong request for the missing details. As it stands, it should not be accepted.","headline":"The benchmark artifact is new, but the central negative result as documented compares identical prompts — the runtime condition never shows the runtime features.","tokens_in":7122,"tokens_out":2836,"would_cite":false,"duration_ms":27088,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current large language models perform poorly on Jupyter runtime-aware tasks, and providing runtime context does not help them.","keywords":["Jupyter notebooks","runtime context","code generation","output prediction","benchmark","large language models","next cell prediction","execution state"],"falsifier":"Run the same benchmark with a fine-tuned model trained to attend to runtime variables, or feed the runtime state as structured JSON rather than prose, and check whether exact match with runtime exceeds the without-runtime baseline by a significant margin. A human expert shown the same prompts predicting outputs or next cells with high accuracy would also confirm that the information is usable and the models, not the task, are the limiting factor.","tokens_in":6225,"feed_emoji":"📓","tokens_out":7708,"duration_ms":63602,"temperature":0.7,"pith_summary":"The paper introduces Themisto, a benchmark built from real Jupyter notebook development trajectories, to test whether large language models can use runtime state to predict the next code cell a developer will run and the output of a given cell. Across five frontier models, exact-match scores are low—at most 18% for output prediction and 14% for next-cell prediction—and adding a serialized snapshot of runtime variables, memory, and execution time to the prompt does not improve performance; for one model it substantially hurts it. The authors conclude that current models are not yet able to effectively leverage runtime context, and argue that runtime-aware code generation is an understudied problem that deserves more attention.","feed_headline":"Adding runtime state doesn't help LLMs predict notebook code","feed_subtitle":"Five frontier models score below 18% exact match on 200 Jupyter trajectories, with or without runtime snapshots.","key_machinery":"The central object is the development trajectory: a sequence of (code, output, runtime snapshot) triples captured from real Jupyter sessions, where each snapshot records the executed code, its output, execution time, memory usage, and a serialized dictionary of runtime variables with names, sizes, and repr values. The two tasks—cell output prediction and next-cell code prediction—are scored with exact match, ROUGE-L, and ChrF. The load-bearing mechanism is the inference prompt: the trajectory is serialized into the prompt, and the only difference between the two experimental conditions is whether the runtime-variable dictionary is appended, so any score difference is attributable to the presence of runtime information.","core_discovery":"The central claim is that, as of early 2025, frontier LLMs perform poorly on Jupyter runtime-aware tasks and are not yet able to leverage runtime context. The evidence is a controlled comparison on 200 examples: each model is evaluated twice, once with the full trajectory of prior cell executions and once with an additional text-serialized snapshot of the runtime environment (variable names, sizes, and repr values, plus memory and execution time). Scores are statistically similar between the two conditions, and the best exact-match rates are 18% for output prediction and 14% for next-cell prediction. The paper interprets this as evidence that the information is present but the models cannot use it, and that new post-training or architectural approaches are needed.","pith_inferences":["The serialization format is plain-text prose; a structured representation (e.g., JSON with only relevant variables) might be easier for LLMs to exploit, so the negative result may be an artifact of the format rather than a fundamental limitation.","The benchmark's small size (200 examples from four notebooks and two original tasks) and exclusion of exception-throwing cells means the negative result may not generalize to broader notebook code.","Next-cell prediction asks for the exact next user action, which is inherently unpredictable; a softer evaluation, such as selecting the next cell from a candidate set or classifying the next operation, might still reveal usable runtime signal.","If runtime state proves useful after fine-tuning, that would suggest training objectives beyond next-token prediction, such as prediction of variable values or execution traces, are needed to unlock the information."],"forward_implications":["Runtime-aware code completion in Jupyter notebooks is currently unsolved: appending runtime state to the prompt does not improve next-cell or output prediction.","The benchmark offers a way to measure progress, letting future models be checked for whether they can exploit runtime context.","The low scores on both tasks, despite strong static code-generation performance, suggest that modeling program state is a distinct capability current LLMs lack.","The drop in Claude-3.5's output-prediction exact match when runtime info was added indicates that injecting extra context can actively mislead a model, not merely fail to help.","Because the only difference between conditions is the runtime snapshot, any future method that improves the runtime condition can be attributed to better use of runtime information."],"supporting_citations":[{"why":"Supplies the JuNE dataset of real Jupyter development trajectories, the raw material for the benchmark.","marker":"Titov et al., 2025"},{"why":"Documents the GPT-4o and GPT-4o-mini baselines evaluated in the study.","marker":"Hurst et al., 2024"},{"why":"Documents the Claude-3.5 baseline whose output-prediction score dropped when runtime info was added.","marker":"Anthropic, 2024"},{"why":"Documents the Gemini 1.5 Pro baseline model tested in the benchmark.","marker":"Team et al., 2024"},{"why":"Documents the DeepSeek-V3 baseline, which achieved the best next-cell exact match.","marker":"Liu et al., 2024"},{"why":"Provides a prior benchmark for code output prediction; the paper uses it as a comparison point for static snapshot tasks.","marker":"Gu et al., 2024"},{"why":"Introduces runtime data and state-prediction tasks; the paper references it as evidence that runtime reasoning is hard for LLMs.","marker":"Chen et al., 2024"},{"why":"Motivates the choice of exact match, ROUGE-L, and ChrF as evaluation metrics.","marker":"Evtikhiev et al., 2023"}],"fun_headline_variants":["Runtime info doesn't help LLMs in notebooks","LLMs can't use runtime context for code prediction","Only 18% accuracy when LLMs see runtime state","Jupyter runtime data? LLMs still guess blind","Understudied: LLMs ignoring runtime in notebooks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The negative result depends on the assumption that the text-serialized runtime snapshot placed into the prompt is a fair way to give models access to runtime context; if a different serialization, ordering, or fine-tuned model could exploit this information, the conclusion that runtime context is of no benefit would collapse.","fun_headline_variants_meta":{"raw":{"variants":["Runtime info doesn't help LLMs in notebooks","LLMs can't use runtime context for code prediction","Only 18% accuracy when LLMs see runtime state","Jupyter runtime data? LLMs still guess blind","Understudied: LLMs ignoring runtime in notebooks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000333,"raw_usage":{"total_tokens":1747,"prompt_tokens":737,"completion_tokens":1010,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":353,"completion_tokens_details":{"reasoning_tokens":934}},"tokens_in":353,"tokens_out":1010,"duration_ms":9797,"temperature":1.0,"reasoning_tokens":934,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:34:33.127424+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same benchmark with a fine-tuned model trained to attend to runtime variables, or feed the runtime state as structured JSON rather than prose, and check whether exact match with runtime exceeds the without-runtime baseline by a significant margin. A human expert shown the same prompts predicting outputs or next cells with high accuracy would also confirm that the information is usable and the models, not the task, are the limiting factor.","supporting_citations":[],"review_version":1}