Themisto: Jupyter-Based Runtime Benchmark

Konstantin Grotov; Sergey Titov

REVIEW 3 major objections 4 minor 13 references

Themisto: Jupyter-Based Runtime Benchmark

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Current large language models perform poorly on Jupyter runtime-aware tasks, and providing runtime context does not help them.

desk verdict The benchmark artifact is new, but the central negative result as documented compares identical prompts — the runtime condition never shows the runtime features. read the letter →

arxiv 2504.12365 v1 pith:XT2RTVAO submitted 2025-04-16 cs.SE cs.AIcs.LG

Konstantin Grotov , Sergey Titov This is my paper · ORCID

classification cs.SEcs.AIcs.LG

keywords Jupyternotebooksruntimecontextcodegenerationoutputpredictionbenchmarklargelanguagemodelsnextcellexecutionstate

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Themisto, a benchmark built from real Jupyter notebook development trajectories, to test whether large language models can use runtime state to predict the next code cell a developer will run and the output of a given cell. Across five frontier models, exact-match scores are low—at most 18% for output prediction and 14% for next-cell prediction—and adding a serialized snapshot of runtime variables, memory, and execution time to the prompt does not improve performance; for one model it substantially hurts it. The authors conclude that current models are not yet able to effectively leverage runtime context, and argue that runtime-aware code generation is an understudied problem that deserves more attention.

What carries the argument

The central object is the development trajectory: a sequence of (code, output, runtime snapshot) triples captured from real Jupyter sessions, where each snapshot records the executed code, its output, execution time, memory usage, and a serialized dictionary of runtime variables with names, sizes, and repr values. The two tasks—cell output prediction and next-cell code prediction—are scored with exact match, ROUGE-L, and ChrF. The load-bearing mechanism is the inference prompt: the trajectory is serialized into the prompt, and the only difference between the two experimental conditions is whether the runtime-variable dictionary is appended, so any score difference is attributable to the presence of runtime information.

What would settle it

Run the same benchmark with a fine-tuned model trained to attend to runtime variables, or feed the runtime state as structured JSON rather than prose, and check whether exact match with runtime exceeds the without-runtime baseline by a significant margin. A human expert shown the same prompts predicting outputs or next cells with high accuracy would also confirm that the information is usable and the models, not the task, are the limiting factor.

Watch

Extended reading notes

Core claim

The central claim is that, as of early 2025, frontier LLMs perform poorly on Jupyter runtime-aware tasks and are not yet able to leverage runtime context. The evidence is a controlled comparison on 200 examples: each model is evaluated twice, once with the full trajectory of prior cell executions and once with an additional text-serialized snapshot of the runtime environment (variable names, sizes, and repr values, plus memory and execution time). Scores are statistically similar between the two conditions, and the best exact-match rates are 18% for output prediction and 14% for next-cell prediction. The paper interprets this as evidence that the information is present but the models cannot use it, and that new post-training or architectural approaches are needed.

Load-bearing premise

The negative result depends on the assumption that the text-serialized runtime snapshot placed into the prompt is a fair way to give models access to runtime context; if a different serialization, ordering, or fine-tuned model could exploit this information, the conclusion that runtime context is of no benefit would collapse.

Editorial extensions

If this is right

Runtime-aware code completion in Jupyter notebooks is currently unsolved: appending runtime state to the prompt does not improve next-cell or output prediction.
The benchmark offers a way to measure progress, letting future models be checked for whether they can exploit runtime context.
The low scores on both tasks, despite strong static code-generation performance, suggest that modeling program state is a distinct capability current LLMs lack.
The drop in Claude-3.5's output-prediction exact match when runtime info was added indicates that injecting extra context can actively mislead a model, not merely fail to help.
Because the only difference between conditions is the runtime snapshot, any future method that improves the runtime condition can be attributed to better use of runtime information.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The serialization format is plain-text prose; a structured representation (e.g., JSON with only relevant variables) might be easier for LLMs to exploit, so the negative result may be an artifact of the format rather than a fundamental limitation.
The benchmark's small size (200 examples from four notebooks and two original tasks) and exclusion of exception-throwing cells means the negative result may not generalize to broader notebook code.
Next-cell prediction asks for the exact next user action, which is inherently unpredictable; a softer evaluation, such as selecting the next cell from a candidate set or classifying the next operation, might still reveal usable runtime signal.
If runtime state proves useful after fine-tuning, that would suggest training objectives beyond next-token prediction, such as prediction of variable values or execution traces, are needed to unlock the information.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short note: the new thing here is a benchmark of Jupyter execution trajectories with two tasks, next-cell prediction and output prediction, plus baseline numbers for five models. The data artifact is real, and the authors are transparent about scale and diversity limits.

The problem is the 'Runtime' condition. Table 2 compares models with and without runtime information, and the paper says the runtime scores show models cannot leverage runtime context. But Appendix A.2, which is the only documented inference setup, gives a prompt that contains only code and output pairs. No runtime variables, no memory, no execution time. The template for the runtime setting is identical to the no-runtime setting. If that is actually what was run, then the negative result is vacuous — you compared a thing to itself. If the real experiments used a different prompt with runtime fields, the paper must publish that construction. Either way, as submitted, the central claim is unsupported.

The rest of the paper is ordinary. The tasks are close adaptations of CruxEval and REval, the sample is 200 examples from the authors' own JuNE dataset, and there are no error bars or significance tests, which matters because the reported differences between models are small. The exception exclusion and the single naive prompt format are additional limitations the authors partly acknowledge.

On the positive side, the trajectory data with per-step runtime state is a legitimate new artifact for the Jupyter/notebook-assistant community. The paper is clearly written and the baseline comparisons are useful as a starting point. But the key finding, that adding runtime context doesn't help, cannot be accepted on this evidence.

Recommendation: send to peer review only if the authors can supply the actual runtime prompt and re-run the comparison, or make the current version conditional with a strong request for the missing details. As it stands, it should not be accepted.

Referee Report

3 major / 4 minor

Summary. The paper introduces Themisto, a benchmark built from re-executed Jupyter notebook trajectories in the JuNE dataset. It defines two tasks: cell output prediction and next cell prediction, with trajectory features including code, output, execution time, memory bytes, runtime variables, and a hash index. The authors evaluate five frontier LLMs under 'No Runtime' and 'Runtime' prompt conditions, report generally low exact-match scores (e.g., 0.16-0.19 for output prediction, 0.06-0.14 for next cell prediction), and find that adding runtime information does not improve scores. They conclude that current LLMs cannot effectively leverage runtime context and call for more research on runtime-aware code models. The paper self-identifies low variability in Section 4 as a main threat to validity.

Significance. If the negative result were fully supported, Themisto would be a useful community asset for a genuinely understudied problem: using live kernel state in notebook assistants. The data is released on Zenodo, the two tasks are concrete and plausible proxies for interactive notebook help, and the metric choices (exact match, ROUGE-L, ChrF) are standard for this area. However, the claimed absence of benefit from runtime context is not currently established, because the documented inference prompts do not contain runtime information, and the small, low-variability sample limits generalizability. The resource itself remains potentially valuable for future runtime-aware models, but the central empirical claim needs to be demonstrated with a correctly documented and statistically grounded comparison.

major comments (3)

[Appendix A.2 / Table 2] The two prompt templates printed in Appendix A.2 are identical for the 'Runtime' and 'No Runtime' conditions: both list only 'Code: ...' and 'Output: ...' pairs, with no 'Runtime Variables', 'Execution Time', or 'Memory Usage' fields, even though Table 1 and Appendix A.4 describe these features as part of the trajectory. As documented, the comparison in Table 2 is therefore vacuous for the claim that 'models are not yet able to effectively leverage the runtime context'. The paper must either print the actual prompt construction used for the Runtime condition, including the full serialization of runtime variables, or state clearly that only the code/output pairs were used; without this, the central negative result is unsupported.
[Section 2.3 / Table 2] The tables report point estimates without confidence intervals, error bars, or significance tests. The observed Runtime-vs-NoRuntime differences are mostly 0.01-0.02 in exact match, and on 200 examples a difference of one or two correct predictions can shift the score materially; the 'drop by half' for Claude-3.5 (0.18 to 0.09) corresponds to a small absolute count. The conclusion that runtime context provides no benefit needs per-example variance estimates (e.g., bootstrap CIs) or a paired significance test before it can be treated as a definitive negative result.
[Section 2.2 / Section 4] The benchmark is derived from only four re-executed notebooks and a 200-example subsample, with exception-containing outputs explicitly removed because 'foundational models struggle with stack traces'. The paper acknowledges low variability in Section 4, but the abstract and Section 2.3 draw a broad conclusion about 'the current generation of LLMs' as a whole. The conclusion should be explicitly restricted to the sampled trajectory distribution, or the evaluation needs to be extended to more notebooks and a larger sample before claims about the general inability to use runtime context can be made.

minor comments (4)

[Table 3] The row labels 'GPT-4oRuntime' and 'GPT-4o-miniRuntime' appear to be missing a space; the intended labels are likely 'GPT-4o Runtime' and 'GPT-4o-mini Runtime'.
[Section 2.3] The phrase 'these results are compensated by higher scores on ROUGE-L and ChrF' is awkward; the intended meaning is likely that the modest exact-match scores are accompanied by higher lexical-similarity scores.
[Section 2.2] The paper states that four notebooks from the JuNE dataset were re-executed, but it does not explain how those four were selected out of the 29 notebooks available; a brief selection criterion would improve reproducibility.
[Appendix A.2] The 'Next Cell Prediction' template ends with 'Predict the next code cell that would logically follow:' without the closing code block marker or a period in the extracted text; please ensure the printed template exactly matches the one used in the experiments.

Circularity Check

1 steps flagged · score 7.0 of 10

The paper's own Appendix A.2 prompt templates contain no runtime variables, timing, or memory data, so the central claim that runtime context does not help LLMs reduces to a vacuous comparison by construction.

self definitional [Appendix A.2 (Inference Setup), Section 2.3 (Baselines), Table 2]
"Previous code cells and their outputs: Code: {{code}} Output: {{output}} ... Code: {{code}} Output: {{output}} Predict the output for this code: {{code_to_predict}} [...] This indicates that the models are not yet able to effectively leverage the runtime context for this task."

The only inference templates printed in A.2 list just 'Code:' and 'Output:' pairs; none of the runtime features promised in Table 1 (execution time, memory bytes, runtime variables, hash index) appears, and no separate 'Runtime' template is provided. As documented, the 'Runtime' and 'No Runtime' conditions are therefore the same prompt. The near-identical scores in Table 2 and the conclusion that models 'are not yet able to effectively leverage the runtime context' are then entailed by the prompt definition itself: runtime information was never actually inserted, so observing no benefit from it is a tautology rather than an empirical finding.

full rationale

The benchmark's data source is the authors' own JuNE dataset (Titov et al. 2025, with overlapping authorship), so there is a self-citation dependence. That alone is not circular: the absolute scores (e.g., 0.10-0.19 exact match on output prediction and 0.06-0.14 on next cell prediction) are an independent empirical measurement against five external LLMs, and the paper's own Section 4 limitation about low variability is an honest external-validity caveat, not a circularity. The load-bearing circularity is in the runtime/no-runtime contrast. The paper claims to show that adding runtime information does not help, but its documented inference prompt contains no runtime information; thus the central negative claim is forced by construction. Because the absolute poor performance and the benchmark itself still have independent content, the paper is partially, not wholly, circular. If the authors' actual implementation used a richer prompt than the one printed in A.2, the runtime comparison would be independently testable; on the manuscript as written, however, the comparison is degenerate.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

All choices are benchmark design decisions; none are fitted to optimize scores, but they affect the difficulty and composition of the benchmark and therefore shape the reported low accuracies. No new physical or mathematical entities are introduced.

free parameters (3)

Length filtering threshold (0.99 quantile) = 0.99 quantile
Cells and outputs longer than the 0.99 quantile are removed; this choice affects which examples enter the benchmark and may bias toward shorter, easier instances.
Benchmark sample size and allocation = 200 (180 Q2/Q3, 10 Q1, 10 Q4)
The 200-example set with stratified allocation across output-length quartiles is a hand-selected design decision that shapes the difficulty and diversity of the benchmark.
Exception exclusion = 0 exceptions allowed
All samples whose predicted output contains an exception were excluded, which removes a substantial class of runtime behaviors and improves achievable exact match.

assumptions (3)

domain assumption JuNE trajectories are representative of real Jupyter notebook development
The benchmark is sampled from four re-executed notebooks from the JuNE dataset, which the paper itself says covers only two tasks and 20 participants (Threat to Validity).
domain assumption Serialized runtime state text is a valid representation of runtime context for LLMs
Runtime variables, memory bytes, execution time, and code/output history are concatenated into prompts; the negative result is interpreted as model limitation, not representation failure (Section 2.3).
domain assumption Exact match, ROUGE-L, and ChrF are appropriate metrics for these open-ended generation tasks
Metrics are inherited from prior code generation evaluation (Evtikhiev et al., 2023) and applied without task-specific calibration.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Themisto: Jupyter-Based Runtime Benchmark." pith.science (2026). https://pith.science/paper/XT2RTVAO

@misc{pith2026250412365,
  author       = {Pith},
  title        = {Pith review of: Themisto: Jupyter-Based Runtime Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XT2RTVAO}},
  note         = {Machine review of arXiv:2504.12365}
}

read the original abstract

In this work, we present a benchmark that consists of Jupyter notebooks development trajectories and allows measuring how large language models (LLMs) can leverage runtime information for predicting code output and code generation. We demonstrate that the current generation of LLMs performs poorly on these tasks and argue that there exists a significantly understudied domain in the development of code-based models, which involves incorporating the runtime context.

Figures

Figures reproduced from arXiv: 2504.12365 by the authors.

**Figure 2.** Diversity metrics comparison between output and code. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 2 canonical work pages

[2]

Evaluating large language models trained on code

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374,

arXiv
[3]

Stepcoder: Improve code generation with reinforcement learning from compiler feedback

Shihan Dou, Yan Liu, Haoxiang Jia, Limao Xiong, Enyu Zhou, Wei Shen, Junjie Shan, Caishuang Huang, Xiao Wang, Xiaoran Fan, et al. Stepcoder: Improve code generation with reinforcement learning from compiler feedback. arXiv preprint arXiv:2402.01391,

arXiv
[5]

Cruxeval: A benchmark for code reasoning, understanding and execution

Alex Gu, Baptiste Rozi `ere, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065,

arXiv
[6]

Gpt-4o system card

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276,

arXiv
[7]

A survey on large language models for code generation

Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. arXiv preprint arXiv:2406.00515,

arXiv
[8]

Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770,

Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770,

arXiv
[10]

Code execution with pre-trained language models

Chenxiao Liu, Shuai Lu, Weizhu Chen, Daxin Jiang, Alexey Svyatkovskiy, Shengyu Fu, Neel Sun- daresan, and Nan Duan. Code execution with pre-trained language models. arXiv preprint arXiv:2305.05383, 2023a. Jiate Liu, Yiqin Zhu, Kaiwen Xiao, Qiang Fu, Xiao Han, Wei Yang, and Deheng Ye. Rltf: Re- inforcement learning from unit test feedback. Trans. Mach. Lea...

arXiv 2023
[2004]

Deepseek-v3 technical report

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437,

arXiv

Show all 13 references

[2015]

Kgtorrent: A dataset of python jupyter note- books from kaggle

5 Published as a conference paper at ICLR 2025 Luigi Quaranta, Fabio Calefato, and Filippo Lanubile. Kgtorrent: A dataset of python jupyter note- books from kaggle. In2021 IEEE/ACM 18th International Conference on Mining Software Repos- itories (MSR), pp. 550–554. IEEE,

2025
[2021]

Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context

Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.05530,

arXiv
[2023]

Rlef: Grounding code llms in execution feedback with reinforcement learning

Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Taco Cohen, and Gabriel Synnaeve. Rlef: Grounding code llms in execution feedback with reinforcement learning. arXiv preprint arXiv:2410.02089,

arXiv
[2024]

Reasoning runtime behavior of a program with llm: How far are we? arXiv preprint cs.SE/2403.16437,

Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. Reasoning runtime behavior of a program with llm: How far are we? arXiv preprint cs.SE/2403.16437,

arXiv
[2025]

Dataset containing logs of code evolution in Jupyter notebooks, comprising over 100 hours of execution logs from 20 participants solving data science tasks. A A PPENDIX A.1 D IVERSITY OF THE SAMPLES IN BENCHMARK Levenshtein N-gram Jaccard 0.0 0.2 0.4 0.6 0.8 1.0 Score Output: ...

2025

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [2]

Evaluating large language models trained on code

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374,

arXiv

[2] [3]

Stepcoder: Improve code generation with reinforcement learning from compiler feedback

Shihan Dou, Yan Liu, Haoxiang Jia, Limao Xiong, Enyu Zhou, Wei Shen, Junjie Shan, Caishuang Huang, Xiao Wang, Xiaoran Fan, et al. Stepcoder: Improve code generation with reinforcement learning from compiler feedback. arXiv preprint arXiv:2402.01391,

arXiv

[3] [5]

Cruxeval: A benchmark for code reasoning, understanding and execution

Alex Gu, Baptiste Rozi `ere, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065,

arXiv

[4] [6]

Gpt-4o system card

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276,

arXiv

[5] [7]

A survey on large language models for code generation

Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. arXiv preprint arXiv:2406.00515,

arXiv

[6] [8]

Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770,

Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770,

arXiv

[7] [10]

Code execution with pre-trained language models

Chenxiao Liu, Shuai Lu, Weizhu Chen, Daxin Jiang, Alexey Svyatkovskiy, Shengyu Fu, Neel Sun- daresan, and Nan Duan. Code execution with pre-trained language models. arXiv preprint arXiv:2305.05383, 2023a. Jiate Liu, Yiqin Zhu, Kaiwen Xiao, Qiang Fu, Xiao Han, Wei Yang, and Deheng Ye. Rltf: Re- inforcement learning from unit test feedback. Trans. Mach. Lea...

arXiv 2023

[8] [2004]

Deepseek-v3 technical report

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437,

arXiv