Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

What I cannot execute, I do not understand: Training and Evaluating LLMs on Program Execution Traces

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Pith's one-line read Training a language model on step-by-step execution traces—rather than static code—improves program output prediction from roughly 49% to around 80% and extends correct prediction to 14,000-step runs.

desk verdict A genuinely new dynamic-scratchpad mechanism inside a solid but under-evidenced empirical study that deserves peer review, provided contamination and small-sample concerns are addressed. read the letter →

arxiv 2503.05703 v1 pith:CIPBBFZ4 submitted 2025-02-10 cs.LG cs.AIcs.PL

classification cs.LGcs.AIcs.PL
keywords executiontracesscratchpaddynamicprogramoutputpredictioncodereasoningPythonbytetracingsynthetictestgenerationlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that an LLM can learn to execute code by training on real execution traces, and that this beats training on static code or on input-output pairs alone. It introduces Execution Tuning (E.T.), which turns roughly 300,000 Python functions—made runnable with synthetic inputs generated by an LLM and fuzzing—into more than 1.5 million execution traces, and fine-tunes Llama 3.1 8B to predict successive program states. On CruxEval and MBPP, trace-trained models reach around 80% output-prediction accuracy, compared with roughly 49% for direct-output fine-tuning. For very long runs, the paper's dynamic scratchpad—one self-contained state updated step by step instead of a growing history—extends correct predictions to executions of about 14,000 steps.

What carries the argument

The load-bearing mechanism is the dynamic scratchpad: instead of emitting the whole execution history, the model predicts the next state as a complete standalone snapshot—locals, globals, iterator iteration counts, stack, and return value—and that snapshot is fed back as the next input. This turns execution into a sequence of self-contained state-update predictions, so errors do not compound by replaying a long history, and the model can also be trained to jump several steps ahead at once. The supporting machinery is the trace-collection pipeline: a custom Python tracer built on sys.settrace records line and opcode events with full state, and synthetic inputs generated by an LLM plus fuzzing make hundreds of thousands of functions executable without manual test annotations.

What would settle it

A concrete check would hold out a set of Python functions and inputs that could not have been in the model's pretraining—fresh names, unusual variable names, post-cutoff code—trace them with the same pipeline, and compare trace-trained versus direct-trained output accuracy; if the gap disappears, the headroom comes from distribution overlap or memorization rather than learned execution. A second check would run the dynamic scratchpad on executions beyond roughly 14,000 steps and see whether accuracy collapses at a characteristic length.

Watch

Extended reading notes

Core claim

The core claim is that explicitly modeling program execution traces is a scalable and effective training signal for code reasoning. Fine-tuning on step-by-step traces, whether line-level or bytecode-level and including locals, globals, stack, and iterator positions, produces substantially better function-output prediction than fine-tuning on input-output pairs alone. The format of the scratchpad determines whether the gains survive long executions: the dynamic scratchpad, in which the model rewrites a single self-contained state at each step, lets the model chain correct predictions over roughly 14,000 steps on a 4-bit binary counter and 619 steps on Collatz, while accumulating a full trace fails at much shorter lengths.

Load-bearing premise

The load-bearing premise is that the automatically generated synthetic inputs used to build the training traces are representative of the evaluation distribution, and that the evaluation functions were not already memorized by the base model or duplicated in the training corpus; the paper reports no decontamination.

Editorial extensions

If this is right

  • On CruxEval, output prediction improves from 49.3% for direct fine-tuning to up to about 79.7% with a compact scratchpad, with dynamic scratchpads at 73–74%.
  • On MBPP's nested-loop subset, compact scratchpad with step-in reaches 80.6% accuracy, and instruction-level dynamic scratchpad with search reaches 88.2%.
  • Dynamic scratchpads handle long executions that break history-style scratchpads: Collatz up to 619 correct steps and a binary counter up to 14,055 correct steps.
  • Step-skipping works: models trained to predict N steps ahead reach similar accuracy with far fewer steps when paths are selected by confidence, and optimal path selection would make them nearly perfect.
  • Adding trace data to a code SFT mix gives only small downstream gains—around 1.2 points on GSM8K and slight HumanEval/MBPP changes—suggesting trace tuning helps state reasoning more than code generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the dynamic-scratchpad recipe—replace an accumulated history with a predict-and-update state snapshot—is a general strategy for making LLM reasoning robust over long horizons in any stateful domain, not just Python.
  • Editorial extension: the string-indexing and string-method failures suggest trace training does not repair tokenization-level weaknesses, so adding character-level or subword-aware indexing supervision would be a natural testable extension.
  • Editorial extension: because the long-run evaluations use three well-known algorithmic functions with only the function name changed, part of the measured headroom could reflect memorization; evaluating on freshly generated functions would separate learned execution from retrieval.
  • Editorial extension: the confidence (negative log-likelihood) ordering across prediction horizons suggests an inference-time policy of picking the horizon with the lowest NLL rather than always stepping one step at a time, which could transfer to other chain-of-thought settings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper introduces Execution Tuning (E.T.), a training pipeline in which LLMs are fine-tuned to model Python execution traces rather than only static code. The authors collect roughly 300k executable Python functions with automatically generated inputs, build a custom tracer that captures line-level and bytecode-level state, and train Llama-3.1-8B models on several trace representations: direct output prediction, full scratchpads, compact scratchpads, and the newly proposed dynamic scratchpads. They evaluate output prediction on CruxEval, a nested-loop subset of MBPP, and three algorithmic tasks with long executions, reporting around 80% accuracy on CruxEval and MBPP and successful rollouts of up to about 14,000 steps with dynamic scratchpads. The paper also studies mixing E.T. data into a code SFT setting, finding no conclusive downstream coding gains and a small GSM8K improvement.

Significance. If the evaluation is clean, this is a useful contribution: it shows a scalable way to build trace data without manual unit tests, provides a systematic comparison of trace granularities and scratchpad schemes, and introduces dynamic scratchpads that are plausibly better for long executions. The paper is also commendably transparent about downstream limitations, reporting small or mixed gains on HumanEval/MBPP in Table 5 and including an error analysis of indexing and string-manipulation failures. However, the headline claims are currently vulnerable because the paper does not demonstrate that evaluation functions are absent from the trace training corpus, and the long-execution evidence rests on a very small set of well-known algorithms. These issues are fixable with additional analysis and experiments, but they are load-bearing for the central claims.

major comments (4)
  1. [Section 2.1 and Tables 2-4] The ~300k-function trace corpus is built from 'unrestricted Python code' and filtered only by line coverage and similarity; the paper never states that CruxEval, MBPP, or the three algorithmic tasks used in Section 3.4 were excluded. Since CruxEval and MBPP are public benchmarks composed of common Python functions, training/evaluation overlap is a plausible alternative explanation for the observed 49.3% to ~80% improvement over direct output prediction and for the long-execution successes. Please report exact and near-duplicate overlap statistics between every evaluation function and the trace corpus, including the generated inputs, or hold out all benchmark-derived functions, or release the training data. Without one of these, the central comparison is not an evaluation of generalization to unseen execution.
  2. [Section 3.4, Table 4, Appendix A/B] The long-execution evidence for dynamic scratchpads is based on only 9 inputs per task (5 for Fibonacci) and three standard algorithms (Collatz, binary counter, iterative Fibonacci), with the function name replaced by f but the algorithm body and inputs, including n up to 3038, unchanged. Pretraining memorization of these textbook algorithms is not ruled out; the non-monotonic success pattern in Table 6 (e.g., binary counter Line-1 succeeds at n=1127 and n=3038 but fails at n=2620) is the kind of brittle behavior one would expect from memorized rollouts. Please add a larger and more diverse set of long-execution tasks, preferably functions designed for this paper, and report multiple seeds or confidence intervals so that the dynamic-scratchpad advantage is not driven by a handful of inputs.
  3. [Section 3.3 and Table 3] The MBPP evaluation uses 'slightly fewer than 100 functions'; with that sample size, differences such as 78.5% (Instruction-1) versus 73.1% (Line-1) or 80.6% versus 77.4% for compact scratchpad variants are within a few percentage points of sampling noise. Please report the exact number of functions, the selection criteria in more detail, and confidence intervals or bootstrap estimates. The claim that MBPP reveals 'higher disparity between tracing strategies' is load-bearing for the method comparison and is currently under-supported by the reported data.
  4. [Tables 1-4 and Sections 3.1-3.4] All reported accuracies are single point estimates without error bars or seed variance. Fine-tuning and evaluation are stochastic, and several comparisons are close (e.g., compact scratchpad 79.7% versus scratchpad 78.7% on CruxEval; Line-1 73.3% versus Instruction-1 73.5%). Please report at least three seeds with means and standard deviations, or bootstrap confidence intervals, so that the ordering of methods can be meaningfully assessed. This is particularly important because the paper makes comparative claims about scratchpad strategies and granularities.
minor comments (4)
  1. [Section 3.1] There is a typo: 'Unsuprisingly' should be 'Unsurprisingly'.
  2. [Section 3.3] The sentence introducing the step-in variant reads 'we introduce a variant of the compact scratchpad in this the model is able to step in other called functions'; please rephrase for clarity.
  3. [Appendix B.1 and Table 6] There are small spelling errors: 'Dijsktra' should be 'Dijkstra' in Table 6 and in the text, and 'Collazt' should be 'Collatz' in Appendix B.1.
  4. [References] The reference list contains two identical Austin et al. entries labeled 2021a and 2021b; if they are the same paper, please consolidate them.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the trace-training results are empirical evaluations against external benchmarks, with no derivation that reduces to its own inputs.

full rationale

The paper's central claims are empirical: models are fine-tuned on execution traces collected from about 300k Python functions with LLM- and fuzz-generated inputs, then evaluated on output prediction for CruxEval, MBPP, and three algorithmic long-execution tasks. There is no equation or fitted parameter that is later renamed as a prediction; the comparison between trace-based models and direct output prediction is a genuine head-to-head evaluation. The paper explicitly flags the only ground-truth-assisted result (Dijkstra-based path selection) as not directly comparable, and it labels Crux-I results as strict lower bounds due to exact-match scoring. The self-citations to CruxEval, Nye et al., and Ni et al. are methodological antecedents and evaluation choices, not load-bearing justifications of the results: CruxEval is an established benchmark with independently collected functions, and the scratchpad variants are prior methods being compared against, not assumptions that force the reported accuracies. The absence of reported decontamination and the possibility of pretraining memorization of standard algorithmic tasks are validity risks, but they are not circularity in the definitional sense used here, because the paper does not define its predictions in terms of the training data or invoke a self-authored uniqueness theorem to exclude alternatives. The results would stand or fall on replication and decontamination checks, which is an external-validity concern rather than a circular-reasoning concern.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No novel physical or mathematical entities are introduced. The empirical method rests on the tracer's correctness, the representativeness of synthetic inputs, and the absence of train/eval overlap. The step-skip horizon and SFT data-mix ratio are hand-chosen hyperparameters rather than fitted constants in a derivation.

free parameters (2)
  • Step skip horizon n = 1 to 10
    Hand-chosen range of lookahead for dynamic scratchpad training; the model is evaluated on all n and the most confident n is selected. This is a method hyperparameter, not a fitted physical constant.
  • E.T. data mix ratio in SFT = 5% or 10%
    Chosen by hand for the downstream experiments in Table 5; only two ratios are explored.
assumptions (4)
  • domain assumption The custom sys.settrace-based tracer captures all state needed for correct execution semantics.
    Section 2.1. If the tracer misses or misorders events, the training targets are wrong. The paper deliberately discards C events, which is an admitted simplification.
  • domain assumption LLM-generated synthetic inputs plus fuzzing, after coverage filters, produce representative executions.
    Section 2.1. The pipeline uses no manual test annotations, and representativeness of the generated inputs is assumed rather than demonstrated against the evaluation distribution.
  • domain assumption Evaluation functions are not memorized or present in the trace training data.
    No decontamination is reported. CruxEval and MBPP functions are standard and could overlap with the ~300k function corpus, and the three long-execution algorithms are widely available in pretraining data.
  • domain assumption Program output prediction is a valid proxy for code reasoning.
    The paper uses output prediction as the primary evaluation task and explicitly calls it a proxy, but does not establish how well it correlates with other code-understanding capabilities.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What I cannot execute, I do not understand: Training and Evaluating LLMs on Program Execution Traces." pith.science (2026). https://pith.science/paper/CIPBBFZ4

@misc{pith2026250305703,
  author       = {Pith},
  title        = {Pith review of: What I cannot execute, I do not understand: Training and Evaluating LLMs on Program Execution Traces},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CIPBBFZ4}},
  note         = {Machine review of arXiv:2503.05703}
}
read the original abstract

Code generation and understanding are critical capabilities for large language models (LLMs). Thus, most LLMs are pretrained and fine-tuned on code data. However, these datasets typically treat code as static strings and rarely exploit the dynamic information about their execution. Building upon previous work on trace modeling, we study Execution Tuning (E.T.), a training procedure in which we explicitly model real-world program execution traces without requiring manual test annotations. We train and evaluate models on different execution trace granularities (line and instruction-level) and strategies on the task of output prediction, obtaining around 80% accuracy on CruxEval and MBPP, and showing the advantages of dynamic scratchpads (i.e., self-contained intermediate computations updated by the model rather than accumulated as a history of past computations) on long executions (up to 14k steps). Finally, we discuss E.T.'s practical applications.

Figures

Figures reproduced from arXiv: 2503.05703 by the authors.

Figure 1
Figure 1. Given a natural number, a function returns the number of iterations required to arrive [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the data pipeline in E.T. We start from Python functions made executable [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Prompt for Instruction-1. We build a custom tracer leveraging Python’s built-in sys.settrace. We capture all Python function call, return, line and opcode events, and step into user-defined auxiliary functions (but not into functions from imported modules). We deliberately ignore C events because with￾out access to the source code emitting these events, the C traces would introduce noise to the data. We note that fo… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Plot showing individual state prediction accuracy (e.g., for Return, specifically for this plot and unlike in the rest of the article, we mean return statement accuracy, not full execution accuracy) when increasing N lines into the future, compared to the predictions N…
Figure 5
Figure 5. Figure 5: Plot showing individual state prediction performance when increasing N instructions into [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Can Large Language Models Reason About Complex Execution Paths? An Empirical Study on Python

    cs.SE 2025-11 conditional novelty 6.0 of 10

    State-of-the-art LLMs solve over 60% of complex Python path constraints in test generation, but their path classification is unreliable, especially for infeasible paths.

  2. Execution-First Synthetic Tool-Use Trace Generation for LLM Agents

    cs.SE 2026-07 conditional novelty 5.0 of 10

    An execution-first data generator that validates tool traces before synthesizing user tasks improves Qwen agents' tool use and answer quality, with masked reasoning supervision beating full-trace supervision.

Reference graph

Works this paper leans on

25 extracted references · 3 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    To code, or not to code? exploring impact of code in pre-training, 2024

    Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. To code, or not to code? exploring impact of code in pre-training, 2024. URL https://arxiv.org/abs/2408.10914

  3. [4]

    Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J

    Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models. CoRR, abs/2108.07732, 2021 b . URL https://arxiv.org/abs/2108.07732

  4. [5]

    Learning to Execute Programs with Instruction Pointer Attention Graph Neural Networks

    David Bieber, Charles Sutton, Hugo Larochelle, and Daniel Tarlow. Learning to execute programs with instruction pointer attention graph neural networks. CoRR, abs/2010.12621, 2020. URL https://arxiv.org/abs/2010.12621

  5. [6]

    Static prediction of runtime errors by learning to execute programs with external resource descriptions, 2022

    David Bieber, Rishab Goel, Daniel Zheng, Hugo Larochelle, and Daniel Tarlow. Static prediction of runtime errors by learning to execute programs with external resource descriptions, 2022. URL https://arxiv.org/abs/2203.03771

  6. [7]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

  7. [8]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

  8. [9]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

Show all 25 references
  1. [10]

    A note on two problems in connexion with graphs

    Edsger W Dijkstra. A note on two problems in connexion with graphs. Numerische mathematik, 1 0 (1): 0 269--271, 1959

  2. [11]

    Min, Gail Kaiser, Junfeng Yang, and Baishakhi Ray

    Yangruibo Ding, Jinjun Peng, Marcus J. Min, Gail Kaiser, Junfeng Yang, and Baishakhi Ray. Semcoder: Training code language models with comprehensive semantics, 2024. URL https://arxiv.org/abs/2406.01006

  3. [12]

    Self-play with execution feedback: Improving instruction-following capabilities of large language models, 2024

    Guanting Dong, Keming Lu, Chengpeng Li, Tingyu Xia, Bowen Yu, Chang Zhou, and Jingren Zhou. Self-play with execution feedback: Improving instruction-following capabilities of large language models, 2024. URL https://arxiv.org/abs/2406.13542

  4. [13]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

  5. [14]

    Gaunt, Marc Brockschmidt, Nate Kushman, and Daniel Tarlow

    Alexander L. Gaunt, Marc Brockschmidt, Nate Kushman, and Daniel Tarlow. Lifelong perceptual programming by example. CoRR, abs/1611.02109, 2016. URL http://arxiv.org/abs/1611.02109

  6. [15]

    Rlef: Grounding code llms in execution feedback with reinforcement learning, 2024

    Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Taco Cohen, and Gabriel Synnaeve. Rlef: Grounding code llms in execution feedback with reinforcement learning, 2024. URL https://arxiv.org/abs/2410.02089

  7. [16]

    Neural turing machines

    Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. CoRR, abs/1410.5401, 2014. URL http://arxiv.org/abs/1410.5401

  8. [17]

    Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I. Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065, 2024

  9. [18]

    Starcoder 2 and the stack v2: The next generation, 2024

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Z...

  10. [19]

    Next: Teaching large language models to reason about code execution, 2024

    Ansong Ni, Miltiadis Allamanis, Arman Cohan, Yinlin Deng, Kensen Shi, Charles Sutton, and Pengcheng Yin. Next: Teaching large language models to reason about code execution, 2024. URL https://arxiv.org/abs/2404.14662

  11. [20]

    Maxwell I. Nye, Anders Johan Andreassen, Guy Gur - Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models. C...

  12. [21]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

  13. [22]

    Code llama: Open foundation models for code, 2024

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong...

  14. [23]

    Learning to execute

    Wojciech Zaremba and Ilya Sutskever. Learning to execute. CoRR, abs/1410.4615, 2014. URL http://arxiv.org/abs/1410.4615

  15. [24]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  16. [25]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  17. [26]

    >zKxW7 yCJSN yk6N !t

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.