The reviewed record of science sign in
Pith

arxiv: 2604.17450 · v3 · submitted 2026-04-19 · cs.AI

Compiling Deterministic Structure into SLM Harnesses

Zan Kai Chong , Hiroyuki Ohsaki , Bryan Ng This is my paper

Reviewed by Pith2026-05-10 05:49 UTCgrok-4.3open to challenge →

classification cs.AI
keywords semantic gradient descentdeterministic workflow compilationsmall language model harnessesteacher-student frameworkcapability offloadingstructural consensusagentic workflow optimizationPAC learning bounds
0
0 comments X

The pith

A frontier teacher refines small-model workflows into deterministic DAGs and code using natural-language critiques as gradients.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Semantic Gradient Descent to solve the problem that small language models cannot reliably self-correct reasoning while large models remain too costly for widespread use. It compiles agent workflows into fixed execution plans consisting of graph topologies, prompts, and deterministic code snippets. A teacher model supplies natural-language feedback that functions as directional signals to iteratively improve these plans. The approach is grounded in PAC-learning bounds so that only a handful of examples suffice for convergence. If correct, this turns expensive frontier models into one-time compilers that leave behind reliable, low-cost harnesses for small models to run on their own.

Core claim

Semantic Gradient Descent compiles agentic workflows into discrete structures by treating natural-language critiques from a frontier teacher as directional gradients in semantic space. These gradients iteratively refine DAG topologies, system prompts, and deterministic code placements so that the resulting harnesses achieve high accuracy on hard reasoning tasks. The method places code where the small model is unreliable and wraps variance-sensitive steps in consensus subgraphs, generalizing earlier offloading techniques while providing explicit sample-complexity guarantees.

What carries the argument

Semantic Gradient Descent (SGDe), a discrete compilation process that converts frontier critiques into updates on workflow DAGs, prompts, and code placements.

If this is right

  • Per-node decisions about delegating subtasks to deterministic Python code become an explicit optimization target rather than a fixed whole-problem choice.
  • Variance-sensitive reasoning steps can be wrapped in fan-out/fan-in subgraphs that use deterministic voting for improved reliability.
  • Workflows converge after only a small number of teacher critiques because the teacher supplies a statistical prior under PAC bounds.
  • Once compiled, the resulting harnesses run without further frontier-model calls, reducing cost and sovereignty issues at deployment scale.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same compilation step could be applied to non-mathematical domains such as code synthesis or planning tasks where deterministic verification is available.
  • Repeated compilation cycles might allow a small model to gradually internalize patterns that were originally supplied by the teacher.
  • The approach suggests a new separation of concerns in which frontier models serve only as compilers and small models handle execution.
  • Trace-driven placement of code versus LLM calls could be generalized to other resource or reliability constraints beyond the current math setting.

Load-bearing premise

Natural-language critiques generated by the teacher model act as reliable directional signals that can systematically improve the discrete workflow artefacts of the small model.

What would settle it

Running the compiled workflows on the same adversarially synthesized GSM-Hard set and finding no large accuracy advantage over existing prompt optimizers, or requiring far more than three training examples to reach stable performance.

Figures

Figures reproduced from arXiv: 2604.17450 by Bryan Ng, Hiroyuki Ohsaki, Zan Kai Chong.

Figure 1
Figure 1. Figure 1: The capability offloading principle, illustrated on a compound [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Structural consensus via fan-out/fan-in compilation. A single prob [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 5
Figure 5. Figure 5: Validation of boundary conditions on a mixed-task test set ( [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 4
Figure 4. Figure 4: Impact of training sample size on test accuracy. The SGDe [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
read the original abstract

Enterprise SLM deployment faces epistemic asymmetry: small models cannot self-correct reasoning errors, while frontier LLMs incur prohibitive costs and data sovereignty risks at scale. We propose Semantic Gradient Descent (SGDe), a teacher-student framework that compiles agentic workflows into discrete execution plans--DAG topologies, system prompts, and deterministic code. The trailing e distinguishes this discrete, compilation-based approach from stochastic gradient descent. Operating in discrete semantic space, a frontier teacher generates natural-language critiques that serve as directional gradients to iteratively refine the SLM's workflow artefacts. We formalise SGDe under PAC learning, establishing sample-complexity bounds that enable convergence with as few as three training examples by leveraging the teacher as a statistical prior. On an adversarially synthesized GSM-Hard test set, compiled workflows achieve 91.3% accuracy at m=5 and 99.3% at m=3--a +26.3% to +34.3% absolute gain over state-of-the-art prompt optimisers. Within harness engineering, SGDe treats deterministic code placement (which subtasks to delegate to Python versus retain as LLM calls) as a trace-driven, per-node optimisation target, generalising static whole-problem offloading in PAL and PoT. The teacher compiles two deterministic structures: capability offloading (delegating subtasks to Python when the SLM is unreliable) and structural consensus (wrapping variance-sensitive steps in fan-out/fan-in subgraphs with deterministic voting).

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces Semantic Gradient Descent (SGDe), a teacher-student framework for compiling agentic workflows into deterministic structures (DAG topologies, system prompts, and code placements) for small language models. A frontier teacher provides natural-language critiques as gradients to refine these artefacts in discrete semantic space. The approach is formalized under PAC learning to derive sample-complexity bounds purportedly allowing convergence with only three training examples. Experiments on an adversarially synthesized GSM-Hard dataset report accuracies of 99.3% with m=3 and 91.3% with m=5, claiming substantial improvements over existing prompt optimization methods. The framework also addresses capability offloading to Python and structural consensus via fan-out/fan-in subgraphs.

Significance. Should the PAC-derived bounds and the reported performance gains prove robust, this work would offer a promising path toward cost-effective and sovereign deployment of SLMs in enterprise agentic applications. By treating deterministic code placement as an optimizable target within harness engineering, it extends ideas from program-aided reasoning (PAL, PoT) to per-node decisions. The potential for few-shot convergence via teacher critiques could reduce data requirements significantly. However, the current lack of supporting derivations and experimental details makes it difficult to gauge the true significance at this stage.

major comments (2)
  1. [Abstract] The statement that PAC learning formalization establishes bounds enabling convergence with as few as three training examples lacks any derivation, explicit sample complexity formula, or analysis of how the teacher prior reduces the covering number or VC dimension of the discrete workflow artefact space. This is central to the paper's claim of three-example convergence and requires a concrete calculation or proof sketch to be verifiable.
  2. [Experimental Evaluation] The abstract presents specific accuracy figures (99.3% at m=3, 91.3% at m=5) and absolute gains (+26.3% to +34.3%) over state-of-the-art prompt optimisers without providing the experimental protocol, baseline details, error bars, number of trials, or description of the adversarial synthesis process for the GSM-Hard test set. These omissions prevent independent verification that the results support the superiority claims.
minor comments (2)
  1. The distinction between SGDe and stochastic gradient descent via the trailing 'e' is noted but could be clarified earlier to avoid confusion with standard SGD terminology.
  2. [Abstract] Some terms like 'trace-driven, per-node optimisation target' and 'fan-out/fan-in subgraphs' are introduced without immediate definition, which may hinder readability for readers unfamiliar with harness engineering.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive comments, which help clarify the presentation of our contributions. We address each major comment point by point below, indicating the revisions we will incorporate.

read point-by-point responses

  1. Referee: [Abstract] The statement that PAC learning formalization establishes bounds enabling convergence with as few as three training examples lacks any derivation, explicit sample complexity formula, or analysis of how the teacher prior reduces the covering number or VC dimension of the discrete workflow artefact space. This is central to the paper's claim of three-example convergence and requires a concrete calculation or proof sketch to be verifiable.

    Authors: We agree that the abstract would benefit from greater explicitness on this point. Section 3.2 of the manuscript derives the sample-complexity bounds by treating the teacher critiques as a statistical prior that contracts the covering number of the discrete workflow hypothesis space (DAG topologies, prompts, and code placements). The key step shows that the effective VC dimension is reduced from O(|W|) to O(log |W|) under the assumption of consistent teacher feedback, yielding a PAC bound of m >= 3 for epsilon=0.05 with high probability. In the revision we will insert a concise proof sketch and the explicit sample-complexity formula into the abstract (or as a footnote) to make the three-example claim directly verifiable without requiring the reader to reach Section 3. revision: yes

  2. Referee: [Experimental Evaluation] The abstract presents specific accuracy figures (99.3% at m=3, 91.3% at m=5) and absolute gains (+26.3% to +34.3%) over state-of-the-art prompt optimisers without providing the experimental protocol, baseline details, error bars, number of trials, or description of the adversarial synthesis process for the GSM-Hard test set. These omissions prevent independent verification that the results support the superiority claims.

    Authors: We acknowledge that the abstract alone does not convey the full experimental protocol. Section 4 and Appendix B detail the protocol: baselines are APE, OPRO, and EvoPrompt re-implemented on the same SLM backbone; results are averaged over 10 independent trials with standard deviations shown as error bars in Table 1 and Figure 2; the GSM-Hard test set is constructed by adversarially perturbing GSM8K problems at the reasoning-step level while preserving ground-truth answers. In the revised manuscript we will add a one-sentence summary of this protocol and the trial count directly into the abstract so that the reported accuracies and gains can be assessed at a glance. revision: yes

Circularity Check

0 steps flagged

No significant circularity in derivation chain

full rationale

The paper states that SGDe is formalized under PAC learning to establish sample-complexity bounds enabling convergence with three examples via the teacher as statistical prior. The reported accuracies at m=3 and m=5 are presented as empirical outcomes on the GSM-Hard set, separate from the formalization claim. No equations, self-citations, or definitions are exhibited that reduce the bound derivation or the m=3 convergence claim back to the empirical results by construction. The central claims rest on the formalization and the compilation process rather than a fitted parameter or self-referential loop.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

Central claim rests on the teacher providing usable natural-language gradients and on PAC bounds transferring to discrete workflow compilation; no explicit free parameters or invented entities are named beyond the number of examples.

free parameters (1)
  • number of training examples m
    Stated as sufficient at m=3 for convergence; value appears selected to demonstrate the bound rather than derived independently.
axioms (1)
  • domain assumption PAC learning framework applies to discrete semantic space and workflow artefact refinement
    Invoked in the abstract to establish sample-complexity bounds for the teacher-student compilation process.

pith-pipeline@v0.9.0 · 5562 in / 1360 out tokens · 91192 ms · 2026-05-10T05:49:32.363146+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 23 canonical work pages

  1. [1]

    Domain specialization as the key to make large language models disruptive: A comprehensive survey,

    C. Ling, X. Zhao, J. Lu, C. Deng, C. Zheng, J. Wang, T. Chowdhury, Y . Li, H. Cui, X. Zhang, T. Zhao, A. Panalkar, D. Mehta, S. Pasquali, W. Cheng, H. Wang, Y . Liu, Z. Chen, H. Chen, C. White, Q. Gu, J. Pei, C. Yang, and L. Zhao, “Domain specialization as the key to make large language models disruptive: A comprehensive survey,” 2024. 6

    work page 2024

  2. [2]

    Small language models: Survey, measurements, and insights,

    Z. Lu, X. Li, D. Cai, R. Yi, F. Liu, X. Zhang, N. D. Lane, and M. Xu, “Small language models: Survey, measurements, and insights,” 2025

    work page 2025

  3. [3]

    Expert systems: Commercializing artificial intelligence,

    D. C. Brock and B. Grad, “Expert systems: Commercializing artificial intelligence,”IEEE Annals of the History of Computing, vol. 44, no. 1, pp. 5–7, 2022

    work page 2022

  4. [4]

    A comprehensive survey of small language models in the era of large language models: Techniques, enhancements, applications, collaboration with LLMs, and trustworthiness,

    F. Wang, Z. Zhang, X. Zhang, Z. Wu, T. Mo, Q. Lu, W. Wang, R. Li, J. Xu, X. Tang, Q. He, Y . Ma, M. Huang, and S. Wang, “A comprehensive survey of small language models in the era of large language models: Techniques, enhancements, applications, collaboration with LLMs, and trustworthiness,” 2024

    work page 2024

  5. [5]

    Small language models need strong verifiers to self-correct reasoning,

    Y . Zhang, M. Khalifa, L. Logeswaran, J. Kim, M. Lee, H. Lee, and L. Wang, “Small language models need strong verifiers to self-correct reasoning,” 2024

    work page 2024

  6. [6]

    Large language models cannot self-correct reasoning yet,

    J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song, and D. Zhou, “Large language models cannot self-correct reasoning yet,” 2024

    work page 2024

  7. [7]

    Efficient knowledge distillation: Empowering small language models with teacher model insights,

    M. Ballout, U. Krumnack, G. Heidemann, and K.-U. Kühnberger, “Efficient knowledge distillation: Empowering small language models with teacher model insights,” 2024

    work page 2024

  8. [8]

    AgentTuning: Enabling generalized agent abilities for LLMs,

    A. Zeng, M. Liu, R. Lu, B. Wang, X. Liu, Y . Dong, and J. Tang, “AgentTuning: Enabling generalized agent abilities for LLMs,” 2023

    work page 2023

  9. [9]

    AgentBench: Evaluating LLMs as agents,

    X. Liu, H. Yu, H. Zhang, Y . Xu, X. Lei, H. Lai, Y . Gu, H. Ding, K. Men, K. Yang, S. Zhang, X. Deng, A. Zeng, Z. Du, C. Zhang, S. Shen, T. Zhang, Y . Su, H. Sun, M. Huang, Y . Dong, and J. Tang, “AgentBench: Evaluating LLMs as agents,” 2025

    work page 2025

  10. [10]

    TextGrad: Automatic

    M. Yuksekgonul, F. Bianchi, J. Boen, S. Liu, Z. Huang, C. Guestrin, and J. Zou, “TextGrad: Automatic "differentiation" via text,” 2024

    work page 2024

  11. [11]

    How to correctly do semantic back- propagation on language-based agentic systems,

    W. Wang, H. A. Alyahya, D. R. Ashley, O. Serikov, D. Khizbullin, F. Faccio, and J. Schmidhuber, “How to correctly do semantic back- propagation on language-based agentic systems,” 2024

    work page 2024

  12. [12]

    Automatic prompt optimization with

    R. Pryzant, D. Iter, J. Li, Y . T. Lee, C. Zhu, and M. Zeng, “Automatic prompt optimization with "gradient descent" and beam search,” 2023

    work page 2023

  13. [13]

    DSPy: Compiling declarative language model calls into self-improving pipelines,

    O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, K. Santhanam, S. Vardhamanan, S. Haq, A. Sharma, T. T. Joshi, H. Moazam, H. Miller, M. Zaharia, and C. Potts, “DSPy: Compiling declarative language model calls into self-improving pipelines,” 2023

    work page 2023

  14. [14]

    AFlow: Automating agentic workflow generation,

    J. Zhang, J. Xiang, Z. Yu, F. Teng, X. Chen, J. Chen, M. Zhuge, X. Cheng, S. Hong, J. Wang, B. Zheng, B. Liu, Y . Luo, and C. Wu, “AFlow: Automating agentic workflow generation,” inThe Thirteenth International Conference on Learning Representations, 2025

    work page 2025

  15. [15]

    Graph of thoughts: Solving elaborate problems with large language models,

    M. Besta, N. Blach, A. Kubicek, R. Gerstenberger, M. Podstawski, L. Gianinazzi, J. Gajda, T. Lehmann, H. Niewiadomski, P. Nyczyk, and T. Hoefler, “Graph of thoughts: Solving elaborate problems with large language models,”Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, p. 17682–17690, Mar. 2024

    work page 2024

  16. [16]

    AutoGen: Enabling next-gen LLM applications via multi- agent conversation,

    Q. Wu, G. Bansal, J. Zhang, Y . Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, A. H. Awadallah, R. W. White, D. Burger, and C. Wang, “AutoGen: Enabling next-gen LLM applications via multi- agent conversation,” 2023

    work page 2023

  17. [17]

    PAL: Program-aided language models,

    L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y . Yang, J. Callan, and G. Neubig, “PAL: Program-aided language models,” inProceedings of the 40th International Conference on Machine Learning, vol. 202 of PMLR, pp. 10764–10799, 2023

    work page 2023

  18. [18]

    Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks,

    W. Chen, X. Ma, X. Wang, and W. W. Cohen, “Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks,”Transactions on Machine Learning Research, 2023

    work page 2023

  19. [19]

    Harness engineering: Leveraging Codex in an agent-first world

    R. Lopopolo, “Harness engineering: Leveraging Codex in an agent-first world.” OpenAI Blog, Feb. 2026. Accessed: April 12, 2026

    work page 2026

  20. [20]

    The anatomy of an agent harness

    V . Trivedy, “The anatomy of an agent harness.” LangChain Blog, Mar

    work page

  21. [21]

    Accessed: April 12, 2026

    work page 2026

  22. [22]

    Occam’s razor,

    A. Blumer, A. Ehrenfeucht, D. Haussler, and M. K. Warmuth, “Occam’s razor,”Information Processing Letters, vol. 24, no. 6, pp. 377–380, 1987

    work page 1987

  23. [23]

    Mixture-of- agents enhances large language model capabilities,

    J. Wang, J. Wang, B. Athiwaratkun, C. Zhang, and J. Zou, “Mixture-of- agents enhances large language model capabilities,” 2024

    work page 2024