Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

XGrammar-2 claims that dynamic structured generation for LLM agents—where the required output format changes across requests and within a request—can be served with over 6× faster grammar compilation than prior engines and near-zero end-to-

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 12:02 UTC pith:RKCMNGGY

load-bearing objection Good systems idea with a real efficiency win, but the repetition-compression algorithm as printed over-accepts, so the 100% schema-validity claim doesn't hold up. the 4 major comments →

arxiv 2601.04426 v3 pith:RKCMNGGY submitted 2026-01-07 cs.AI

XGrammar-2: Efficient Dynamic Structured Generation Engine for Agentic LLMs

Linzhang Li , Yixin Dong , Guanjie Wang , Ziyi Xu , Alexander Jiang , Tianqi Chen This is my paper
classification cs.AI
keywords structured generationconstrained decodingtool callingagentic LLMsEarley parsergrammar cachingtag dispatchjust-in-time compilation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that agentic LLM workloads—tool calling, response protocols—demand structure that changes per request and per token, and that existing constrained-decoding engines, which precompile a fixed grammar, pay prohibitive costs when the grammar is dynamic. To fix this, XGrammar-2 introduces TagDispatch, a grammar construct that switches the active constraint when a tag like a tool name appears, and a Cross-Grammar Cache that reuses token-mask computation across grammars that share substructures. The engine further uses an Earley-parser-based adaptive cache, just-in-time compilation, and repetition state compression. The central claim, backed by experiments, is that dynamic grammars compile over 6× faster and add near-zero end-to-end latency, making per-request and per-call structure enforcement practical at scale. A sympathetic reader would care because this is what makes tool calling with dozens or hundreds of tools feasible in production serving.

Core claim

The central discovery is that dynamic structured generation can be made nearly free by changing what gets cached. Instead of compiling the whole grammar ahead of time and caching token masks for every parser state, the engine caches at the substructure level, identifying reusable FSM sub-automata through a hierarchical hashing algorithm, and compiles only the masks encountered at runtime via just-in-time compilation. TagDispatch expresses intra-request switching as a first-class EBNF construct, so a tag such as '<function=get_weather>' routes decoding into the corresponding JSON-schema grammar and then back to free text. The engine replaces the pushdown-automaton state representation with an

What carries the argument

The load-bearing pieces are the Earley-parser-based adaptive token mask cache, the Cross-Grammar Cache with its hierarchical FSM hashing, and TagDispatch. The adaptive cache stores token validity only for scannable Earley states—states whose next symbol is a terminal—avoiding the exponential state blow-up of pushdown-automaton caches. The Cross-Grammar Cache assigns consistent hash values to FSMs, so when two grammars share a substructure (a JSON schema, a tool argument pattern), the cached accepted/rejected/uncertain token sets are reused across requests, with lookahead mismatches repaired by rechecking only the uncertain tokens. TagDispatch is an EBNF intrinsic that uses an Aho–Corasick au

Load-bearing premise

The engine's correctness rests on the assumption that its compressed repetition states and hash-based cache reuse accept exactly the same set of token sequences as the original grammar—so that no schema-invalid token can ever be produced.

What would settle it

Build a grammar containing a repetition R{20,1024} and decode a large corpus through the compressed engine, checking whether any emitted string has fewer than 20 or more than 1024 repetitions; or construct two non-deterministic FSMs that hash to the same value but accept different languages, then observe whether a cache hit ever yields a token mask that the original grammar would reject. Either outcome would refute the 100% schema-validity claim and the semantic-preservation assumption.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Dynamic tool calling with dozens or hundreds of tools becomes practical: grammar construction drops from seconds to roughly 10 ms, reducing time-to-first-token in agentic serving.
  • End-to-end latency with constraint enforcement stays within about 6% of unconstrained decoding, so output reliability gains no longer cost throughput.
  • Cross-grammar cache reuse means even when entire grammars differ across requests (near zero full-structure reuse), roughly half the substructure computation can be reused, making per-request compilation affordable.
  • TagDispatch makes intra-request structure switching expressible and compact, avoiding the rapid growth of hand-translated EBNF dispatchers as the number of tags increases.
  • The combination of just-in-time compilation and repetition compression cuts preprocessing time by over three orders of magnitude on JSON schema benchmarks while keeping per-token mask generation under about 130 microseconds.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the semantic-preservation assumptions hold, the same architecture could extend to other dynamic structures—multi-step code generation, changing SQL schemas, or chained tool protocols—where the active grammar is not known until generation time.
  • The hierarchical hash-based substructure reuse suggests a fingerprinting approach that serving systems could use to version grammars incrementally, avoiding recompilation when only a subset of tools or fields change.
  • The configurable just-in-time tradeoff between prefill-time and decode-time compilation could be tuned automatically per hardware backend or per workload, a knob the paper leaves as a fixed constant.
  • A natural testable extension is applying repetition-state compression to streaming or incremental validation of long structured outputs, where the compressed state could serve as a compact representation of the current constraint frontier.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents XGrammar-2, a structured generation engine targeting dynamic agentic LLM workloads. It introduces TagDispatch for intra-request tag-triggered structure switching, a cross-grammar cache for inter-request substructure reuse, an Earley-based adaptive token mask cache, JIT compilation, and repetition state compression. The experiments claim over 6x faster compilation than prior engines and near-zero end-to-end overhead in LLM serving systems.

Significance. The problem is timely and practically important: dynamic tool calling and response protocols are central to modern agentic LLM serving. The proposed design—especially the TagDispatch abstraction and cross-grammar cache—is plausible and would be a meaningful contribution if the efficiency and correctness claims hold. The paper also reports integration with SGLang and vLLM, which increases the potential impact. However, the correctness of the core repetition compression algorithm is not established, and as printed the algorithm is incorrect, which undermines the central 100% schema-validity claim. The end-to-end evaluation also contains baseline-comparison issues that need to be addressed.

major comments (4)
  1. [3.6, Algorithm 3 (Appendix B)] Algorithm 3's min<t branch is incorrect. For R{min,max} with min<t<max, the expression Concat(Repeat(t,max), Expand(0,max-t)) accepts repetition counts in [t, 2max-t]; the union with Expand(min,t) covers [min, 2max-t] rather than [min,max]. Concretely, for R{2,5} with t=3, the printed algorithm accepts 6 and 7 repetitions, violating the schema bound. This directly contradicts the §4.5 claim of 100% schema-valid tool-call arguments. The runtime k-check mentioned in §3.6 is not specified or proven to restore exactness; the cache built from the over-approximating expression can accept the token that starts a 7th repetition. Please correct the algorithm or provide a formal correctness proof, including a precise description and proof of any runtime bound check.
  2. [4.3, Figure 9] The comparison against llguidance is uninterpretable for two of the four models. The text reports that llguidance produces empty outputs for Qwen3-0.6B and induces language drift from English to other languages for Llama3.1-8B. For these models, the reported latency and throughput numbers are not measuring the same task as the other engines. The claim that 'XGrammar-2 shows a small latency and better compatibility' is not supported unless these cases are handled separately or excluded, or a valid llguidance fallback configuration is provided and evaluated.
  3. [4.4, Table 4] The ablation shows that the full optimization stack yields a per-token mask generation time of 126.49 us, which is about 2.8x slower than the Earley baseline of 45.50 us. The abstract and text emphasize 'near-zero overhead,' but this is only demonstrated in end-to-end measurements where mask generation may be hidden by other latencies. Please provide an end-to-end overhead breakdown (e.g., time-to-first-token and per-token latency relative to unconstrained decoding) and discuss the discrepancy between the per-token mask time and the near-zero-overhead claim.
  4. [3.3, Appendix C] The cross-grammar cache relies on the claim that if two FSMs have equal hash values, they must have the same structure. No collision analysis or formal proof is given, and Appendix C concedes inconsistent hashes for non-DFA or duplicated FSMs. If a hash collision ever occurs, the reused token mask cache could accept invalid tokens or reject valid ones, which would again violate the §4.5 correctness guarantee. Please state the precise invariant ('equal hash implies isomorphic automaton') and prove it for the supported grammar subset, or provide strong empirical validation on the evaluated grammar classes.
minor comments (5)
  1. [4.3] Typo: 'SgLang' should be 'SGLang'. Also, the model name 'Qwen-0.6B' is inconsistent with 'Qwen3-0.6B' used in the text.
  2. [Figure 9] The bar chart would be easier to read with numerical labels on each bar or a separate table, since the y-axis values in the text are dense and hard to map.
  3. [4.1, Table 1] In the static setting, all requests use the same 5/20/50 tools, so the full-structure reuse rate should be 100%. The reported 99.0% needs an explanation (e.g., measurement noise or a small fraction of mismatched metadata).
  4. [4.5] The correctness claim is stated as 'by construction' but no proof or machine-checked verification is provided. A formal theorem would be valuable, especially given the complexity of the compression and caching logic.
  5. [General] The paper says the engine is 'open-source' but does not provide a repository URL. Please include an artifact link; this is important for reproducibility, especially since no error bars or repeated runs are reported for the efficiency numbers.

Circularity Check

0 steps flagged

No significant circularity: XGrammar-2's efficiency results are benchmarked against independent engines; its reliance on the authors' XGrammar is foundational but not load-bearing for the new dynamic-generation claims.

full rationale

The central efficiency claims are not derived from assumed values. Compilation and mask-generation times are measured against independent engines (llguidance, Outlines, XGrammar) on external datasets (CONFETTI, BFCL-v3, JSONSchemaBench), and the ablation study measures JIT, cross-grammar cache, and repetition-state compression against a baseline Earley implementation. No parameter is fitted to the reported target metric and then relabeled as a prediction. The main self-citation is to XGrammar [9], the prior engine this work extends; it is cited for cache semantics and a foundational parser, but the dynamic dispatch (TagDispatch) and cross-grammar reuse are independently implemented and measured. Section 4.5's 'by construction' semantic-preservation assertion is an equivalence claim to XGrammar rather than a circular reduction, though it is under-proved. The printed Algorithm 3 has a genuine soundness risk: in the min<t branch, choice = Concat(Repeat(t,max), Expand(0,max-t)) accepts up to 2*max-t repetitions instead of max, which would contradict the reported 100% schema validity in Section 4.5. Appendix C's admission of inconsistent hashes for non-DFA or duplicated FSMs similarly weakens cache-reuse guarantees. These are correctness/robustness concerns, not circularity, and do not change the verdict that the efficiency derivation is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 2 invented entities

The central efficiency claims rest on tuning constants (JIT lookahead count K, repetition threshold t) whose values are not reported, and on unproved semantic-preservation assumptions in the hash/cache and repetition-compression paths. These are the main things the reader pays for upstream: if any is wrong, the near-zero-overhead/100%-validity claims can fail.

free parameters (2)
  • JIT compile-ahead count K = unspecified (adjusted for best performance)
    Section 3.5: 'we will try to calculate K most time-consuming token mask, when the LLM is prefilling (K is a fixed value, which is adjusted for the best performance).' The optimal K is not derived from first principles; it is a tuning knob affecting the compile/runtime trade-off measured in the ablation.
  • Repetition compression threshold t = unspecified (chosen constant)
    Algorithm 3 (Section 3.6) uses constant t to decide when to expand vs. compress repetitions; t changes the state/cache trade-off, and the paper reports no sensitivity analysis.
axioms (5)
  • standard math Earley parser correctness for CFGs used here; scan/predict/complete rules in Table 6 are sound and complete.
    The token-mask cache in §3.4 and formal definitions in Appendix D rely on the standard Earley parsing algorithm [10,23]; correctness assumed.
  • standard math Aho-Corasick automaton gives exact simultaneous tag matching in dispatching mode.
    TagDispatch (§3.2) uses AC automaton [2] for online matching; standard algorithm assumed.
  • domain assumption Hash-based FSM equivalence is safe for cache reuse: same hash => same structure.
    Cross-grammar cache (§3.3) reuses masks when FSM hashes match. Appendix C admits inconsistent hashes for non-DFA FSMs and duplicated FSMs, and the assertion that false-positive matches cannot occur is not proven.
  • ad hoc to paper Repetition state compression preserves exact language of R{l,r}.
    Algorithm 3 (§3.6) compresses large repetition ranges to a single state accepting R{0,t+1} plus runtime counter k; the claim 'this guarantees the correctness' is asserted, not proved, and requires each repetition to consume ≥1 character.
  • domain assumption CONFETTI, BFCL-v3, and JSONSchemaBench are representative of dynamic agentic structured-generation workloads.
    All efficiency conclusions in §4 are drawn from these datasets and specific hardware; transfer to other dynamic workloads is assumed.
invented entities (2)
  • TagDispatch intrinsic no independent evidence
    purpose: First-class grammar construct to switch between free-form text and structured sub-grammars on tag matches
    New construct; no artifact link or formal semantics provided to independently verify its correctness/expressiveness.
  • Cross-grammar cache pool no independent evidence
    purpose: Reuse token-mask caches across FSMs with shared substructures
    Efficiency depends on the hash-sufficiency assumption; no external falsifiable handle.

pith-pipeline@v1.3.0-alltime-deepseek · 18015 in / 15851 out tokens · 151161 ms · 2026-08-03T12:02:32.546581+00:00 · methodology

0 comments
read the original abstract

Modern LLM agents increasingly rely on dynamic structured generation, such as tool calling and response protocols. Unlike traditional structured generation with static structures, these workloads vary both across requests and within a request, posing new challenges to existing engines. We present XGrammar-2, a structured generation engine for dynamic agentic workloads. Our design is based on two key ideas: first-class support for tag-triggered structure switching, and fine-grained reuse across requests with different output structures. Concretely, XGrammar-2 introduces TagDispatch for dynamic structural dispatching and Cross-Grammar Cache for substructure-level cache reuse across grammars. It further improves efficiency with an Earley-based adaptive token mask cache, just-in-time compilation, and repetition state compression. Experiments show that XGrammar-2 achieves over 6x faster compilation than prior structured generation engines, and incurs near-zero end-to-end overhead in modern LLM serving systems.

Figures

Figures reproduced from arXiv: 2601.04426 by Alexander Jiang, Guanjie Wang, Linzhang Li, Tianqi Chen, Yixin Dong, Ziyi Xu.

Figure 1
Figure 1. Figure 1: Some examples of tool calling and response proto [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our approach. We design a new dynamic dispatching semantics, TagDispatch (§3.2), to efficiently support [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The definition and the constructed automata from [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The perfect cache hit and the partial cache hit. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Repetition State Compression [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Average Per-token Overhead in Llama’s Tool Calling [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Compilation Time in Llama’s Tool Calling Format [PITH_FULL_IMAGE:figures/full_fig_p007_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: End-to-end Function Calling Latency. The results in [PITH_FULL_IMAGE:figures/full_fig_p008_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Comparison between the Earley Parser and PDA [PITH_FULL_IMAGE:figures/full_fig_p009_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: JSONSchemaBench [PITH_FULL_IMAGE:figures/full_fig_p014_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?

    cs.AI 2026-05 unverdicted novelty 8.0

    VibeServe demonstrates that AI agents can synthesize bespoke LLM serving systems end-to-end, remaining competitive with vLLM in standard settings while outperforming it in six non-standard scenarios involving unusual ...

  2. Schema Key Wording as an Instruction Channel in Structured Generation under Constrained Decoding

    cs.CL 2026-04 unverdicted novelty 7.0

    Schema-key wording functions as an implicit instruction channel under constrained decoding, with experiments showing that rephrasing only the keys can substantially change accuracy on math benchmarks while prompt, mod...

Reference graph

Works this paper leans on

40 extracted references · 16 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Information technology — Syntactic metalanguage — Extended BNF

    1996. Information technology — Syntactic metalanguage — Extended BNF

    1996

  2. [2]

    Aho and Margaret J

    Alfred V. Aho and Margaret J. Corasick. 1975. Efficient string matching: an aid to bibliographic search.Commun. ACM18, 6 (June 1975), 333–340. doi:10.1145/ 360825.360855

    arXiv 1975

  3. [3]

    Tamer Alkhouli, Katerina Margatina, James Gung, Raphael Shu, Claudia Zaghi, Monica Sunkara, and Yi Zhang. 2025. CONFETTI: Conversational Function- Calling Evaluation Through Turn-Level Interactions. arXiv:2506.01859 [cs.CL] https://arxiv.org/abs/2506.01859

    Pith/arXiv arXiv 2025

  4. [4]

    John Bridle. 1989. Training Stochastic Model Recognition Algorithms as Net- works can Lead to Maximum Mutual Information Estimation of Parameters. In Advances in Neural Information Processing Systems, D. Touretzky (Ed.), Vol. 2. Morgan-Kaufmann. https://proceedings.neurips.cc/paper_files/paper/1989/file/ 0336dcbab05b9d5ad24f4333c7658a0e-Paper.pdf

    1989

  5. [5]

    Sahil Chaudhary. 2023. Code Alpaca: An Instruction-following LLaMA model for code generation. https://github.com/sahil280114/codealpaca

    2023

  6. [6]

    N. Chomsky. 1956. Three models for the description of language.IRE Transactions on Information Theory2, 3 (1956), 113–124. doi:10.1109/TIT.1956.1056813

    arXiv 1956

  7. [7]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai D...

    Pith/arXiv arXiv 2025

  8. [8]

    Daniel Deutsch, Shyam Upadhyay, and Dan Roth. 2019. A General-Purpose Algorithm for Constrained Sequential Inference. InProceedings of the 23rd Con- ference on Computational Natural Language Learning (CoNLL), Mohit Bansal and Aline Villavicencio (Eds.). Association for Computational Linguistics, Hong Kong, China, 482–492. doi:10.18653/v1/K19-1045

    doi:10.18653/v1/k19-1045 2019

  9. [9]

    Yixin Dong, Charlie F Ruan, Yaxing Cai, Ruihang Lai, Ziyi Xu, Yilong Zhao, and Tianqi Chen. 2024. Xgrammar: Flexible and efficient structured generation engine for large language models.Proceedings of Machine Learning and Systems 7(2024)

    2024

  10. [10]

    Jay Earley. 1970. An efficient context-free parsing algorithm.Commun. ACM13, 2 (1970), 94–102. doi:10.1145/362007.362035

    arXiv 1970

  11. [11]

    Saibo Geng, Hudson Cooper, Michał Moskal, Samuel Jenkins, Julian Berman, Nathan Ranchin, Robert West, Eric Horvitz, and Harsha Nori. 2025. Gen- erating Structured Outputs from Language Models: Benchmark and Studies. arXiv:2501.10868 [cs.CL] https://arxiv.org/abs/2501.10868

    Pith/arXiv arXiv 2025

  12. [12]

    Guidance-ai. 2024. GitHub - guidance-ai/llguidance: Super-fast Structured Out- puts — github.com. https://github.com/guidance-ai/llguidance. [Accessed 13-10- 2025]

    2024

  13. [13]

    Caleb Helbling. 2020. Directed Graph Hashing.CoRRabs/2002.06653 (2020). arXiv:2002.06653 https://arxiv.org/abs/2002.06653

    Pith/arXiv arXiv 2020

  14. [14]

    2024.ModelTC/lightllm

    hiworldwzj, shihaobai, sufubao, WANDY666, FlyingFlame, llehtahw, LiangLiu, wxd000000, fuheaven, XHPlus, Chielo, Yang Yong, and_gate, sangchengmeng, wangzhihong, singularity, Shuo Yang, Wu SiYu, Tracin, Elsa Granger, Hamel Husain, S A G A R, SunXiaoye, Tao Peng, Uranus, Yunfeng Bai, Yunqian Fan, bingo, liuhuakai, and XFPlus. 2024.ModelTC/lightllm. https://...

    2024

  15. [15]

    Michael Kuchnik, Virginia Smith, and George Amvrosiadis. 2023. Validating large language models with relm.Proceedings of Machine Learning and Systems5 (2023), 457–476

    2023

  16. [16]

    Dominik Kundel. 2025. OpenAI Harmony Response Format. https://cookbook. openai.com/articles/openai-harmony/. Accessed: 2025-10-27

    2025

  17. [17]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Mem- ory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles

    2023

  18. [18]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Loge...

    Pith/arXiv arXiv 2023

  19. [19]

    Meta-AI. 2024. Tool calling with Llama. https://www.llama.com/resources/ cookbook/toolcalling-with-llama/. Accessed: 2025-10-27

    2024

  20. [20]

    2026.What is Foundry Agent Service?Microsoft

    Microsoft. 2026.What is Foundry Agent Service?Microsoft. https://learn. microsoft.com/en-us/azure/ai-foundry/agents/overview Accessed: 2026-02-22; Microsoft Learn documentation on Azure AI Foundry Agent Service overview

    2026

  21. [21]

    MLC team. 2023. MLC-LLM. https://github.com/mlc-ai/mlc-llm

    2023

  22. [22]

    Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yan- ping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, Jilin Chen, Alex Beutel, and Ahmad Beirami. 2024. Controlled Decoding from Language Models. arXiv:2310.17022 [cs.LG] https://arxiv.org/abs/2310.17022

    Pith/arXiv arXiv 2024

  23. [23]

    Andreas Opedal, Ran Zmigrod, Tim Vieira, Ryan Cotterell, and Jason Eisner

  24. [24]

    OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Apple- baum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark, Dan Cook, Marat Dukhan, Casey Dvorak, Kevin Fives,...

    Pith/arXiv arXiv 2025

  25. [25]

    OpenAI Help Center. 2025. Apps in ChatGPT. https://help.openai.com/en/articles/ 11487775-apps-in-chatgpt. https://help.openai.com/en/articles/11487775-apps- in-chatgpt Accessed: 2026-02-27

    arXiv 2025

  26. [26]

    O’Brien, Carrie J

    Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simulacra of Human Behavior. InIn the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23)(San Francisco, CA, USA)(UIST ’23). Association for Computing Machinery, New York, NY, USA....

    2023

  27. [27]

    Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E

    Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. 2025. The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models. In Forty-second International Conference on Machine Learning

    2025

  28. [28]

    Qwen. 2024. Function Calling – Qwen. https://qwen.readthedocs.io/en/latest/ framework/function_call.html. Accessed: 2025-10-27

    2024

  29. [29]

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiao- qing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cris- tian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, ...

    Pith/arXiv arXiv 2024

  30. [30]

    Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. 2021. PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models. arXiv:2109.05093 [cs.CL] https://arxiv.org/abs/2109.05093

    Pith/arXiv arXiv 2021

  31. [31]

    Saurous, and Yoon Kim

    Bailin Wang, Zi Wang, Xuezhi Wang, Yuan Cao, Rif A. Saurous, and Yoon Kim

  32. [32]

    Ran Wang, Xiaoxuan Liu, Hao Ren, Gang Chen, Fanchao Qi, and Maosong Sun. 2025. WGRAMMAR: Leverage Prior Knowledge to Accelerate Structured Decoding. arXiv:2507.16768 [cs.AI] https://arxiv.org/abs/2507.16768

    Pith/arXiv arXiv 2025

  33. [33]

    arXiv:2305.19234 [cs.CL] https://arxiv.org/abs/2305.19234

    Grammar Prompting for Domain-Specific Language Generation with Large Language Models. arXiv:2305.19234 [cs.CL] https://arxiv.org/abs/2305.19234

    Pith/arXiv arXiv

  34. [34]

    Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, and Luis Ceze

  35. [35]

    Willard and Rémi Louf

    Brandon T. Willard and Rémi Louf. 2023. Efficient Guided Generation for Large Language Models. arXiv:2307.09702 [cs.CL] https://arxiv.org/abs/2307.09702

    Pith/arXiv arXiv 2023

  36. [36]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung- Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, Carlsbad, CA, 521–538. https: //www.usenix.org/conference/osdi22/presentation/yu

    2022

  37. [37]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Lan- guage Model Programs. arXiv:2312.07104 [cs.AI] https://arxiv.org/abs/2312.07104 A The Hash Algorithm for Simple Cycle Structure ...

    Pith/arXiv arXiv 2024

  38. [38]

    Pengcheng Yin and Graham Neubig. 2017. A Syntactic Neural Model for General- Purpose Code Generation. arXiv:1704.01696 [cs.CL] https://arxiv.org/abs/1704. 01696

    Pith/arXiv arXiv 2017

  39. [2023]

    arXiv:2307.02982 [cs.CL] https://arxiv.org/abs/2307.02982

    Efficient Semiring-Weighted Earley Parsing. arXiv:2307.02982 [cs.CL] https://arxiv.org/abs/2307.02982

    Pith/arXiv arXiv

  40. [2025]

    arXiv:2501.01005 [cs.DC] https://arxiv.org/abs/2501.01005

    FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving. arXiv:2501.01005 [cs.DC] https://arxiv.org/abs/2501.01005

    Pith/arXiv arXiv