REVIEW 17 cited by
XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The applications of LLM Agents are becoming increasingly complex and diverse, leading to a high demand for structured outputs that can be parsed into code, structured function calls, and embodied agent commands. These developments bring significant demands for structured generation in LLM inference. Context-free grammar is a flexible approach to enable structured generation via constrained decoding. However, executing context-free grammar requires going through several stack states over all tokens in vocabulary during runtime, bringing non-negligible overhead for structured generation. In this paper, we propose XGrammar, a flexible and efficient structure generation engine for large language models. XGrammar accelerates context-free grammar execution by dividing the vocabulary into context-independent tokens that can be prechecked and context-dependent tokens that need to be interpreted during runtime. We further build transformations to expand the grammar context and reduce the number of context-independent tokens. Additionally, we build an efficient persistent stack to accelerate the context-dependent token checks. Finally, we co-design the grammar engine with LLM inference engine to overlap grammar computation with GPU executions. Evaluation results show that XGrammar can achieve up to 100x speedup over existing solutions. Combined with an LLM inference engine, it can generate near-zero overhead structure generation in end-to-end low-LLM serving.
Forward citations
Cited by 17 Pith papers
-
Imprompt: A Language Framework for Prompt Programming
Prompt programs can be typed like programs and compiled in different ways; Imprompt formalizes two such compilers and tests them on tax-law reasoning and data-labeling tasks.
-
DINGO: Constrained Inference for Diffusion LLMs
DINGO is a dynamic programming decoder over a token-level DFA that makes diffusion LLM outputs obey regular expressions while maximizing the product of per-position probabilities.
-
Why Are GUI Agents Correct but Late? Decode on the Decision-Time Critical Path, Tested with Pre-Compiled Policy Trees
Removing autoregressive decode from the decision-time critical path via pre-compiled guarded policy trees recovers contested GUI action windows when outcomes are enumerable in advance.
-
From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World
A practical evaluation protocol for AI pentesting agents that uses validated vulnerability discovery, LLM semantic matching, and bipartite scoring to assess performance in realistic, complex targets.
-
NEURA: A Unified and Retargetable Compilation Framework for Coarse-Grained Reconfigurable Architectures
NEURA flattens CGRA control flow into a pure predicated dataflow IR and reports 2.20× kernel and up to 2.71× application speedups over high-performance SOTA baselines.
-
The Format Tax
Structured-output instructions alone impose a large accuracy tax on open-weight LLMs; decoupling freeform reasoning from formatting recovers most of it, while recent closed models largely avoid the tax.
-
xGR: Efficient Generative Recommendation Serving at Scale
On real-world recommendation datasets, xGR sustains about 2.9–3.5× the throughput of vLLM/xLLM under a 200 ms P99 latency cap through GR-specific KV-cache, beam-search, and scheduling optimizations.
-
Orchestrating Dual-Boundaries: An Arithmetic Intensity Inspired Acceleration Framework for Diffusion Language Models
ODB-dLLM accelerates diffusion LLM inference by adaptively shortening responses and using a jump-share speculative decoding strategy, achieving 2.6-6.3x over Fast-dLLM.
-
Correctness-Guaranteed Code Generation via Constrained Decoding
A constrained decoding algorithm using a context-sensitive tree of parsers guarantees generated programs in the sLua language are semantically correct and, for the game API, run without errors.
-
A Practical Approach for Building Production-Grade Conversational Agents with Workflow Graphs
A production e-commerce chatbot using a workflow graph with node-specific prompts and response-masked fine-tuning reports large gains in accuracy and format compliance, beating a GPT-4o-based agent in human preference tests.
-
Agents of Diffusion: Enhancing Diffusion Language Models with Multi-Agent Reinforcement Learning for Structured Data Generation (Extended Version)
AoD pairs a frozen diffusion language model with two LLM agents that iteratively rewrite prompts from natural-language feedback, reporting better JSON diversity and validity, though the claimed RL mechanism and theore...
-
Research Challenges in Relational Database Management Systems for LLM Queries
Open-source SQL-plus-LLM database integrations fail on most representative LLM queries and run orders of magnitude slower than enterprise APIs, due to missing structured outputs, poor batching, and plan-unaware LLM calls.
-
WGRAMMAR: Leverage Prior Knowledge to Accelerate Structured Decoding
A structured decoding engine that precompiles static grammar templates and uses context-free operators with mask caching to cut time-to-first-token overhead by up to 250x versus XGrammar.
-
Surfer-H Meets Holo1: Cost-Efficient Web Agent Powered by Open Weights
The paper claims a state-of-the-art, low-cost open-weight web agent, but its WebVoyager result is inflated by training on the evaluation benchmark.
-
Earley-Driven Dynamic Pruning for Efficient Structured Decoding
A dependency-reachability pruning algorithm that culls dead Earley parser states in constrained decoding, implemented in the Formatron engine.
-
Precise Robot Command Understanding Using Grammar-Constrained Large Language Models
A fine-tuned LLM plus grammar canonicalizer and feedback loop yields higher valid robot-command rates on HuRIC than a fine-tuned LLM or grammar NLU alone.
-
The Hitchhiker's Guide to Agentic AI: From Foundations to Systems
A survey-style reference book mapping the full agentic-AI stack from transformer internals to production deployment, with no new research result.
Discussion (0). Sign in to comment.