Pith. sign in

REVIEW 3 major objections 5 minor 5 cited by

STATIC flattens the prefix tree of allowed semantic IDs into a static CSR matrix, making constrained decoding an O(1) vectorized lookup during LLM-based generative retrieval on TPUs/GPUs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 20:37 UTC pith:L6PHPGXB

load-bearing objection A credible, well-engineered constrained-decoding system with a real production deployment, but the O(1) overhead claim needs qualification. the 3 major comments →

arxiv 2602.22647 v2 pith:L6PHPGXB submitted 2026-02-26 cs.IR cs.CLcs.LG

Vectorizing the Trie: Efficient Constrained Decoding for LLM-based Generative Retrieval on Accelerators

Zhengyang Su , Isay Katsman , Yueqi Wang , Ruining He , Lukasz Heldt , Raghunandan Keshavan , Shao-Chuan Wang , Xinyang Yi
show 5 more authors
Mingyan Gao Onkar Dalal Lichan Hong Ed Chi Ningren Han
This is my paper
classification cs.IR cs.CLcs.LG
keywords generative retrievalconstrained decodingprefix treeCSR sparse matrixsemantic IDsTPU/GPU accelerationbeam searchcold-start recommendation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that constrained decoding—restricting a generative retrieval LLM to a fixed set of valid item identifiers—can be made nearly free on accelerator hardware by flattening the prefix tree of allowed identifiers into a static sparse transition matrix and performing all validity checks as vectorized, branch-free lookups. It reports a per-step latency overhead of 0.033 ms on a 3-billion-parameter production model with a 20-million-item freshness constraint, which amounts to 0.25% of inference time and is 948x faster than a CPU trie and 47–1033x faster than on-device binary-search baselines. This matters because generative retrieval models otherwise spend inference budget generating identifiers for unavailable or stale items, and strict output-space control is what makes business-logic constraints such as freshness, inventory, or category practical at scale. The paper also shows that constraining decoding to a cold-start item set alone improves cold-start recommendation recall from 0% to 4.29% on one benchmark subset.

Core claim

The core discovery is that a prefix-tree constraint does not need to be traversed as a graph at decode time. The paper flattens the trie into a Compressed Sparse Row (CSR) transition matrix, where each trie node becomes a state index and each allowed transition becomes a (token, next-state) entry. At every decoding step, the current node for each beam is used to slice a fixed-width block of rows from this matrix, a range mask sanitizes the entries for nodes with fewer children, and a scatter operation projects the valid tokens into a dense log-probability mask. The result is a single static computation graph with coalesced memory reads—no CPU round-trips, no pointer chasing, no dynamic contr

What carries the argument

The central object is the Sparse Transition Matrix (STM): a CSR matrix whose rows are trie nodes, columns are semantic token IDs, and values are child node IDs, plus row pointers into a stacked (edge, token, next-state) layout. The Vectorized Node Transition Kernel (VNTK) turns a trie traversal into four vectorized operations—boundary lookup, speculative fixed-length DynamicSlice, Range-based sanitization, and Scatter projection—so that every beam advances through the constraint structure in one branch-free pass. For the first d=2 layers, a dense bit-packed mask tensor handles the high branch factor cheaply; deeper layers use the sparse kernel. This hybrid design is what converts an irregula

Load-bearing premise

The load-bearing premise is that the constraint set is known in advance and stays fixed for the lifetime of the compiled kernel, and that later trie levels have small maximum branch factors; if constraints change frequently or deep branching becomes large, the reported near-zero overhead does not hold.

What would settle it

Measure end-to-end latency of STATIC when the constraint set is updated every N seconds: rebuild the CSR matrix, recompile the kernel, and include that time in the per-step overhead. Also construct a constraint set of 10^7 uniform-random Semantic IDs where levels 3+ have a maximum branch factor comparable to |V| and measure VNTK time; if latency grows linearly with branch factor at deep levels, the O(1)-in-|C| claim holds only for clustered, real-world Semantic ID distributions.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Strict constraint compliance becomes practical at production scale: a generative retriever can be forced to output only fresh, in-stock, or category-valid items without paying a large latency penalty.
  • Per-step constraint overhead of 0.25% of inference time means constraints can be enabled by default in high-throughput serving, not reserved for rare or offline use cases.
  • Since I/O complexity is O(1) in constraint-set size, latency stays nearly flat as the constrained vocabulary grows from 10^5 to 10^8 items, unlike binary-search methods whose latency grows logarithmically.
  • Cold-start items become reachable: constraining decoding to the cold-start item set improves Recall@1 from 0% to 1.24–4.39% across the tested datasets, suggesting the model can rank within a set it was never trained on.
  • The kernel design is portable across JAX/XLA and PyTorch/Inductor environments, so the same static graph approach can be adopted on both TPU and GPU serving stacks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The reported 0.033 ms/step excludes the cost of rebuilding the CSR matrix and recompiling the kernel when the constraint set changes; for fast-moving inventories or rolling freshness windows, the practical per-update cost could dominate unless dynamic sparse updates are added.
  • The near-constant latency with respect to vocabulary size depends on the empirical clustering of Semantic IDs making later trie levels sparse; a constraint set crafted with uniform-random deep prefixes could push the maximum branch factor up and expose the O(B_t) slice cost in VNTK.
  • The dense first-two-layers strategy assumes |V|^2 memory is affordable; for vocabularies much larger than 2048 tokens, the dense mask alone could become a memory bottleneck, suggesting a need for hierarchical or sharded constraint storage.
  • The cold-start result hints that constrained decoding can act as a general 'output-space filter' for zero-shot slicing of a corpus—bypassing retraining for regional, safety, or availability restrictions—which could be tested directly on other recommendation benchmarks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces STATIC, a constrained decoding scheme for LLM-based generative retrieval on accelerators. The authors flatten the prefix trie of admissible Semantic IDs into a static CSR transition matrix and, for the first two levels, a dense mask. Decoding-time constraint enforcement is performed by a branch-free Vectorized Node Transition Kernel (VNTK) that dynamically slices a fixed number B_t of children per node and projects them into a dense log-probability mask. Claims include O(1) I/O complexity with respect to constraint-set size, a measured per-step overhead of 0.033 ms (0.25% of inference) for a 20M-item fresh-video constraint on TPU v6e, a 948x speedup over a CPU trie, 47-1033x speedups over on-device binary-search baselines, positive YouTube A/B metrics, and improved Amazon cold-start Recall@1.

Significance. If the results hold, STATIC would be a practically important systems contribution: it makes strict constrained decoding feasible at production scale on TPUs/GPUs, a gap in existing generative-retrieval deployments. The paper has real strengths: the code is released; latency measurements are reported as means over 100 trials with standard deviations; the online A/B metrics include 95% confidence intervals; and the comparison baselines (unconstrained, CPU trie, PPV) are independent and meaningful. The cold-start experiments are a secondary application but add value. The central weakness is the precision and generality of the O(1) complexity claim, which is not supported by the algorithm as written or by the reported scaling experiments.

major comments (3)
  1. [§4.4, Algorithm 2, Appendix D] The claim of O(1) I/O with respect to constraint-set size is not supported. Algorithm 2's Phase 2 (lines 5-6) performs DynamicSlice of length B_t per node, so per-step I/O is O(B_t), and Appendix D (Figure 4) explicitly shows O(B) scaling. The argument in §4.4 that min(|V|^ℓ, |C|) keeps B_t low bounds the total number of prefixes, not the maximum number of children of a single node; a clustered SID set can place up to min(|V|,|C|) children at one node. RQ-VAE SIDs are explicitly designed to share prefixes (§3.1), so clustering is the realistic regime. No B_t statistics are reported for the production 20M-item trie, and the scalability benchmarks in Figures 2-3 use uniformly random constraint sets, which suppress clustering. Moreover, §5.3 states that 'STATIC's time complexity scales logarithmically with respect to constraint set size,' contradicting the O(1) claim. The paper should eithe
  2. [§4.2, §5.4, §7] The method assumes a static constraint set: the CSR matrix is constructed offline, dense first-layer masks are precomputed, and dynamic sparse updates are listed only as future work. The YouTube 'Last 7 Days' freshness set changes every day, yet the reported 0.033 ms/step overhead excludes the offline rebuild/recompilation cost. The paper should specify how often the index is rebuilt in production, what that rebuild costs (time, TPU/CPU resources, serving interruption), and whether the measurement in Table 1 includes any amortized update. Without these details, the applicability of the headline overhead to freshness constraints is incomplete.
  3. [Abstract, Table 1] The '47-1033x speedup over hardware-accelerated binary-search baselines' conflates exact and approximate baselines. PPV Exact is 1033x slower than STATIC, but PPV Approximate verifies only the top-50 logits and is therefore not directly comparable; Hash Bitmap also has a non-negligible false-positive rate. The paper does disclose this in §5.2, but the abstract and contributions should not present the approximate baselines as exact competitors. Please separate 'exact' comparisons (CPU Trie, PPV Exact) from 'approximate/reference' ones and state the speedup range accordingly.
minor comments (5)
  1. [Appendix D] The benchmark sets |V|=B and then calls B the max branch factor, but with uniformly random SIDs and |C|=10^6 the actual maximum branch factor of the generated trie will not equal B for all B (e.g., when B > |C|, or at deeper levels where prefix counts shrink). Please report the measured max branch factor of the synthesized tries and how it was controlled.
  2. [§4.4] The sentence 'for sufficiently large |V| and practical ranges of |C|, the max branch factor will actually remain quite low' is an empirical heuristic, not a mathematical consequence of the stated bounds. Please phrase it as an assumption and cite the production B_t statistics requested in the major comments.
  3. [Table 3] The cold-start results report Recall@1 without confidence intervals or repeated seeds. Given the small absolute values (e.g., 1.24% vs 0.27%), it would be helpful to state how many test sequences were used and to provide variance estimates, even if the pattern is qualitatively clear.
  4. [Figures 2-3] The figures omit error bars even though Table 4 provides standard deviations. Adding shaded regions or error bars would make the 'almost constant' claim easier to evaluate visually.
  5. [Appendix B] The memory formula's dense-mask term mixes bits and bytes and uses K2 for 'state IDs'; please clarify whether K2 accounts for both the packed mask and the state tensor, and note any alignment or padding overhead. This will make the 90 MB per million items rule easier for practitioners to adapt.

Circularity Check

0 steps flagged

No circularity: the method is a measured engineering construction, not a fitted prediction or self-citation chain.

full rationale

The paper's central derivation is a data-structure transformation: a trie over a constrained Semantic ID set is flattened into a CSR transition matrix, and decoding validity is checked by vectorized slice/project operations. This transformation is definitionally faithful to the trie (Algorithm 2 returns exactly the trie's transitions), and no parameter is fitted to a target result or renamed as a prediction. The latency and scalability claims are supported by direct measurements against independent baselines (unconstrained beam search, CPU trie, PPV exact/approximate, hash bitmap), and the online A/B metrics are empirical outcomes, not outputs derived from the method's assumptions. The self-citations (e.g., PLUM [8], TIGER [24]) are used as sources for the underlying generative retrieval model and Semantic ID construction; they are not invoked as evidence for the correctness of STATIC's vectorized decoding, and the method does not reduce to those citations. The main weakness identified by the skeptic — that the O(1) I/O claim depends on small maximum branch factors and Appendix D shows linear O(B) scaling — is a robustness/generalization caveat, not circularity: the paper itself acknowledges the linear scaling regime in Appendix D, and this does not make the claimed results equivalent to the inputs by construction. No self-definitional step, fitted-input-called-prediction step, or load-bearing self-citation chain is present.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central claims rest on several empirical/domain assumptions about the structure of Semantic IDs, XLA-compatible hardware, and the size distribution of branch factors; none are fitted to make the headline result appear, but the O(1) I/O wording overstates what the kernel actually does.

free parameters (3)
  • dense layer cutoff d = 2
    Chosen by hand to balance dense-mask memory (|V|^d) against sparse-kernel latency; the YouTube deployment uses d=2 and the conclusions about latency depend on it.
  • storage per CSR node K1 = 12 bytes
    Implementation constant used in the memory bound (Appendix B); assumes 4 bytes each for row pointers/columns/values or equivalent.
  • storage per dense state K2 = 4 bytes
    Implementation constant in Appendix B for dense state IDs.
axioms (5)
  • domain assumption The constrained vocabulary C is a fixed, known set of Semantic IDs; the CSR trie is built offline and does not change during inference.
    Section 4.2 constructs the matrix offline; Future Work says dynamic sparse updates are needed for real-time inventory changes. If C changes, latency/memory claims do not transfer.
  • domain assumption Semantic IDs are fixed-length sequences over a common token vocabulary, and any prefix of a valid item is itself a valid constraint state.
    Section 3.1/3.3: constraints are enforced as prefix inclusion; the whole trie representation depends on this.
  • standard math Masking invalid tokens (setting log-prob to -inf) after computing logits exactly implements conditioning on the constraint set under the model's autoregressive factorization.
    Section 3.3/4.1: P(y_t|y_<t)=0 for invalid tokens; this is standard for autoregressive decoding.
  • domain assumption XLA-style compilers cannot efficiently compile data-dependent control flow (pointer chasing), so a branch-free static graph kernel is required for low latency.
    Section 2.3/4.4 and Appendix A.1: the motivation depends on this hardware/compiler behavior.
  • domain assumption The maximum branch factor B_t is small enough at later levels (after dense early layers) that the sparse kernel's O(B_t) work stays negligible.
    Section 5.3/Appendix D: the practical latency claim relies on this empirical property of large constraint tries; not proven in general.

pith-pipeline@v1.3.0-alltime-deepseek · 22115 in / 12907 out tokens · 124542 ms · 2026-08-02T20:37:42.104212+00:00 · methodology

0 comments
read the original abstract

Generative retrieval has emerged as a powerful paradigm for LLM-based recommendation. However, industrial recommender systems often benefit from restricting the output space to a constrained subset of items based on business logic (e.g. enforcing content freshness or product category), which standard autoregressive decoding cannot natively support. Moreover, existing constrained decoding methods that make use of prefix trees (Tries) incur severe latency penalties on hardware accelerators (TPUs/GPUs). In this work, we introduce STATIC (Sparse Transition Matrix-Accelerated Trie Index for Constrained Decoding), an efficient and scalable constrained decoding technique designed specifically for high-throughput LLM-based generative retrieval on TPUs/GPUs. By flattening the prefix tree into a static Compressed Sparse Row (CSR) matrix, we transform irregular tree traversals into fully vectorized sparse matrix operations, unlocking massive efficiency gains on hardware accelerators. We deploy STATIC on a large-scale industrial video recommendation platform serving billions of users. STATIC produces significant product metric impact with minimal latency overhead (0.033 ms per step and 0.25% of inference time), achieving a 948x speedup over a CPU trie implementation and a 47-1033x speedup over a hardware-accelerated binary-search baseline. Furthermore, the runtime overhead of STATIC remains extremely low across a wide range of practical configurations. To the best of our knowledge, STATIC enables the first production-scale deployment of strictly constrained generative retrieval. In addition, evaluation on academic benchmarks demonstrates that STATIC can considerably improve cold-start performance for generative retrieval. Our code is available at https://github.com/youtube/static-constraint-decoding.

Figures

Figures reproduced from arXiv: 2602.22647 by Ed Chi, Isay Katsman, Lichan Hong, Lukasz Heldt, Mingyan Gao, Ningren Han, Onkar Dalal, Raghunandan Keshavan, Ruining He, Shao-Chuan Wang, Xinyang Yi, Yueqi Wang, Zhengyang Su.

Figure 1
Figure 1. Figure 1: This figure showcases the full STATIC pipeline. Figures 1a and 1b present the prefix tree construction for the case [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Scaling of different constraint decoding methods [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Scaling of different constraint decoding methods [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Scaling of the STATIC masking kernel with respect [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. LLMs Need Encoders for Semantic IDs Too

    cs.IR 2026-05 unverdicted novelty 7.0

    PrefixMem encoder for Semantic IDs improves deepest-level accuracy by up to 46% relative and full-SID retrieval recall by up to 22% relative on Pinterest data across LLM families.

  2. Beyond Fixed Depths and Widths: Optimizing Textual Decoding Tries in LLM-based Generative Recommendation

    cs.IR 2026-07 conditional novelty 6.0

    BONSAI constructs variable-depth, low-branching decoding tries for LLM-based generative recommendation and reports 16–22% relative gains over state-of-the-art baselines.

  3. UniPinRec: Unifying Generative Retrieval and Ranking at Pinterest Scale

    cs.IR 2026-05 unverdicted novelty 6.0

    UniPinRec unifies retrieval and ranking into a single model and pipeline deployed at Pinterest, reporting +1% engagement lift, 11.1% lower latency, and 63.6% higher QPS.

  4. CapsID: Soft-Routed Variable-Length Semantic IDs for Generative Recommendation

    cs.IR 2026-05 unverdicted novelty 6.0

    CapsID uses probabilistic capsule routing and confidence-based termination to generate variable-length semantic IDs, improving recall by 9.6% over strong baselines with half the latency of dual-representation systems.

  5. LatticeBridge: Rare-Event Sequential Inference for Faithful Structured Sequence Synthesis

    cs.CL 2026-04 conditional novelty 6.0

    A twisted sequential Monte Carlo decoder with surface automata improves exact constraint satisfaction in structured text generation over standard decoding baselines across three benchmarks.

Reference graph

Works this paper leans on

37 extracted references · 10 linked inside Pith · cited by 5 Pith papers

  1. [1]

    Burton H. Bloom. 1970. Space/time trade-offs in hash coding with allowable errors.Commun. ACM13 (1970), 422–426. https://api.semanticscholar.org/ CorpusID:7931252

    1970

  2. [2]

    2018.JAX: composable transformations of Python+NumPy programs

    James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. 2018.JAX: composable transformations of Python+NumPy programs. http://github.com/jax-ml/jax

    2018

  3. [3]

    Adams, and Emre Sargin

    Paul Covington, Jay K. Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations.Proceedings of the 10th ACM Conference on Rec- ommender Systems(2016). https://api.semanticscholar.org/CorpusID:207240067

    2016

  4. [4]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

  5. [5]

    Jiaxin Deng, Shiyao Wang, Kuo Cai, Lejian Ren, Qigen Hu, Weifeng Ding, Qiang Luo, and Guorui Zhou. 2025. OneRec: Unifying Retrieve and Rank with Genera- tive Recommender and Iterative Preference Alignment.ArXivabs/2502.18965 (2025). https://api.semanticscholar.org/CorpusID:276617997

    Pith/arXiv arXiv 2025

  6. [6]

    Markus Freitag and Yaser Al-Onaizan. 2017. Beam Search Strategies for Neural Machine Translation. InNMT@ACL. https://api.semanticscholar.org/CorpusID: 2229477

    2017

  7. [7]

    Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar

    Ruiqi Guo, Philip Sun, Erik M. Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. 2019. Accelerating Large-Scale Inference with Anisotropic Vector Quantization. InInternational Conference on Machine Learning. https: //api.semanticscholar.org/CorpusID:218614141

    2019

  8. [8]

    Ruining He, Lukasz Heldt, Lichan Hong, Raghu Keshavan, Shifan Mao, Nikhil Mehta, Zhengyang Su, Alicia Y. Tsai, Yueqi Wang, Shao-Chuan Wang, Xinyang Yi, Lexi Baugher, Baykal Cakici, Ed Huai hsin Chi, Cristos Goodrow, Ningren Han, He Ma, Romer Rosales, Abby Van Soest, Devansh Tandon, Su-Lin Wu, Weilong Yang, and Yilin Zheng. 2025. PLUM: Adapting Pre-trained...

    arXiv 2025

  9. [9]

    Ruining He and Julian McAuley. 2016. Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering.Proceedings of the 25th International Conference on World Wide Web(2016). https://api. semanticscholar.org/CorpusID:1964279

    2016

  10. [10]

    2024.Flax: A neural network library and ecosystem for JAX

    Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Ronde- pierre, Andreas Steiner, and Marc van Zee. 2024.Flax: A neural network library and ecosystem for JAX. http://github.com/google/flax

    2024

  11. [11]

    Jia-Wei Hong and H. T. Kung. 1981. I/O complexity: The red-blue pebble game. InSymposium on the Theory of Computing. https://api.semanticscholar.org/ CorpusID:8410593

    1981

  12. [12]

    Jun ichi Aoe. 1989. An Efficient Digital Search Algorithm by Using a Double- Array Structure.IEEE Trans. Software Eng.15 (1989), 1066–1077. https://api. semanticscholar.org/CorpusID:267894119

    1989

  13. [13]

    Jouppi, Cliff Young, Nishant Patil, David A

    Norman P. Jouppi, Cliff Young, Nishant Patil, David A. Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Taraneh Ghaemmaghami, Rajendra Gottipati, William Gulland, Robert Hagmann, C. Richard ...

    2017

  14. [14]

    Clark Mingxuan Ju, Liam Collins, Leonardo Neves, Bhuvesh Kumar, Louis Yufeng Wang, Tong Zhao, and Neil Shah. 2025. Generative Recommendation with Seman- tic IDs: A Practitioner’s Handbook. InProceedings of the 34th ACM International Conference on Information and Knowledge Management (CIKM ’25). 6420–6425. doi:10.1145/3746252.3761612

    arXiv 2025

  15. [15]

    Su et al

    Gemma Team Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram’e, Mor- gane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean-Bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gael Liu, Francesco Visin, Kathleen Kenealy, Lucas B...

    Pith/arXiv arXiv 2025

  16. [16]

    Bader, Aydın Buluç, Franz Franchetti, John R

    Jeremy Kepner, Peter Aaltonen, David A. Bader, Aydın Buluç, Franz Franchetti, John R. Gilbert, Dylan Hutchison, Manoj Kumar, Andrew Lumsdaine, Henning Meyerhenke, Scott McMillan, Carl Yang, John Douglas Owens, Marcin Zalewski, Timothy G. Mattson, and José E. Moreira. 2016. Mathematical foundations of the GraphBLAS.2016 IEEE High Performance Extreme Comput...

    2016

  17. [17]

    Terry Koo, Frederick Liu, and Luheng He. 2024. Automata-based constraints for language model decoding.ArXivabs/2407.08103 (2024). https://api. semanticscholar.org/CorpusID:271097802

    Pith/arXiv arXiv 2024

  18. [18]

    Mingzhen Li, Yi Liu, Xiaoyan Liu, Qingxiao Sun, Xin You, Hailong Yang, Zhongzhi Luan, and Depei Qian. 2020. The Deep Learning Compiler: A Comprehensive Survey.IEEE Transactions on Parallel and Distributed Systems32 (2020), 708–727. https://api.semanticscholar.org/CorpusID:211069666

    2020

  19. [19]

    Hao Liao, Jiwei Zhang, Jianxun Lian, Wensheng Lu, Mingqi Wu, Shuo Wang, Yong Zhang, Yitian Huang, Mingyang Zhou, and Rui Mao. 2025. Eliminating Out- of-Domain Recommendations in LLM-based Recommender Systems: A Unified View. https://api.semanticscholar.org/CorpusID:278339501

    2025

  20. [20]

    Greg Linden, Brent Smith, and Jeremy York. 2003. Amazon.com Recommenda- tions: Item-to-Item Collaborative Filtering.IEEE Internet Comput.7 (2003), 76–80. https://api.semanticscholar.org/CorpusID:263872610

    2003

  21. [21]

    Zhanyun Liu, Shiyao Wang, Xing-Yao Wang, Rong-Qing Zhang, Jiaxin Deng, Honghui Bao, Jinghao Zhang, Wuchao Li, Pengfei Zheng, Xiangyu Wu, Yifei Hu, Qigen Hu, Xinchen Luo, Lejian Ren, Zixing Zhang, Qianqian Wang, Kuo Cai, Yun-Jie Wu, Hongtao Cheng, Zexuan Cheng, Lu Ren, Huanjie Wang, Yi Su, Ruiming Tang, Kun Gai, and Guorui Zhou. 2025. OneRec-Think: In-Text...

    arXiv 2025

  22. [22]

    Smith, and Yejin Choi

    Ximing Lu, Sean Welleck, Peter West, Liwei Jiang, Jungo Kasai, Daniel Khashabi, Ronan Le Bras, Lianhui Qin, Youngjae Yu, Rowan Zellers, Noah A. Smith, and Yejin Choi. 2021. NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics. InNorth American Chapter of the Association for Computational Linguistics. https://api.semanticschol...

    2021

  23. [23]

    Gabriel Poesia, Oleksandr Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable code generation from pre-trained language models.ArXivabs/2201.11227 (2022). https://api.semanticscholar.org/CorpusID:246294475

    Pith/arXiv arXiv 2022

  24. [24]

    Keshavan, Trung Hieu Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Q

    Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan H. Keshavan, Trung Hieu Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Q. Tran, Jonah Samost, Maciej Kula, Ed H. Chi, and Maheswaran Sathiamoorthy. 2023. Rec- ommender Systems with Generative Retrieval.ArXivabs/2305.05065 (2023). https://api.semanticscholar.org/CorpusID:258564854

    Pith/arXiv arXiv 2023

  25. [25]

    Amit Sabne. 2020. Xla: Compiling machine learning for peak performance

    2020

  26. [26]

    Zihua Si, ZhongXiang Sun, Jiale Chen, Guozhang Chen, Xiaoxue Zang, Kai Zheng, Yang Song, Xiao Zhang, and Jun Xu. 2023. Generative Retrieval with Semantic Tree-Structured Identifiers and Contrastive Learning.Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region(2023...

    2023

  27. [27]

    Philip Sun, David Simcha, Dave Dopson, Ruiqi Guo, and Sanjiv Kumar. 2024. SOAR: Improved Indexing for Approximate Nearest Neighbor Search.ArXiv abs/2404.00774 (2024). https://api.semanticscholar.org/CorpusID:268030651

    Pith/arXiv arXiv 2024

  28. [28]

    Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W

    Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W. Cohen, and Don- ald Metzler. 2022. Transformer Memory as a Differentiable Search Index.ArXiv abs/2202.06991 (2022). https://api.semanticscholar.org/CorpusID:246863488

    Pith/arXiv arXiv 2022

  29. [29]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. InNeural Information Processing Systems. https://api.semanticscholar. org/CorpusID:13756489

    2017

  30. [30]

    Orion Weller, Michael Boratko, Iftekhar Naim, and Jinhyuk Lee. 2026. On the Theoretical Limitations of Embedding-Based Retrieval. InThe Fourteenth Inter- national Conference on Learning Representations. https://openreview.net/forum? id=k9CzIvzfaA

    2026

  31. [31]

    Liu Yang, Fabian Paischer, Kaveh Hassani, Jiacheng Li, Shuai Shao, Zhang Gabriel Li, Yun He, Xue Feng, Nima Noorshams, Sem Park, Bo Long, Robert Nowak, Xiaoli Gao, and Hamid Eghbalzadeh. 2024. Unifying Generative and Dense Retrieval for Sequential Recommendation.Trans. Mach. Learn. Res.2025 (2024). https://api.semanticscholar.org/CorpusID:274423313

    2024

  32. [32]

    Haotian Ye, Himanshu Jain, Chong You, Ananda Theertha Suresh, Haowei Lin, James Zou, and Felix X. Yu. 2025. Efficient and Asymptotically Unbiased Con- strained Decoding for Large Language Models.ArXivabs/2504.09135 (2025). https://api.semanticscholar.org/CorpusID:277781124

    Pith/arXiv arXiv 2025

  33. [33]

    Guorui Zhou, Honghui Bao, Jiaming Huang, Jiaxin Deng, Jinghao Zhang, Junda She, Kuo Cai, Lejian Ren, Lu Ren, Qiang Luo, Qianqian Wang, Qigen Hu, Rongzhou Zhang, Ruiming Tang, Shiyao Wang, Wuchao Li, Xiangyu Wu, Xinchen Luo, Xingmei Wang, Yifei Hu, Yunfan Wu, Zhanyu Liu, Zhiyang Zhang, Zixing Zhang, Bo Chen, Bin Wen, Chaoyi Ma, Chengru Song, Chen- glong Ch...

    arXiv 2026

  34. [34]

    Guorui Zhou, Jiaxin Deng, Jinghao Zhang, Kuo Cai, Lejian Ren, Qiang Luo, Qianqian Wang, Qigen Hu, Rui Huang, Shiyao Wang, Weifeng Ding, Wuchao Li, Xinchen Luo, Xing-Yao Wang, Zexuan Cheng, Zixing Zhang, Bin Zhang, Bo-Wen Wang, Chao Ma, Cheng bin Song, Chenhui Wang, Di Wang, Dongxue Meng, Fan Yang, Fang-Peng Zhang, Feng Jiang, Fuxing Zhang, Gang Wang, Guow...

    arXiv 2025

  35. [35]

    ConcretizationTypeError

    Guorui Zhou, Hengrui Hu, Hongtao Cheng, Huanjie Wang, Jiaxin Deng, Jinghao Zhang, Kuo Cai, Lejian Ren, Lu Ren, Liao Yu, Pengfei Zheng, Qiang Luo, Qianqian Wang, Qigen Hu, Rui Huang, Ruiming Tang, Shiyao Wang, Shu-Jun Yang, Tao Wu, Wuchao Li, Xin-Jing LuO, Xing-Yao Wang, Yi Su, Yun-Jie Wu, Zexuan Cheng, Zhanyun Liu, Zixing Zhang, Bin Zhang, Bo-Long Wang, C...

    Pith/arXiv arXiv 2025

  36. [37]

    ""Implementation of Hardware-Accelerated Constrained Decoding algorithms

    Our main configuration variables are fixed to|V|= 2048,𝐿= 8, 𝐾1 = 12, 𝐾2 = 4, and 𝑑= 2. Note since we have20million constraints in the restricted vocabulary, we also have|C|= 20· 106. Now we compute the total memory usage (1)Dense Mask Phase (ℓ=1,2): • The number of bytes we need to store the dense mask is simply 1 8+4 · 20482 = 17301504bytes, or ≈ 17.3 M...

    2048

  37. [2022]

    https://api.semanticscholar.org/ CorpusID:249151871

    FlashAttention: Fast and Memory-Efficient Exact Attention with IO- Awareness.ArXivabs/2205.14135 (2022). https://api.semanticscholar.org/ CorpusID:249151871

    Pith/arXiv arXiv 2022