Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

TrajTok: Technical Report for 2025 Waymo Open Sim Agents Challenge

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TrajTok combines rule-based grid coverage with data filtering to improve realism of next-token traffic simulation, reaching a 0.7852 Realism Meta score and second place in the 2025 Waymo Open Sim Agents Challenge.

desk verdict TrajTok is a genuinely new hybrid tokenizer with a plausible but under-ablated leaderboard claim; the +0.0038 realism gain is not cleanly attributable to the tokenizer. read the letter →

arxiv 2506.21618 v1 pith:WDHQWN4I submitted 2025-06-23 cs.CL cs.AI

Zhiyuan Zhang , Xiaosong Jia , Guanyu Chen , Qifeng Li , Junchi Yan This is my paper · ORCID
classification cs.CLcs.AI
keywords TrajToktrajectorytokenizationnext-tokenpredictionbehaviorgenerationautonomousdrivingsimulationspatial-awarelabelsmoothingcoverageandsymmetryk-diskstokenizer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TrajTok is a trajectory tokenizer for next-token-prediction behavior models that builds its vocabulary by dividing trajectory space into a grid, keeping cells that real 0.5-second trajectories pass through, and expanding those cells to cover nearby trajectories that do not appear in the training data. The paper argues that this hybrid rule-plus-data design gives better coverage, symmetry, and robustness than the purely data-driven k-disks tokenizer, because the tokens are not forced to lie inside the dataset's sampled endpoint distribution. It also replaces standard label smoothing with a spatial-aware version that spreads probability mass over tokens according to the inverse squared distance to the ground-truth trajectory. Adopted in the SMART model and submitted to the 2025 Waymo Open Sim Agents Challenge, TrajTok reaches a Realism Meta score of 0.7852 and ranks second, improving the stated baseline by +0.0038. A sympathetic reader would care because discrete tokenizers are the bottleneck between continuous driving behavior and language-model-style generation, and this design targets rare but physically plausible trajectories that data-driven tokenizers miss.

What carries the argument

The central object is the TrajTok vocabulary, built by a four-step pipeline: (1) flipping trajectories along the x-axis for symmetry; (2) gridding the agent-centric coordinate plane with per-agent-type ranges and intervals; (3) filtering a binary occupancy map by counting trajectory endpoints in a $(k\times k)$ neighborhood, discarding low-count noisy cells and expanding coverage to dense empty cells; (4) generating one token per valid cell, either as the mean of the real trajectories in that cell or, for expanded cells, as a curve interpolation from the origin to the cell center with yaw taken from nearby cells. The other mechanism is spatial-aware label smoothing, where the target probability of a non-ground-truth token $i$ is proportional to $k_i = 1/(\|c_i - c_j\|^2 + \varepsilon_1)$, so the model tolerates spatially close trajectory errors and rejects distant ones. Together they carry the claim that tokenizer coverage, symmetry, and robustness are what move the realism metric.

What would settle it

Retrain SMART-tiny twice with identical heads, training data, and loss, changing only the tokenizer (k-disks vs TrajTok at matched vocabulary sizes), and compare Realism Meta; if the difference disappears or reverses, the central claim that the tokenizer drives the gain is falsified.

Watch

Extended reading notes

Core claim

The paper claims that a good trajectory tokenizer for next-token traffic generation should not merely fit the training distribution; it should cover the space of plausible trajectories and be robust to noise and to left/right symmetry. TrajTok operationalizes this by flipping every extracted trajectory to enforce x-axis symmetry, gridding the agent-centric plane with agent-specific ranges and intervals, filtering out grid cells whose endpoint counts are too low to be real, and expanding valid cells so that token endpoints can cover trajectories absent from the dataset. For expanded cells it synthesizes a token by interpolating a curve from the origin to the cell center, using yaw estimated from nearby real data. Combined with spatial-aware label smoothing, which assigns non-ground-truth probability inversely proportional to squared trajectory error, this tokenizer is claimed to be the reason the SMART model reaches a realism score of 0.7852 and second place in the 2025 challenge, with state-of-the-art Map-based metrics of 0.9207.

Load-bearing premise

The claim that TrajTok causes the +0.0038 realism improvement assumes that the simultaneous switch to separate per-agent prediction heads in the SMART model did not contribute most of the gain, since no experiment changes the tokenizer alone at the final setting.

Editorial extensions

If this is right

  • Tokenizers for next-token-prediction behavior models do not have to stay inside the data distribution: expansion by rule lets the vocabulary represent trajectories that occur in reality but are rare or absent in the training set.
  • Symmetric tokens prevent the model from overfitting to one side of the road or one traffic-rule convention, which should help generalization across left- and right-hand traffic.
  • Filtering low-density endpoint cells removes noisy or physically implausible samples before they become tokens, reducing the influence of dataset noise on training.
  • Vocabulary size needs to be tuned per tokenizer: TrajTok's optimum is roughly 8000 tokens for vehicles and 2000–3000 for pedestrians and bicycles, larger than k-disks' around 2048, without the degradation k-disks suffers at larger sizes.
  • Reaching a Realism Meta of 0.7852 and second place on the 2025 leaderboard shows the hybrid tokenizer is competitive with methods that require closed-loop fine-tuning, without any fine-tuning step.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the tokenizer effect were isolated from the separate-head change, a plausible prediction is that TrajTok would show its largest gains on distribution-shift or rare-scenario splits rather than on the aggregate Realism Meta, because its coverage and robustness mechanisms target exactly those cases.
  • Spatial-aware label smoothing is not specific to trajectories: any vocabulary with a continuous geometry, such as keypoint or action embeddings, could reuse the inverse-error-squared target distribution to encode locality in the loss.
  • A clean testable extension is to plug TrajTok into a frozen shared-head SMART model and compare against k-disks with the same training budget; if the gap shrinks or vanishes, the separate per-agent heads, not the tokenizer, would be the cause of the leaderboard gain.
  • TrajTok's x-axis flipping encodes symmetry as a hard constraint; a softer variant that also flips yaw or augments left/right turns could expose whether hard symmetry ever hurts on asymmetric driving distributions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes TrajTok, a trajectory tokenizer for discrete next-token-prediction behavior generation models, combining rule-based grid generation with data-driven filtering and expansion. It also introduces a spatial-aware label smoothing for cross-entropy loss that distributes non-ground-truth probability mass inversely proportional to the spatial error between token trajectories. The authors integrate these components into the SMART model, replacing the k-disks tokenizer and using separate per-agent-type classification heads, and report a Realism Meta score of 0.7852 (2nd place) in the 2025 Waymo Open Sim Agents Challenge, a +0.0038 improvement over the SMARTtopk32 baseline. A smaller-scale comparison (Table 3) is provided for tokenizer vocabulary sizes under the 2024 metric.

Significance. TrajTok is a sensible and clearly specified alternative to purely data-driven trajectory tokenizers. The idea of using dataset statistics only to filter and expand a rule-generated grid is interesting and potentially valuable; it directly addresses coverage, symmetry, and robustness concerns that are qualitatively demonstrated in Figure 1. The spatial-aware label smoothing is a natural adaptation of a standard technique to a continuous spatial vocabulary. The paper also provides useful hyperparameter details and a tokenizer-size comparison. If the confounds described in the major comments are resolved, the claims would be of solid interest to the traffic-simulation and behavior-generation community. However, as written, the paper does not yet establish that the tokenizer—rather than the simultaneous change in prediction heads and loss smoothing—is responsible for the reported leaderboard gain.

major comments (3)
  1. [Section 3.1-3.2, Table 2] The comparison between TrajTok (0.7852) and SMARTtopk32 (0.7814) changes two components at once: the tokenizer (k-disks to TrajTok) and the classification head (shared head to separate per-agent heads, as stated in Section 3.1). Consequently, the +0.0038 Realism Meta difference cannot be attributed to the tokenizer alone. Please provide an ablation that isolates the tokenizer by holding the head architecture fixed, e.g., TrajTok with the original shared head or k-disks with separate per-agent heads.
  2. [Section 2.3 and Table 3] The spatial-aware label smoothing is never ablated. Table 3 compares k-disks and TrajTok on a training subset, but the text does not state whether both tokenizers are evaluated with the same label smoothing and the same prediction-head configuration. If the TrajTok rows use spatial-aware smoothing and the k-disks rows use standard smoothing, the comparison in Table 3 is also confounded. Please either report ablations (standard vs. spatial-aware smoothing, crossing both tokenizers) or explicitly state that the smoothing was held fixed in Table 3 and in all other comparisons.
  3. [Section 3.2 / Table 2] The reported +0.0038 improvement is based on a single leaderboard submission with no error bars or multiple seeds. Given the magnitude of the difference, the authors should either provide variance estimates from repeated runs or temper the claim about the size of the improvement.
minor comments (6)
  1. [Section 2.1] The word "Donate" appears three times and should be "Denote" in all occurrences.
  2. [Section 2.3, Eq. (10)] The sum in the denominator of Eq. (10) is written with index range starting at m=0; since token labels presumably run from 1 to |V|, please clarify the intended index range and that the sum excludes m=j.
  3. [Table 3 caption] The caption says "Token Size" with columns "Veh Ped Cyc", which is ambiguous: please clarify that these are vocabulary sizes per agent type and explain why the k-disks rows use the same size for all agent types while the TrajTok rows differ.
  4. [Section 3.2] The term "SMARTtopk32" is used in Table 2 but not defined in the text; please clarify its relationship to the "SMART-tiny" model described in Section 3.1.
  5. [Section 1] The introduction claims "there is a lack of comprehensive analysis of trajectory tokenizers," but the paper itself only compares against k-disks; consider softening this claim or adding references to other tokenizer analyses.
  6. [Figure 1] Figure 1 is referenced but not included in the provided text; ensure that the final version is legible and that the caption clearly explains the upper and lower panels.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TrajTok's claims are evaluated on an external leaderboard, and its construction is not defined in terms of the target metric.

full rationale

Sections 2.2 and 2.3 define TrajTok as a vocabulary constructor that fits a grid to WOMD trajectory data and modifies the cross-entropy target by token-space distances. The central claim (abstract, Section 3.2) is a Realism Meta score of 0.7852 on WOSAC 2025, which is computed by the external challenge protocol; no equation in the paper defines Realism Meta in terms of TrajTok's grid range, filtered binary map, label-smoothing kernel, or vocabulary size. The coverage, symmetry, and robustness advantages are stated as design consequences of flipping, filtering, and expansion (Section 2.2 and Section 3.3), not as quantities derived from the evaluation metric. There is no load-bearing self-citation: SMART [5], Trajeglish [3], and CLSFT [6] are independent external works. The one notable weakness is that the submitted run changes the tokenizer and the prediction head simultaneously (Section 3.1: 'We use separate prediction heads instead and set their output dim the same as the vocabulary size each'), so the exact +0.0038 gain cannot be unambiguously assigned to the tokenizer alone. This is an experimental attribution and control issue, not a circular definition, and therefore does not raise the circularity score.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

TrajTok introduces no new physical entities. The method rests on hand-tuned grid and filtering hyperparameters, on an endpoint-based representation of trajectory space, and on an untested assumption that a head architecture change does not drive the reported improvement.

free parameters (4)
  • Grid ranges and intervals per agent type = vehicle: x[-5,20] step 0.1, y[-1.5,4.5] step 0.05; bicycle: x[-1,8] step 0.05, y[-1,1] step 0.05; pedestrian…
    Hand-chosen to cover the dataset's endpoint distribution; they fix vocabulary size and resolution (Table 1).
  • Filtering/expansion kernel size k and density thresholds
    Section 2.2 decides valid cells from values in a k×k neighborhood and from 'few nearby' / 'dense' criteria, but k and the thresholds are never reported; they control noise rejection and coverage.
  • Vocabulary sizes per agent type = vehicle 8040, bicycle 3001, pedestrian 2798
    Selected as optimal via Table 3 scans; the final leaderboard score depends on this tuning.
  • Spatial label smoothing epsilon = 0.1
    Chosen equal to the original model's standard label smoothing (Section 3.1); the inverse-distance constant in Eq. 9 is not specified.
assumptions (5)
  • domain assumption Endpoint position is a sufficient summary for tokenizing half-second trajectories; yaw can be recovered from neighboring endpoints.
    Section 2.2 grids endpoints and averages trajectories in each cell.
  • domain assumption Mirroring trajectories along the x-axis produces valid symmetric trajectories.
    Section 2.2 step 1; reasonable for vehicles but less so for pedestrians.
  • domain assumption WOMD training data and the WOSAC Realism Meta metric are a reliable evaluation basis.
    Tokenizer and model are built on WOMD; the only evaluation is the external challenge.
  • domain assumption Using separate per-agent prediction heads instead of a shared head does not confound the tokenizer comparison.
    Implicit in Section 3.2 attribution of the gain to TrajTok; never tested.
  • domain assumption Curve interpolation from the origin to a grid center with yaw from nearby tokens produces physically plausible trajectories in empty cells.
    Section 2.2 step 4; no validation of the interpolated tokens is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TrajTok: Technical Report for 2025 Waymo Open Sim Agents Challenge." pith.science (2026). https://pith.science/paper/WDHQWN4I

@misc{pith2026250621618,
  author       = {Pith},
  title        = {Pith review of: TrajTok: Technical Report for 2025 Waymo Open Sim Agents Challenge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WDHQWN4I}},
  note         = {Machine review of arXiv:2506.21618}
}
read the original abstract

In this technical report, we introduce TrajTok, a trajectory tokenizer for discrete next-token-prediction based behavior generation models, which combines data-driven and rule-based methods with better coverage, symmetry and robustness, along with a spatial-aware label smoothing method for cross-entropy loss. We adopt the tokenizer and loss for the SMART model and reach a superior performance with realism score of 0.7852 on the Waymo Open Sim Agents Challenge 2025. We will open-source the code in the future.

Figures

Figures reproduced from arXiv: 2506.21618 by the authors.

Figure 1
Figure 1. Visualization and comparison of k-disks and TrajTok. Each colored line represents a trajectory token within 0.5 seconds in agent-centric coordinate system. In the top right image, orange dots indicate the endpoints of trajectory tokens, while blue dots indicate the endpoints of real trajectories in the dataset. Compared to k-disks, TrajTok generates trajectories with better symmetry, coverage, and robustness to nois… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Agent-driven Long-tail Simulation for Autonomous Driving

    cs.RO 2026-07 conditional novelty 5.0 of 10

    LLM agents with structured actions can drive interactive long-tail road users in nuPlan, and SemanticPlan shows current planners still fail safety and semantic completion there.

Reference graph

Works this paper leans on

8 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Revisit mixture models for multi-agent simulation: Experimental study within a unified framework, 2025

    Longzhong Lin, Xuewu Lin, Kechun Xu, Haojian Lu, Lichao Huang, Rong Xiong, and Yue Wang. Revisit mixture models for multi-agent simulation: Experimental study within a unified framework, 2025

    work page 2025

  3. [3]

    The waymo open sim agents challenge

    Nico Montali, John Lambert, Paul Mougin, Alex Kuefler, Nicholas Rhinehart, Michelle Li, Cole Gulino, Tristan Emrich, Zoey Yang, Shimon Whiteson, et al. The waymo open sim agents challenge. Advances in Neural Information Processing Systems, 36: 0 59151--59171, 2023

    work page 2023

  4. [4]

    Trajeglish: Traffic modeling as next-token prediction

    Jonah Philion, Xue Bin Peng, and Sanja Fidler. Trajeglish: Traffic modeling as next-token prediction. arXiv preprint arXiv:2312.04535, 2023

    arXiv 2023

  5. [5]

    Motionlm: Multi-agent motion forecasting as language modeling

    Ari Seff, Brian Cera, Dian Chen, Mason Ng, Aurick Zhou, Nigamaa Nayakanti, Khaled S Refaat, Rami Al-Rfou, and Benjamin Sapp. Motionlm: Multi-agent motion forecasting as language modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8579--8590, 2023

    work page 2023

  6. [6]

    Smart: Scalable multi-agent real-time motion generation via next-token prediction

    Wei Wu, Xiaoxin Feng, Ziyan Gao, and Yuheng Kan. Smart: Scalable multi-agent real-time motion generation via next-token prediction. Advances in Neural Information Processing Systems, 37: 0 114048--114071, 2024

    work page 2024

  7. [7]

    Closed-loop supervised fine-tuning of tokenized traffic models

    Zhejun Zhang, Peter Karkus, Maximilian Igl, Wenhao Ding, Yuxiao Chen, Boris Ivanovic, and Marco Pavone. Closed-loop supervised fine-tuning of tokenized traffic models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    work page 2025

  8. [8]

    Kigras: Kinematic-driven generative model for realistic agent simulation

    Jianbo Zhao, Jiaheng Zhuang, Qibin Zhou, Taiyu Ban, Ziyao Xu, Hangning Zhou, Junhe Wang, Guoan Wang, Zhiheng Li, and Bin Li. Kigras: Kinematic-driven generative model for realistic agent simulation. IEEE Robotics and Automation Letters, 2024

    work page 2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.