REVIEW 4 major objections 4 minor 16 references
Improving Traffic Signal Data Quality for the Waymo Open Motion Dataset
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Vehicle trajectories can reconstruct nearly all missing traffic-signal states in the Waymo Open Motion Dataset, cutting the estimated red-light running rate from 15.7% to 2.9%.
desk verdict Useful pipeline for fixing WOMD signal data, but the headline red-light-running validation is partly circular; still worth serious review. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the pair of vehicle confidence functions $f(d,a)$ and $g(d,v)$: for a vehicle at distance $d$ from the stop bar with acceleration $a$ or velocity $v$, these functions score how strongly its motion evidences a red or green light, with the $d=30$ m cutoff based on FHWA dilemma-zone guidance. Aggregating these scores across vehicles yields a per-movement estimate $\hat{m}_i(t)$ and a confidence $c_i(t)$. The second pillar is the ring-and-barrier feasibility set $\mathcal{X}$—the roughly ten legal green configurations for a four-way intersection—which constrains the final sequence and lets the method resolve conflicts by maximizing a weighted match score and minimizing a weighted conflict score. Temporal filtering enforces a minimum phase duration, and a fixed yellow phase is appended before each red.
What would settle it
Collect independent ground-truth signal states for a sample of WOMD intersections—for example, from controller logs, video, or a separate instrumented vehicle—and compare frame-by-frame with the imputed sequence. The central claim fails if the imputed states disagree with direct observations on a meaningful fraction of frames where the trajectory evidence was sparse, or if the red-light running rate computed against independent ground truth does not approach the 2.9% level.
Extended reading notes
Core claim
The paper's central claim is that a 9-second window of vehicle trajectories, combined with map geometry and standard U.S. signal-timing structure, is enough to reconstruct the full per-movement signal state at an intersection. Each approaching vehicle contributes two confidence scores—one from acceleration and distance to the stop bar, one from velocity and distance—that vote for green or red; the votes are aggregated over a time horizon into an estimated state per movement with a confidence value. Missing states are filled from these estimates, and recorded states are overwritten when the trajectory evidence is confident and disagrees. The candidate states are then restricted to the ring-and-barrier feasible configurations of the intersection, and the configuration maximizing weighted agreement with the imputed states is chosen, with temporal smoothing to avoid implausibly short phases and a fixed yellow inserted before each red. Applied to over 360,000 signalized scenarios in WOMD, the pipeline imputes all 71.7% of missing and unknown states and reduces the estimated red-light running rate from 15.7% to 2.9%.
Load-bearing premise
The load-bearing premise is that near a stop bar most vehicles stop on red and proceed on green, so any disagreement between trajectories and the dataset's signal labels is a data error; if frequent red-light running, queue spillback, or sensor noise produces the same trajectories, the imputed states can be wrong while the reported violation rate still looks low.
Editorial extensions
If this is right
- All 360,000+ signalized scenarios in the Waymo Open Motion Dataset can be released with complete per-lane signal states, removing the largest incompleteness in the dataset's signal channel.
- Models trained on the corrected signals should see a far lower rate of impossible red-light crossings in their training labels, since the paper's estimated violation rate drops from 15.7% to 2.9%.
- The method transfers across intersection geometries and timing plans: simulation accuracy is 96.98%, 97.72%, and 96.08% for two fixed-time plans and one actuated plan.
- The imputation logic extends to movements whose signals were entirely absent from the raw data, so downstream trajectory prediction no longer needs to treat signal state as partially unobserved.
Reading between the lines
- Our reading: because the method needs only trajectories, map geometry, and a NEMA-style signal-structure assumption, the same pipeline should transfer to other motion datasets (for example, the Lyft Level 5 or Argoverse 2 datasets), though the paper does not test this.
- Our reading: the 2.9% residual red-light rate is best treated as an upper bound on real violations under the paper's own assumptions; at intersections with frequent queue spillback or genuine non-compliance, the corrected states could inherit the same bias as the validation metric.
- Our reading: a natural next test is to use the imputed signal states as training targets for a trajectory-conditioned signal-state predictor, and to check whether downstream trajectory-prediction error drops when the corrected labels replace the raw ones.
- Our reading: because the method does not generate flashing signals and assumes standard ring-and-barrier layouts, the paper's own limitations imply that five-leg intersections, separated right-turn lanes, and flashing phases would need a relaxed feasibility set.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a fully automated pipeline to impute and correct traffic signal states in the Waymo Open Motion Dataset. The method identifies signalized intersections from lane geometry, estimates per-movement signal states from vehicle acceleration and velocity near stop bars, and then selects a ring-and-barrier feasible state combination that minimizes conflict with the trajectory-based estimates. The authors validate the approach in SUMO simulation on one intersection with three timing plans (reported accuracy above 96%), qualitatively on three WOMD scenarios, and quantitatively on the entire dataset through a 71.7% imputation rate and a reduction in estimated red-light running scenarios from 15.7% in the original data to 2.9% after correction. The code and improved traffic signal data are open-sourced.
Significance. Improving traffic signal data quality in WOMD is a valuable contribution because the dataset is widely used for prediction, planning, and simulation, and missing or erroneous signal states can degrade downstream models. The open-sourced code and improved data have clear practical value. The simulation experiments provide some evidence that the internal logic works under compliant driver behavior and standard intersection geometry. However, the central real-data claim—that the corrected WOMD signal states are accurate—is not established by the current validation: the headline red-light-violation metric is computed on the same trajectories used to construct the signal states, so it is partly enforced by construction, and the raw and corrected rates are not computed on the same movements. The simulation validation is limited to one intersection and does not address the confounds of queuing, noncompliance, or unusual geometries. If the authors add independent, like-for-like validation on real data, the contribution would be significant; in its current form the accuracy claim is not convincingly supported.
major comments (4)
- [Section 4.3.2, Eqs. (9)-(11)] The red-light violation rate is a circular validation metric. In Section 3.2.2, Eqs. (9)-(11) select the final signal state by minimizing a conflict score between the feasible state and the trajectory-based estimates; the red-light violation rate in Section 4.3.2 then counts red-light conflicts between those same trajectories and the selected states. A low violation rate therefore measures agreement with the objective the method was designed to optimize, not accuracy against any independent ground truth. To support the accuracy claim, the authors should validate against held-out observed signal states from the original WOMD (e.g., the ~28.3% of states that are present), rather than only reporting the trajectory-consistency metric.
- [Section 4.3.2] The comparison between the raw violation rate of 15.7% and the corrected rate of 2.9% is not like-for-like. The raw rate is computed only on movements for which the original data provides a signal state, while the corrected rate is computed on all movements, including the 71.7% that were missing or unknown and therefore could not contribute to the raw violation count. These two numbers have different denominators and different movement sets, so the reduction from 15.7% to 2.9% is not a valid measure of improvement. The authors should compute both rates on the same set of movements (for example, only on movements with known raw states) and should report the definitions explicitly.
- [Section 4.1 and Section 5] The simulation validation is not sufficient to establish real-world accuracy. The SUMO experiment uses a single intersection, trajectories generated by compliant car-following models, and parameters calibrated on the same simulation setup. The authors' own Section 5 concedes that performance may degrade at nonstandard intersections, unusual geometries, or complex lane configurations, yet the abstract and introduction claim robustness and flexibility across diverse real-world geometries. The simulation therefore cannot support the global accuracy claim for the entire WOMD. A more convincing validation would use real intersections with independent ground-truth signal states, or at least a clear sensitivity analysis over intersection types and congestion levels.
- [Section 3.2.1 and Appendix A] The trajectory-based estimation cannot distinguish several important confounds that are not addressed outside simulation. A vehicle stopped near the stop bar on green because of a queue or a blocked intersection is observationally similar in acceleration and velocity to a vehicle stopped on red; similarly, a vehicle moving through on green is observationally similar to one running a red light if the motion is smooth. The confidence functions in Eqs. (21)-(23) use only distance, acceleration, and velocity, with no mechanism to account for queue spillback or stop-line blockage. The authors should either incorporate such mechanisms or explicitly bound the error introduced by these confounds, with supporting real-world evidence.
minor comments (4)
- [Appendix A, Eq. (21)] The piecewise definition of f(d,a) appears to contain conflicting conditions: the first line lists 'd<0, a<0 or a>=0 or d>30' while the third line sets f=1 for d>30. As written, the function is ambiguous for d>30. The intended range likely excludes d>30 from the first condition, but this needs clarification.
- [Abstract and Section 4.3.1] The phrase 'all of which were successfully imputed' is too strong: since there is no ground truth for the missing states on real data, the method assigns a state to every movement, but 'successful' imputation is not demonstrated. The imputation rate metric measures completeness, not correctness.
- [Section 4.3.2] The metric is described as a 'red light violation rate' but is computed as the fraction of scenarios with at least one violation, not the fraction of vehicle crossings that are violations. The authors should use a less ambiguous term such as 'scenario-level red-light violation rate' and clearly distinguish it from a vehicle-level violation rate.
- [Section 4.1, Table 5] The parameter calibration section states that 960 parameter combinations were evaluated on the same three SUMO timing plans used for the accuracy results in Table 4. Reporting accuracy on the calibration set as a validation result overstates generalization; a held-out simulation run or a separate calibration/validation split would be more appropriate.
Circularity Check
The headline real-data validation is circular: the red-light running rate is computed on the same trajectories used to construct the signal states, and the raw-versus-corrected comparison is not like-for-like.
-
fitted input called prediction
[Section 4.3.2 (Red Light Violation Rate); objective defined in Section 3.2.2, Eq. (10)]
"To validate the accuracy of our method in the absence of ground-truth traffic signal states, we calculate the red light violation rate based on observed vehicle trajectories. The rationale is that red light running is a rare event in reality. Thus, if our estimated traffic light states are accurate, there should be a limited number of red light running events. ... Note that when calculating the red light violation rate of the proposed method, we consider all movements, including those missing or unknown in the original dataset."
The signal states being validated are not independent of the trajectories used to validate them. Section 3.2.2 selects x*(t) by Eq. (10) to minimize s_conflict (Eq. 9), a weighted mismatch count against trajectory-derived imputed states, with weights c_i from the same acceleration and velocity observations. The violation rate then counts, on those same trajectories, crossings while x*(t) is red; a crossing can be re-labeled green via Table 2 when c_i >= theta, deleting the violation by construction. The 15.7% to 2.9% drop therefore mostly restates the method's own optimization objective rather than providing external confirmation.
full rationale
The core trajectory-based imputation method is not itself circular: Section 3.2.1 derives estimated movement states from deceleration and velocity patterns, and the SUMO validation in Section 4.1 provides an independent ground-truth check, achieving accuracy above 96% across fixed-time and actuated plans, with parameters calibrated in simulation. However, the paper's principal WOMD-scale real-data validation, the reduction in red-light violation rate from 15.7% to 2.9% in Section 4.3.2, is circular as an accuracy claim. The final state x*(t) is chosen via Eq. (10) to minimize conflict with trajectory-derived estimates, and the violation rate counts conflicts between those same trajectories and x*(t); low violation rates are therefore a restatement of the optimization objective. The comparison is also not like-for-like, because the raw violation rate is evaluated only on originally labeled states, while the corrected rate includes all imputed movements. The simulation validation is independent but assumes compliant driving in SUMO, so it does not resolve the real-data circularity. No load-bearing self-citation chain was found: references to prior SPaT work by one of the authors are contextual rather than foundational. The authors' own Section 5 limitation, noting that performance may degrade at nonstandard intersections, further narrows but does not repair the circular real-data validation. Overall, the central real-data validation reduces by construction, giving a score of 6.
Assumptions & free parameters
free parameters (10)
- a_green =
0.5 m/s^2
- a_red =
-2.0 m/s^2
- v_green =
3.5 m/s
- v_red =
0.5 m/s
- theta =
1.0
- w_big =
100
- w_small =
0.1
- t_min =
30 time steps (3 s at 10 Hz)
- t_yellow =
20 time steps (2 s at 10 Hz)
- Delta_t =
10 time steps (1 s at 10 Hz)
assumptions (6)
- domain assumption Vehicle acceleration and velocity near the stop bar reliably indicate whether the signal is red or green.
- domain assumption Red-light running is rare in real-world driving.
- domain assumption Signalized intersections in WOMD follow the NEMA ring-and-barrier structure, so only enumerated green configurations are possible.
- domain assumption Right-turn movements can be ignored because right turns on red are often permitted.
- domain assumption Original signal labels are correct when they agree with the trajectory estimate.
- domain assumption A fixed yellow duration of 2 seconds is an acceptable approximation across all intersections.
Cite this review
Pith. "Pith review of Improving Traffic Signal Data Quality for the Waymo Open Motion Dataset." pith.science (2026). https://pith.science/paper/LNYFGS5I
@misc{pith2026250607150,
author = {Pith},
title = {Pith review of: Improving Traffic Signal Data Quality for the Waymo Open Motion Dataset},
year = {2026},
howpublished = {\url{https://pith.science/paper/LNYFGS5I}},
note = {Machine review of arXiv:2506.07150}
}
read the original abstract
Datasets pertaining to autonomous vehicles (AVs) hold significant promise for a range of research fields, including artificial intelligence (AI), autonomous driving, and transportation engineering. Nonetheless, these datasets often encounter challenges related to the states of traffic signals, such as missing or inaccurate data. Such issues can compromise the reliability of the datasets and adversely affect the performance of models developed using them. This research introduces a fully automated approach designed to tackle these issues by utilizing available vehicle trajectory data alongside knowledge from the transportation domain to effectively impute and rectify traffic signal information within the Waymo Open Motion Dataset (WOMD). The proposed method is robust and flexible, capable of handling diverse intersection geometries and traffic signal configurations in real-world scenarios. Comprehensive validations have been conducted on the entire WOMD, focusing on over 360,000 relevant scenarios involving traffic signals, out of a total of 530,000 real-world driving scenarios. In the original dataset, 71.7% of traffic signal states are either missing or unknown, all of which were successfully imputed by our proposed method. Furthermore, in the absence of ground-truth signal states, the accuracy of our approach is evaluated based on the rate of red-light violations among vehicle trajectories. Results show that our method reduces the estimated red-light running rate from 15.7% in the original data to 2.9%, thereby demonstrating its efficacy in rectifying data inaccuracies. This paper significantly enhances the quality of AV datasets, contributing to the wider AI and AV research communities and benefiting various downstream applications. The code and improved traffic signal data are open-sourced at https://github.com/michigan-traffic-lab/WOMD-Traffic-Signal-Data-Improvement
Figures
Figures from the paper (10 more)
Reference graph
Works this paper leans on
-
[2]
Gorela: Go relative for viewpoint-invariant motion forecasting, in: 2023 IEEE International Conference on Robotics and Automation (ICRA), IEEE. pp. 7801–7807. Dauner, D., Hallgarten, M., Geiger, A., Chitta, K.,
work page 2023
-
[4]
arXiv preprint arXiv:2403.15098
Unitraj: A unified framework for scalable vehicle trajectory prediction. arXiv preprint arXiv:2403.15098 . Gordon, R.L., Reiss, R.A., Haenel, H., Case, E., French, R.L., Mohaddes, A., Wolcott, R., et al.,
-
[5]
arXiv preprint arXiv:2404.02524
Versatile scene-consistent traffic scenario generation as optimization with diffusion. arXiv preprint arXiv:2404.02524 . Jiao, Y., Li, G., Calvert, S.C., van Cranenburgh, S., van Lint, H.,
-
[6]
Towards Knowledge-driven Autonomous Driving
Towards knowledge-driven autonomous driving. arXiv preprint arXiv:2312.04316 . Lopez, P.A., Behrisch, M., Bieker-Walz, L., Erdmann, J., Fl¨otter¨od, Y.P., Hilbrich, R., L¨ ucken, L., Rummel, J., Wagner, P., Wießner, E.,
-
[8]
Use of connected vehicle data to identify signal timing plans on signalized arterial corridors, in: 2023 IEEE International Automated Vehicle Validation Conference (IA VVC), IEEE. pp. 1–6. Manual, H.C.,
work page 2023
-
[9]
Wayformer: Motion forecasting via simple & efficient attention networks, in: 2023 IEEE International Conference on Robotics and Automation (ICRA), IEEE. pp. 2980–2987. Punzo, V., Borzacchiello, M.T., Ciuffo, B.,
work page 2023
-
[11]
https://waymo.com/open/challenges/2024/motion- prediction/
Waymo motion prediction challenge. https://waymo.com/open/challenges/2024/motion- prediction/. 22 Wen, X., Cui, Z., Jian, S.,
work page 2024
-
[12]
URL: https://en.wikipedia.org/wiki/Disjoint-set_data_structure
Disjoint-set data structure — Wikipedia, The Free Encyclopedia. URL: https://en.wikipedia.org/wiki/Disjoint-set_data_structure. [Online; accessed 30-July- 2024]. Wilson, B., Qi, W., Agarwal, T., Lambert, J., Singh, J., Khandelwal, S., Pan, B., Kumar, R., Hartnett, A., Pontes, J.K., et al.,
work page 2024
Show all 16 references
-
[13]
arXiv preprint arXiv:2301.00493
Argoverse 2: Next generation datasets for self-driving perception and forecasting. arXiv preprint arXiv:2301.00493 . Yan, X., Zou, Z., Feng, S., Zhu, H., Sun, H., Liu, H.X.,
-
[14]
arXiv preprint arXiv:2412.05334
Closed-loop supervised fine-tuning of tokenized traffic models. arXiv preprint arXiv:2412.05334 . Zhang, Z., Liniger, A., Dai, D., Yu, F., Van Gool, L.,
-
[15]
Trafficbots: Towards world models for autonomous driving simulation and motion prediction, in: 2023 IEEE International Conference on Robotics and Automation (ICRA), IEEE. pp. 1522–1529. Zhou, Z., Wang, J., Li, Y.H., Huang, Y.K.,
2023
-
[16]
Those vehicles with𝑑≥−8 m are taken into consideration
be its current acceleration; and𝑣≥0 (unit: m/s) be its velocity. Those vehicles with𝑑≥−8 m are taken into consideration. Then the acceleration confidence function𝑓and the velocity confidence function𝑔are given by: (𝑡is neglected for concise expression) 𝑓(𝑑,𝑎)= 0,if...
1996
-
[2018]
Microscopic traffic simulation using sumo, in: 2018 21st international conference on intelligent transportation systems (ITSC), Ieee. pp. 2575–2582. Mahmud, S., Huang, T., Sharma, A., Day, C.M.,
2018
-
[2023]
arXiv preprint arXiv:2306.16927
End-to-end autonomous driving: Challenges and frontiers. arXiv preprint arXiv:2306.16927 . Coifman, B., Li, L.,
-
[2024]
arXiv preprint arXiv:2406.15349
Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking. arXiv preprint arXiv:2406.15349 . Du, Z., Yan, X., Zhu, J., Sun, W.,
-
[2025]
arXiv preprint arXiv:2504.09103
Impact: Be- havioral intention-aware multimodal trajectory prediction with adaptive context trimming. arXiv preprint arXiv:2504.09103 . Urbanik, T., Tanaka, A., Lozner, B., Lindstrom, E., Lee, K., Quayle, S., Beaird, S., Tsoi, S., Ryus, P., Gettman, D., et al.,
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.