REVIEW 4 major objections 4 minor 18 references
Joint-Local Grounded Action Transformation for Sim-to-Real Transfer in Multi-Agent Traffic Control
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read JL-GAT extends Grounded Action Transformation to multi-agent traffic signal control so policies trained in simulation suffer a smaller performance drop when deployed under shifted dynamics.
desk verdict A plausible extension of GAT to multi-agent traffic control, but the headline comparison is compromised by asymmetric checkpoint selection on the target environment. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the local joint state-action pair $(o^L_{i,t}, a^L_{i,t})$ together with per-agent forward and inverse models. The forward model encodes the claim that a single intersection's next observation in the real environment can be predicted from the joint state and actions of agents within a fixed Manhattan sensing radius, and the inverse model encodes the correction that turns that predicted real next state into a grounded simulated action. The cascading invalidation effect and its two mitigations -- pattern grounding, which alternates which agents are grounded so a grounded agent never relies on a neighbor that is also being grounded, and probabilistic grounding, which independently grounds each agent with some probability -- form the second piece of machinery, managing the fixed-neighbor-action assumption that makes joint-local grounding usable.
What would settle it
In the 4x4 snowy setting, run JL-GAT with probabilistic grounding at $1/N$, ground one agent, then force an in-radius neighbor to switch to a different signal phase before the step executes. If the realized next observation matches the forward model's prediction as closely as when the neighbor's action is unchanged, the cascading invalidation effect is not the bottleneck the paper claims; if the prediction error spikes, the effect and its mitigations are doing real work.
Extended reading notes
Core claim
JL-GAT is a decentralized GAT in which agent $i$'s forward model $f_{i,\phi_+}$ takes the local joint state $o^L_{i,t}$ and local joint action $a^L_{i,t}$ -- agent $i$'s own observation and action together with those of agents $j$ at Manhattan distance $d(i,j) \le r$ -- and predicts the next individual observation $\hat{o}_{i,t+1}$ that would occur in the real environment. The inverse model $h_{i,\phi_-}$ then takes $o^L_{i,t}$, $a^L_{i,t}$, and $\hat{o}_{i,t+1}$ to produce a grounded action $\hat{a}^g_{i,t}$ meant to make the simulator reproduce that real transition. Because both models assume neighboring actions stay fixed while grounding happens, grounding adjacent agents at the same time can invalidate the grounded actions, an effect the paper calls cascading invalidation; JL-GAT mitigates it by grounding according to a fixed pattern that avoids neighbors or by grounding each agent with probability $p_{\text{ground}}$, with $1/N$ recommended as a starting value. Experiments on 1x3 and 4x4 networks under rainy and snowy settings show the hybrid design reduces the sim-to-real gap relative to direct transfer, centralized GAT, and decentralized GAT across all five reported metrics.
Load-bearing premise
The load-bearing premise is that an intersection's next state depends only on the joint observations and actions of agents within a fixed city-block radius $r$, and that those neighbors' actions stay fixed while the agent's action is being grounded.
Editorial extensions
If this is right
- In the tested 1x3 and 4x4 networks, JL-GAT with probabilistic grounding at $1/N$ reduces the sim-to-real gap on average travel time, queue, delay, and reward, and improves the throughput gap, relative to direct transfer, centralized GAT, and decentralized GAT.
- Removing neighbor states or neighbor actions from either the forward or the inverse model sends performance back toward direct-transfer levels, so each piece of the joint-local input is load-bearing.
- Because each agent retains its own forward and inverse models, JL-GAT does not require concatenating a global state that grows with the number of intersections; the method keeps a decentralized, scalability-friendly structure.
- The cascading invalidation effect warns that simultaneous independent grounding in a multi-agent network can be self-defeating; grounding must be scheduled or randomized, and the paper recommends $1/N$ as the starting probability.
- Adding an uncertainty-based dynamic grounding rate to JL-GAT further reduces the sim-to-real gap in the 1x3 setting.
Reading between the lines
- Editorial extension: pattern grounding is only demonstrated on a 1x3 network, so whether an alternating schedule remains effective on larger grids is an open test.
- Editorial extension: the fixed Manhattan radius $r$ is the main untested boundary of the method; varying $r$ or learning which neighbors matter would directly probe it.
- Editorial extension: the $1/N$ starting probability suggests a scaling rule of thumb -- per-agent grounding probability should fall as the network grows -- but that scaling law is a conjecture the paper does not verify.
- Editorial extension: pattern grounding and the uncertainty-based grounding rate from Section 6.6 could be combined so that the grounding schedule itself becomes adaptive.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes JL-GAT, a multi-agent extension of Grounded Action Transformation (GAT) for traffic signal control. JL-GAT keeps per-agent forward and inverse models (as in decentralized GAT) but augments each agent's inputs with the observations and actions of neighboring agents within a Manhattan sensing radius. The paper identifies a ``cascading invalidation effect'' that arises when multiple grounded agents assume fixed neighbor actions, and proposes two mitigations: pattern grounding and probabilistic grounding. Experiments are conducted in a sim-to-sim protocol (CityFlow as E_sim, SUMO as E_real) under rainy and snowy settings on 1x3 and 4x4 networks. The reported results claim that JL-GAT consistently reduces the sim-to-real gap in ATT, queue, delay, throughput, and reward relative to direct transfer, centralized GAT, and decentralized GAT.
Significance. If the empirical claims hold, JL-GAT would be a useful contribution to sim-to-real transfer for multi-agent traffic signal control, as it preserves scalability while incorporating local interaction structure, and it identifies a concrete multi-agent failure mode absent from single-agent GAT. The paper ships open code, which is a strength for reproducibility. However, the reported evaluation contains an asymmetric checkpoint-selection protocol and a suspicious baseline result; until these are resolved, the central claim that JL-GAT consistently reduces the sim-to-real gap is not yet established.
major comments (4)
- [Section 6.3, captions of Tables 1 and 2] The evaluation protocol is asymmetric across methods. For every GAT-based configuration, the reported epoch is selected based on the lowest average travel time in E_real (SUMO), the target environment, while Direct Transfer is selected based on the lowest ATT in E_sim (CityFlow), the source environment. This allows the GAT methods, including the proposed JL-GAT, to use target-environment performance for model selection, while Direct Transfer cannot. The claimed gap reduction could therefore be substantially inflated by target-dependent checkpoint selection rather than by grounding. The authors should report results with checkpoints selected on E_sim for all methods, or otherwise use a common held-out criterion, and should also report standard deviations for Direct Transfer, which are currently omitted despite six trials.
- [Tables 1 and 2, 4x4 row] The Centralized GAT baseline reports exactly the same values as Direct Transfer across all metrics with zero standard deviation (e.g., ATT 485.63 ±0.00 in Table 1). This is not a plausible outcome for a trained GAT baseline, which would be expected to show at least some seed-to-seed variation unless the baseline was never actually trained or the reported results were copied. This baseline is used to support the claim that JL-GAT outperforms centralized GAT at scale. The authors should verify that the centralized GAT baseline was run to completion, report its training curves or logs, or correct the table if it is erroneous.
- [Section 5.3] Pattern grounding is described only for a 1x3 network example (ground the first and last agent in one epoch, then the middle agent in the next), and no protocol is given for how patterns are defined in the 4x4 experiments reported in Tables 1 and 2. Meanwhile, the paper states that probabilistic grounding ``does not directly overcome the cascading invalidation effect.'' Since the main 4x4 results include JL-GAT (Pattern), the mechanism underlying those results is underspecified. The authors should define the pattern set used for 4x4 and, ideally, provide a direct analysis of how often grounding conflicts occur and how each mitigation reduces them.
- [Table 6] In the ablation study, the rows ``Inverse Model w/o Neigh. States'' and ``Inverse Model w/o Neigh. Actions'' report values exactly equal to Direct Transfer with ±0.00 standard deviations. While the text says these methods ``failed to improve'' the initialization, a zero variance identical to the direct-transfer numbers suggests these configurations may not have executed any learning or grounding (e.g., the grounded action may have been the original action by construction). Please clarify whether these ablations actually trained and what their learned models output, or, if the design made them equivalent to Direct Transfer by construction, say so explicitly.
minor comments (4)
- [Algorithm 1 (Section E)] The algorithm input line says ``real-world dataset Dsim''; this should be ``Dreal'' to match the notation used throughout the paper.
- [Section 6.1 / Section 5.2.1] The sensing radius r is a key hyperparameter of JL-GAT, but no r value is reported for the experiments. Please state the radius used for the 1x3 and 4x4 networks and, if possible, provide a sensitivity analysis over r.
- [Equation (7)] The notation ag_i,t in the CCE loss is ambiguous: the inverse model is trained to predict the grounded action \hat a^g_i,t, but ag_i,t is described as sampled from trajectories collected in E_sim. Clarify whether ag_i,t is the original (non-grounded) action taken in E_sim, and use distinct notation to avoid confusion.
- [Figures 4 and 5] The figure captions should explicitly state that the ``best episode'' is selected by performance in E_real, consistent with the caption of Table 1, so that the checkpoint-selection protocol is transparent to the reader.
Circularity Check
Target-environment checkpoint selection in Tables 1-2 fits the reported GAT performance to Ereal, making the claimed sim-to-real gap reduction partially an artifact of selection rather than of grounding.
-
fitted input called prediction
[Section 6.3; Table 1 and Table 2 notes]
"For each GAT configuration and network size pair, we run six independent trials and identify the best epoch in each trial based on the lowest average travel time (ATT) in Ereal. ... Note that Direct Transfer is reported as the policies from the best performing epoch (by lowest ATT) in Esim being tested in Ereal after six trials of 300 epochs."
The GAT-based methods select their reported checkpoint by minimizing target-environment ATT (Ereal), which is the same quantity used to measure the claimed improvement. Direct Transfer selects its checkpoint in Esim and is then tested once in Ereal, so it receives no target-information advantage. The central claim that JL-GAT 'consistently reduces the sim-to-real performance gap' (Section 7) is therefore supported by a comparison in which the GAT numbers are partially fitted to the evaluation environment while the baseline is not. The apparent gap reduction in Tables 1-2 is at least partly an artifact of target-dependent checkpoint selection rather than of the grounding mechanism itself.
full rationale
The paper's derivation chain (Equations 1-7) is not internally circular: the forward and inverse models are fitted to Dreal and Dsim respectively and then used to produce grounded actions for policy training in Esim, and evaluating the resulting policy in Ereal is the intended use of GAT rather than a tautology. The GAT framework is attributed to Hanna and Stone (2017), and the authors' prior work is cited mainly for implementation details such as MSE/CCE losses, the sim-to-sim setup, and the gap metric, not as an unverified uniqueness theorem. The main circularity concern is the empirical protocol. Section 6.3 and the Table 1/2 notes state that every GAT configuration selects its reported epoch by lowest ATT in Ereal, whereas Direct Transfer selects by lowest ATT in Esim and is then tested once in Ereal. This is a fitted-input leak: the GAT results are partially constructed from target-environment performance, so the central claim that JL-GAT consistently reduces the sim-to-real gap rests on an asymmetric comparison. The gap reduction could be substantially explained by target-informed checkpoint selection rather than by grounding. This warrants a partial-circularity score of 6 rather than 0; however, the methodological derivation itself is independent, and the issue could be fixed by evaluating all methods with the same checkpoint rule, such as a fixed epoch or source-selected epoch.
Assumptions & free parameters
free parameters (3)
- Sensing radius r =
1 for all experiments (3x3 local grid)
- Grounding probability pground =
1/N (0.33 for 1x3, 0.0625 for 4x4); also 0.2, 0.5, 0.8 tested in Table 3
- Pattern grounding schedule =
Alternating first/last agent vs middle agent in 1x3 network
assumptions (4)
- standard math Neural networks for forward and inverse models are universal function approximators, so they can fit the simulator and real transition dynamics.
- domain assumption Each agent's next observation depends only on the local joint observation and action within Manhattan radius r.
- domain assumption Neighboring agents' actions within radius r remain fixed during the grounding step.
- domain assumption The sim-to-sim setup, with CityFlow as Esim and SUMO with modified weather parameters as Ereal, is a valid proxy for real-world sim-to-real transfer.
Cite this review
Pith. "Pith review of Joint-Local Grounded Action Transformation for Sim-to-Real Transfer in Multi-Agent Traffic Control." pith.science (2026). https://pith.science/paper/SBPFHPJK
@misc{pith2026250715174,
author = {Pith},
title = {Pith review of: Joint-Local Grounded Action Transformation for Sim-to-Real Transfer in Multi-Agent Traffic Control},
year = {2026},
howpublished = {\url{https://pith.science/paper/SBPFHPJK}},
note = {Machine review of arXiv:2507.15174}
}
read the original abstract
Traffic Signal Control (TSC) is essential for managing urban traffic flow and reducing congestion. Reinforcement Learning (RL) offers an adaptive method for TSC by responding to dynamic traffic patterns, with multi-agent RL (MARL) gaining traction as intersections naturally function as coordinated agents. However, due to shifts in environmental dynamics, implementing MARL-based TSC policies in the real world often leads to a significant performance drop, known as the sim-to-real gap. Grounded Action Transformation (GAT) has successfully mitigated this gap in single-agent RL for TSC, but real-world traffic networks, which involve numerous interacting intersections, are better suited to a MARL framework. In this work, we introduce JL-GAT, an application of GAT to MARL-based TSC that balances scalability with enhanced grounding capability by incorporating information from neighboring agents. JL-GAT adopts a decentralized approach to GAT, allowing for the scalability often required in real-world traffic networks while still capturing key interactions between agents. Comprehensive experiments on various road networks under simulated adverse weather conditions, along with ablation studies, demonstrate the effectiveness of JL-GAT. The code is publicly available at https://github.com/DaRL-LibSignal/JL-GAT/.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
is the discount factor. At each time step t, agent i observes its own state oi,t ∈ Ωi, selects an action ai,t ∈ Ai, and receives a reward ri,t. Agent actions are taken simultaneously and comprise a global action at, which transitions the environment from a global state st to a global next state st+1, where global states consist of observations oi,t for ea...
work page 2018
-
[8]
Towards adapting deep visuomotor representations from simulated to real environments
Eric Tzeng, Coline Devin, Judy Hoffman, Chelsea Finn, Xingchao Peng, Sergey Levine, Kate Saenko, and Trevor Darrell. Towards adapting deep visuomotor representations from simulated to real environments. arXiv preprint arXiv:1511.07111, 2(3),
-
[10]
Using simulation and domain adaptation to improve efficiency of deep robotic grasping
Konstantinos Bousmalis, Alex Irpan, Paul Wohlhart, Yunfei Bai, Matthew Kelcey, Mrinal Kalakrishnan, Laura Downs, Julian Ibarz, Peter Pastor, Kurt Konolige, et al. Using simulation and domain adaptation to improve efficiency of deep robotic grasping. In 2018 IEEE international conference on robotics and automation (ICRA), pages 4243–4250. IEEE,
work page 2018
-
[11]
Mark Cutler, Thomas J. Walsh, and Jonathan P. How. Reinforcement learning with multi-fidelity simula- tors. In 2014 IEEE International Conference on Robotics and Automation (ICRA) , pages 3888–3895,
work page 2014
-
[12]
Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean-Baptiste Mouret
doi:10.1109/ICRA.2014.6907423. Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean-Baptiste Mouret. Robots that can adapt like animals. Nature, 521 (7553):503–507, may
-
[13]
URL https://doi.org/10.1038%2Fnature14422
doi:10.1038/nature14422. URL https://doi.org/10.1038%2Fnature14422. Josiah Hanna and Peter Stone. Grounded action transformation for robot learning in simulation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31,
-
[16]
An imitation from observation approach to transfer learning with dynamics mismatch
12 Joint-Local Grounded Action Transformation Siddarth Desai, Ishan Durugkar, Haresh Karnan, Garrett Warnell, Josiah Hanna, and Peter Stone. An imitation from observation approach to transfer learning with dynamics mismatch. In Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS 2020), December 2020b. Hua Wei...
work page 2020
-
[2002]
Diagnosing reinforcement learning for traffic signal control
Guanjie Zheng, Xinshi Zang, Nan Xu, Hua Wei, Zhengyao Yu, Vikash Gayah, Kai Xu, and Zhenhui Li. Diagnosing reinforcement learning for traffic signal control. arXiv preprint arXiv:1905.04716,
arXiv 1905
Show all 18 references
-
[2004]
Sumo–simulation of urban mobility: an overview
Michael Behrisch, Laura Bieker, Jakob Erdmann, and Daniel Krajzewicz. Sumo–simulation of urban mobility: an overview. In Proceedings of SIMUL 2011, The Third International Conference on Advances in System Simulation. ThinkMind,
2011
-
[2014]
arXiv preprint arXiv:1412.3474,
-
[2015]
Multi-task domain adaptation for deep learning of instance grasping from simulation
Kuan Fang, Yunfei Bai, Stefan Hinterstoisser, Silvio Savarese, and Mrinal Kalakrishnan. Multi-task domain adaptation for deep learning of instance grasping from simulation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3516–3523. IEEE,
2018
-
[2017]
Reproducible and low-cost sim-to- real environment for traffic signal control
Yiran Zhang, Khoa V o, Longchao Da, Tiejin Chen, Xiaoou Liu, and Hua Wei. Reproducible and low-cost sim-to- real environment for traffic signal control. In Proceedings of the ACM/IEEE 16th International Conference on Cyber-Physical Systems (with CPS-IoT Week 2025), pages 1–2,
2025
-
[2018]
A survey on traffic signal control methods
Hua Wei, Guanjie Zheng, Vikash Gayah, and Zhenhui Li. A survey on traffic signal control methods. arXiv preprint arXiv:1904.08117, 2019a. Haoyuan Jiang, Ziyue Li, Hua Wei, Xuantang Xiong, Jingqing Ruan, Jiaming Lu, Hangyu Mao, and Rui Zhao. X-light: Cross-city traffic signal c...
1904 arXiv
-
[2019]
Cityflower: An efficient and realistic traffic simulator with embedded machine learning models
Longchao Da, Chen Chu, Weinan Zhang, and Hua Wei. Cityflower: An efficient and realistic traffic simulator with embedded machine learning models. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 368–373. Springer, 2024a. Longchao Da,...
2023
-
[2020]
A survey of sim-to-real methods in rl: Progress, prospects and challenges with foundation models
Longchao Da, Justin Turnau, Thirulogasankar Pranav Kutralingam, Alvaro Velasquez, Paulo Shakarian, and Hua Wei. A survey of sim-to-real methods in rl: Progress, prospects and challenges with foundation models. arXiv preprint arXiv:2502.13187,
-
[2021]
Uncertainty quantification and confidence calibration in large language models: A survey
Xiaoou Liu, Tiejin Chen, Longchao Da, Chacha Chen, Zhen Lin, and Hua Wei. Uncertainty quantification and confidence calibration in large language models: A survey. arXiv preprint arXiv:2503.15850,
-
[2024]
Sim-to-real transfer in deep reinforcement learning for robotics: a survey
Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In 2020 IEEE symposium series on computational intelligence (SSCI), pages 737–744. IEEE,
2020
-
[2025]
Hanna, Garrett Warnell, and Peter Stone
Siddharth Desai, Haresh Karnan, Josiah P. Hanna, Garrett Warnell, and Peter Stone. Stochastic grounded action transformation for robot learning in simulation. In IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS 2020), October 2020a. Haresh Karnan, Siddh...
2020
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.