Pith. sign in

REVIEW 3 major objections 5 minor 49 references

A co-evolving worst-case estimator hardens multi-agent traffic signal controllers against demand shifts without changing their architectures.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 14:43 UTC pith:ABML3SKV

load-bearing objection Solid incremental systems paper: co-evolving CB-WCE wrapper hardens existing MARL signal controllers in SUMO, with large queue cuts on Monaco, but the K=11 synthetic demand set is the real limit on the robustness claim. the 3 major comments →

arxiv 2607.09899 v1 pith:ABML3SKV submitted 2026-07-10 eess.SY cs.SY

A Distributionally Robust Multi-agent Reinforcement Learning Framework for Intelligent Intersection Control

Shuwei Pei , Joran Borger , Arda Kosay , Bayu Jayawardhana , Muhammed O. Sayin , Saeed Ahmed This is my paper
classification eess.SY cs.SY
keywords multi-agent reinforcement learningdistributionally robust optimizationtraffic signal controlcontextual banditintelligent transportation systemsdemand uncertaintyworst-case robustness
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard multi-agent reinforcement learning for traffic lights optimizes only average performance under nominal demand, so policies often collapse into gridlock when flows shift in space or time. This paper shows that a slower Contextual-Bandit Worst-Case Estimator can co-evolve with the controllers, continually sampling adversarial mixtures of demand scenarios during training. The same value-based, actor-critic, and policy-gradient algorithms then learn policies that cut both worst-case and average network queues, prevent unbounded queue growth, and improve performance on an unseen demand pattern. On the Monaco city network, for example, robust retraining of a PPO controller reduced worst-case queue length by about 74 percent and average-case queue length by about 75 percent. The result matters because urban traffic authorities care as much about reliability under adverse conditions as about average delay, and the method requires no redesign of existing controller architectures.

Core claim

An algorithm-agnostic distributionally robust MARL framework built around a co-evolving Contextual-Bandit Worst-Case Estimator fortifies traffic-signal controllers against spatial-temporal demand shifts. The estimator runs on a slower timescale, generating demand mixtures that maximize congestion under the current multi-agent policy; simultaneous retraining steers the controllers toward better worst-case and average-case performance without modifying their architectures. On a synthetic 5x5 grid and the heterogeneous Monaco City network, across IA2C, MA2C, PPO, and IQL-LR, the approach prevents unbounded queue growth, yields large reductions in worst-case and average queues (notably 74.39 per

What carries the argument

The Contextual-Bandit Worst-Case Estimator (CB-WCE): a slower-timescale adversary that compresses network speed and density into a state, outputs mixture weights over a finite set of demand scenarios, and is trained to maximize network waiting time so the faster MARL controllers are continually exposed to the currently hardest demand mixtures.

Load-bearing premise

A fixed family of eleven synthetic origin-destination demand groups, and convex mixtures of them held constant for ten-minute windows, is rich enough to stand in for real urban demand uncertainty.

What would settle it

Retrain the controllers with the CB-WCE on the eleven synthetic groups, then evaluate on multi-day real traffic traces that include incidents, special events, or weather shifts not representable as mixtures of those groups; if worst-case queues still grow unbounded relative to the nominal baseline, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Existing MARL traffic controllers can be made far more resilient by co-training with a CB-WCE without redesigning networks, rewards, or action spaces.
  • Worst-case and average-case queue performance can improve together rather than trading off against each other.
  • Policies trained this way retain gains under held-out demand distributions, supporting zero-shot use on novel traffic patterns.
  • The method scales from homogeneous grids to irregular real-city networks with heterogeneous phase sets.
  • Agencies can replace the synthetic demand family with clusters from their own sensor data to target local bottlenecks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The dual-timescale structure (window-level adversary plus step-level controllers) is likely portable to other multi-agent allocation problems with non-stationary demand, such as warehouse routing or grid energy dispatch.
  • If the finite demand family is too sparse, the estimator may overfit synthetic bottlenecks that do not match real incidents, so continuous or data-driven demand embeddings are a natural next test.
  • Once robust training is in place, dense neighbor fingerprints can become redundant or even limiting, suggesting state design should be re-tuned after the robustness objective is added.
  • Coupling the CB-WCE with explicit emission or safety constraints could produce multi-objective robust controllers without separate reward engineering.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes an algorithm-agnostic distributionally robust MARL framework for traffic signal control that couples existing controllers (IA2C, MA2C, PPO, IQL-LR) with a co-evolving Contextual-Bandit Worst-Case Estimator (CB-WCE). The CB-WCE operates on a slower window timescale, samples convex mixtures of a finite set of K=11 synthetic OD demand groups (Eq. 7), and is trained to maximize network-wide cumulative waiting time (Eq. 8–9), thereby steering controller updates toward bottleneck scenarios without altering the underlying MARL architectures. Evaluations on a synthetic 5×5 grid and the heterogeneous Monaco City network report large reductions in both worst-case and average-case queue lengths (e.g., PPO-Monaco: 74.39% worst-case and 75.45% average-case queue reduction) together with improved zero-shot performance on a held-out Group 12 demand pattern, and qualitative time-series showing prevention of unbounded queue growth.

Significance. If the reported gains transfer beyond the synthetic uncertainty set, the work supplies a practical, architecture-agnostic recipe for hardening MARL traffic controllers against demand shifts—an operationally important gap relative to average-case training. Strengths include systematic evaluation across four algorithm families (value-based, actor-critic, policy-gradient) and two network topologies, public code lineage, and explicit co-evolution of the adversary rather than a frozen estimator. The finite-support DRO framing and windowed adversarial sampling are cleanly integrated with standard MARL training loops, making the method immediately usable by practitioners who already deploy IA2C/MA2C/PPO-style controllers.

major comments (3)
  1. The central robustness claim (Abstract, §VI, PPO-Monaco 74.39%/75.45% figures) rests on the assumption that a fixed finite support of K=11 synthetic OD groups and their 600 s convex mixtures (Eq. 7, §V-B) constitutes a sufficient uncertainty set for real urban demand variability. Group 12 is either a remapped Hangzhou pattern or the single MoST real-life demand; neither constitutes an independent operational distribution (incidents, weather, unmodeled OD shifts). Without additional held-out families or sensitivity analysis on K and mixture support, the large relative gains and “strong zero-shot generalization” remain simulation-internal and may not transfer.
  2. Relative percentage improvements are inflated by near-collapse of several baselines (near-zero speeds, unbounded queues in Figs. 6 and 9, especially IQL-LR Group 11 and PPO Group 3). The manuscript acknowledges this statistical artifact in §VI-C but still leads with the 74–75% and >100% speed figures in the abstract and contribution list. Absolute metrics (or normalized improvements relative to a non-collapsing baseline) should be reported as primary evidence so that the magnitude of the claimed robustness gain can be assessed independently of baseline failure modes.
  3. Phase II freezes the controller while training the CB-WCE for 500 episodes, then Phase III co-evolves both for 1000 episodes (§V-C). No ablation compares this schedule against (i) a frozen estimator throughout retraining (as in the precursor [9]) or (ii) continued uniform cycling over the K groups for an equal number of episodes. Without these controls it is unclear how much of the reported lift is attributable to co-evolution versus simply longer training under a more diverse demand schedule.
minor comments (5)
  1. Reward coefficient a=0.2 (§III-C) and the choice of Δt=5 s / 2 s clearance are stated without sensitivity; a short ablation or justification relative to the more common 10 s / 5 s literature values would help reproducibility.
  2. Table I and the observation design paragraph note that wait is dropped for Monaco “to mitigate observation noise,” yet no quantitative comparison of wave-only versus wave+wait is provided.
  3. Figs. 4 and 7 use dual color scales that are hard to read in grayscale; consider adding numeric annotations or separate bar plots for the key worst-case groups.
  4. The arXiv identifier 2607.09899 and the precursor citation [9] (arXiv:2512.18558) should be cross-checked for consistency of claimed novelty (co-evolving vs. fixed estimator).
  5. Minor typographical issues: “donotkeep” (p. 2), inconsistent capitalization of “Group” vs. “group,” and occasional missing spaces around citations.

Circularity Check

1 steps flagged

No significant circularity: empirical MARL training/evaluation results on fixed demand groups; precursor self-citation is acknowledged but not load-bearing for the new Monaco/co-evolution claims.

specific steps
  1. self citation load bearing [Section I (Introduction), contributions list, and Algorithm 1 discussion]
    "Our recent work in [9], a precursor to this paper, introduced a distributionally robust multi-agent reinforcement learning (DR-MARL) framework for localized traffic grids by employing a contextual-bandit worst-case estimator (CB-WCE) [17] that adaptively reweights traffic scenarios during training. ... In this present work, we extend our previous results in [9] as follows. Firstly, we do not keep the estimator fixed during retraining (as shown later in Algorithm 1 ...)"

    The framework and CB-WCE idea are introduced via self-citation of the authors' own precursor. However this is not load-bearing: the paper immediately distinguishes the new co-evolving protocol, larger Monaco network, and multi-algorithm evaluation, and all quantitative claims rest on fresh simulation measurements rather than on unverified assertions from [9].

full rationale

The paper is an empirical systems paper whose central claims (queue reductions of ~74% worst-case / ~75% average for PPO-Monaco, prevention of unbounded growth, zero-shot gains on Group 12) are measured by comparing baseline vs. DR-retrained policies on independent fixed-horizon rollouts under the same demand groups (Section V-D, VI). The CB-WCE maximizes network waiting time (Eq. 8–9) while controllers minimize local queue/wait rewards (Eq. 3–4); the two-timescale co-evolution (Algorithm 1) is a training procedure, not a definitional identity that forces the reported evaluation metrics. The finite K=11 synthetic OD set and convex mixtures (Eq. 7, V-B) are an explicit modeling choice whose sufficiency is an assumption, not a circular derivation. Self-citation of the precursor [9] appears in the introduction and contributions but is explicitly scoped as limited (small grids, fixed estimator); the present work supplies new co-evolution, larger networks, and multi-algorithm results that stand independently. No fitted parameter is renamed a prediction, no uniqueness theorem is imported to force the architecture, and no known result is merely renamed. Score 1 reflects only the ordinary presence of a non-load-bearing self-citation of prior work by overlapping authors.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 1 invented entities

Central claims depend on a finite synthetic demand family, fixed control/window timescales, hand-chosen reward weights, and the modeling choice that a contextual bandit maximizing network waiting time is a valid adversary for distributional robustness. No new physical entities; the CB-WCE is an algorithmic construct adapted from prior work. Free parameters are training/protocol knobs that shape which policies are learned and how large the reported gains appear.

free parameters (6)
  • reward wait weight a = 0.2
    Grid reward uses a=0.2 to balance queue count vs waiting time (Eq. 3); chosen by scale balancing, not derived.
  • control interval Δt and clearance = 5 s / 2 s
    Δt=5 s and 2 s yellow/all-red chosen for temporal resolution vs literature 10 s / 5 s yellow; affects policy adaptiveness.
  • demand window T_win and episode length = 600 s / 6600 s / K=11
    T_win=600 s, H=6600 s, K=11 groups define the adversary timescale and training exposure; justified as emulating 10-minute demand persistence.
  • scenario demand intensities = ~2000–3000 veh/h (grid); ~2383 veh/h (Monaco)
    Grid dominant flows ~2000 veh/h (~3000 total); Monaco baseline ~2383.3 veh/h redistributed—scenario magnitudes set by design.
  • training episode budgets = 1e6 steps / 500 / 1000 episodes
    Baseline 1e6 steps; CB-WCE init 500 episodes; robust retrain 1000 episodes—hand-set schedules that determine convergence claims.
  • learning rates l_θ, l_ψ and network architectures
    Controller and estimator step sizes and CNN/GCN choices are free training hyperparameters not derived from theory.
axioms (5)
  • domain assumption Network-wide signal control is a decentralized MARL problem with local observations, discrete conflict-free phases, and Markovian SUMO transitions (Section III).
    Standard traffic-MARL modeling choice; load-bearing for all baselines and DR variants.
  • ad hoc to paper Demand uncertainty is adequately represented by a finite support of K synthetic OD groups and their convex mixtures (Eq. 5–7, Section V-B).
    Finite-support DRO objective and CB-WCE actions live entirely on this discrete scenario set.
  • ad hoc to paper Maximizing network-wide cumulative waiting time under mixed demand identifies scenarios that improve controller worst-case robustness (Eq. 8–9).
    Defines the adversary objective; not proven equivalent to min_k J_k(θ).
  • domain assumption Two-timescale co-evolution (window-level CB-WCE, step-level MARL) remains stable and tracks evolving worst cases (Algorithm 1).
    Assumed workable without theoretical convergence guarantees for the coupled system.
  • domain assumption Standard MARL update objectives (IA2C, MA2C, PPO, IQL-LR) with replay buffers are valid base learners under nonstationary adversarial demand.
    Framework is algorithm-agnostic only if these learners still converge under CB-WCE sampling.
invented entities (1)
  • Contextual-Bandit Worst-Case Estimator (CB-WCE) co-evolving with MARL traffic controllers no independent evidence
    purpose: Generate adaptive adversarial demand mixtures on a slower timescale to steer distributionally robust training without changing controller architectures.
    Adapted from prior CB-WCE / Group-DRO style work ([17], precursor [9]); not a physical entity. independent_evidence false because its value is only shown inside the paper’s simulations.

pith-pipeline@v1.1.0-grok45 · 23683 in / 3684 out tokens · 38364 ms · 2026-07-14T14:43:18.904095+00:00 · methodology

0 comments
read the original abstract

Multi-agent reinforcement learning (MARL) has emerged as a promising approach for traffic signal control. However, standard MARL policies typically optimize for expected returns under nominal conditions, leaving them highly vulnerable to spatial-temporal demand shifts and catastrophic congestion under adverse scenarios. To address this critical limitation, this paper proposes an algorithm-agnostic Distributionally Robust (DR) MARL framework integrating an adaptive Contextual-Bandit Worst-Case Estimator (CB-WCE). Operating on a slower timescale, the CB-WCE co-evolves with the traffic controllers by dynamically generating adversarial demand mixtures during training. This steers the learning process to fortify policies against bottleneck scenarios without requiring modifications to the underlying MARL architectures. The framework is evaluated across value-based, actor-critic, and policy-gradient methods on both a synthetic 5x5 grid and a heterogeneous Monaco City network. Empirical results demonstrate that the DR framework prevents unbounded queue growth and profoundly enhances both worst-case robustness and average-case efficiency. Notably, for the Proximal Policy Optimization (PPO) architecture in the Monaco environment, on average, robust retraining reduced the worst-case queue length by 74.39% and improved the average-case network-wide queue length by 75.45%. Furthermore, the retrained policies exhibit strong zero-shot generalization to unseen traffic distributions, highlighting the framework's scalability and potential for resilient real-world urban deployment.

Figures

Figures reproduced from arXiv: 2607.09899 by Arda Kosay, Bayu Jayawardhana, Joran Borger, Muhammed O. Sayin, Saeed Ahmed, Shuwei Pei.

Figure 1
Figure 1. Figure 1: Schematic of the proposed DR-MARL training framework for [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Synthetic 5×5 grid network, with an example intersection shown in the middle. Horizontal approaches have two lanes (one left-turn and one shared through/right) with a speed limit of 20 m/s, while vertical approaches have a single mixed lane with a speed limit of 11 m/s. The grid’s regular structure allows for controlled experiments on coordination under homogeneous local control logic. N NE E SE S SW W NW … view at source ↗
Figure 3
Figure 3. Figure 3: Monaco traffic network with controller locations overlaid by direc [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Horizon- and rollout-averaged performance of baseline and distri [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Performance distributions of baseline (light fill, dashed) and DR [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Temporal evolution of worst-case queue length (upper) and average [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Performance distributions of baseline (light fill, dashed) and DR [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗
Figure 7
Figure 7. Figure 7: Horizon- and rollout-averaged performance of baseline and distribu [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 10
Figure 10. Figure 10: Performance comparison at the Monaco city intersection using the [PITH_FULL_IMAGE:figures/full_fig_p011_10.png] view at source ↗
Figure 9
Figure 9. Figure 9: Temporal evolution of worst-case queue length (upper) and average [PITH_FULL_IMAGE:figures/full_fig_p011_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

49 extracted references · 7 linked inside Pith

  1. [1]

    Real-world carbon dioxide impacts of traffic congestion,

    M. Barth and K. Boriboonsomsin, “Real-world carbon dioxide impacts of traffic congestion,”Transportation Research Record, vol. 2058, no. 1, pp. 163–171, 2008

    2058

  2. [2]

    J. D. Stanaway, A. Afshinet al., “Global, regional, and national compar- ative risk assessment of 84 behavioural, environmental and occupational, and metabolic risks or clusters of risks for 195 countries and territories, 1990–2017: A systematic analysis for the global burden of disease study 2017,”The Lancet, vol. 392, no. 10159, pp. 1923–1994, 2018

    1990

  3. [3]

    Applying the degree of urbanisation to the globe: A new harmonised definition reveals a different picture of global urbanisation,

    L. Dijkstra, A. J. Florczyk, S. Freire, T. Kemper, M. Melchiorri, M. Pesaresi, and M. Schiavina, “Applying the degree of urbanisation to the globe: A new harmonised definition reveals a different picture of global urbanisation,”Journal of Urban Economics, vol. 125, p. 103312, 2021

    2021

  4. [4]

    The SCOOT on-line traffic signal optimisation technique,

    P. Hunt, D. Robertson, R. Bretherton, and M. C. Royle, “The SCOOT on-line traffic signal optimisation technique,”Traffic Engineering & Control, vol. 23, no. 4, pp. 190–192, 1982

    1982

  5. [5]

    Two traffic-responsive area traffic control methods: SCAT and SCOOT,

    J. Luk, “Two traffic-responsive area traffic control methods: SCAT and SCOOT,”Traffic Engineering & Control, vol. 25, no. 1, pp. 14–22, 1984

    1984

  6. [6]

    N. H. Gartner,Demand-Responsive Decentralized Urban Traffic Control. US Department of Transportation, Research and Special Programs Administration, 1982

    1982

  7. [7]

    The PRODYN real time traffic algorithm,

    J.-J. Henry, J. L. Farges, and J. Tuffal, “The PRODYN real time traffic algorithm,” inControl in Transportation Systems. Elsevier, 1984, pp. 305–310

    1984

  8. [8]

    Adaptive traffic control systems: Domestic and foreign state of practice,

    A. Stevanovic, “Adaptive traffic control systems: Domestic and foreign state of practice,” National Cooperative Highway Research Program (NCHRP), Synthesis 403, 2010

    2010

  9. [9]

    Distribu- tionally robust multi-agent reinforcement learning for intelligent traffic control,

    S. Pei, J. Borger, A. Kosay, M. O. Sayin, and S. Ahmed, “Distribu- tionally robust multi-agent reinforcement learning for intelligent traffic control,”IFAC-PapersOnLine, 2026, accepted to the 23rd IFAC World Congress, Busan, Republic of Korea. arXiv:2512.18558

    arXiv 2026

  10. [10]

    R. S. Sutton and A. G. Barto,Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press, 1998

    1998

  11. [11]

    Reinforcement-learning-based job-shop scheduling for intelligent intersection management,

    S.-C. Huang, K.-E. Lin, C.-Y . Kuo, L.-H. Lin, M. O. Sayin, and C.-W. Lin, “Reinforcement-learning-based job-shop scheduling for intelligent intersection management,” inProceedings of the Design, Automation & Test in Europe Conference & Exhibition (DATE), 2023, pp. 1–6

    2023

  12. [12]

    Learning decentralized traffic signal controllers with multi- agent graph reinforcement learning,

    Y . Zhang, Z. Yu, J. Zhang, L. Wang, T. H. Luan, B. Guo, and C. Yuen, “Learning decentralized traffic signal controllers with multi- agent graph reinforcement learning,”IEEE Transactions on Mobile Computing, vol. 23, no. 6, pp. 7180–7195, 2023

    2023

  13. [13]

    Robust multi-agent reinforcement learning via adversarial regularization: Theoretical foundation and stable algorithms,

    A. Bukharin, Y . Li, Y . Yu, Q. Zhang, Z. Chen, S. Zuo, C. Zhang, S. Zhang, and T. Zhao, “Robust multi-agent reinforcement learning via adversarial regularization: Theoretical foundation and stable algorithms,” 13 inAdvances in Neural Information Processing Systems (NeurIPS), vol. 36, 2023, pp. 68 121–68 133

    2023

  14. [14]

    Travel time reliability to airport: Review and assessment,

    A. Jose and S. Ram, “Travel time reliability to airport: Review and assessment,”Transportation Research Procedia, vol. 48, pp. 2771–2783, 2020

    2020

  15. [15]

    Distributionally robust neural networks for group shifts: On the importance of reg- ularization for worst-case generalization,

    S. Sagawa, P. W. Koh, T. Hashimoto, and P. Liang, “Distributionally robust neural networks for group shifts: On the importance of reg- ularization for worst-case generalization,” inProceedings of the 8th International Conference on Learning Representations (ICLR), 2020

    2020

  16. [16]

    Fairness without demographics in repeated loss minimization,

    T. Hashimoto, M. Srivastava, H. Namkoong, and P. Liang, “Fairness without demographics in repeated loss minimization,” inProceedings of the 35th International Conference on Machine Learning (ICML), 2018, pp. 1929–1938

    2018

  17. [17]

    Distributionally robust multi-agent reinforcement learning for dynamic chute mapping,

    G. Liu, S. Iloglu, M. Caldara, J. W. Durham, and M. M. Zavlanos, “Distributionally robust multi-agent reinforcement learning for dynamic chute mapping,” inProceedings of the 42nd International Conference on Machine Learning (ICML), 2025, pp. 38 722–38 743

    2025

  18. [18]

    Monaco SUMO Traffic (MoST) scenario: A 3D mobility scenario for cooperative ITS,

    L. Codeca and J. H ¨arri, “Monaco SUMO Traffic (MoST) scenario: A 3D mobility scenario for cooperative ITS,”EPiC Series in Engineering, vol. 2, pp. 43–55, 2018

    2018

  19. [19]

    Multi-agent deep reinforcement learning for large-scale traffic signal control,

    T. Chu, J. Wang, L. Codec `a, and Z. Li, “Multi-agent deep reinforcement learning for large-scale traffic signal control,”IEEE Transactions on Intelligent Transportation Systems, vol. 21, no. 3, pp. 1086–1095, 2019

    2019

  20. [20]

    Proximal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,”arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  21. [21]

    Multi-agent reinforcement learning: Independent vs. cooperative agents,

    M. Tanet al., “Multi-agent reinforcement learning: Independent vs. cooperative agents,” inProceedings of the 10th International Conference on Machine Learning (ICML), 1993, pp. 330–337

    1993

  22. [22]

    Using a deep reinforcement learning agent for traffic signal control,

    W. Genders and S. Razavi, “Using a deep reinforcement learning agent for traffic signal control,”arXiv:1611.01142, 2016

    Pith/arXiv arXiv 2016

  23. [23]

    Traffic light control using deep policy-gradient and value-function-based reinforcement learning,

    S. S. Mousavi, M. Schukat, and E. Howley, “Traffic light control using deep policy-gradient and value-function-based reinforcement learning,” IET Intelligent Transport Systems, vol. 11, no. 7, pp. 417–423, 2017

    2017

  24. [24]

    IntelliLight: A reinforcement learning approach for intelligent traffic light control,

    H. Wei, G. Zheng, H. Yao, and Z. Li, “IntelliLight: A reinforcement learning approach for intelligent traffic light control,” inProceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD), 2018, pp. 2496–2505

    2018

  25. [25]

    Coordinated deep reinforcement learners for traffic light control,

    E. Van der Pol and F. A. Oliehoek, “Coordinated deep reinforcement learners for traffic light control,” inProceedings of the NIPS Workshop on Learning, Inference and Control of Multi-Agent Systems, vol. 8, 2016, pp. 21–38

    2016

  26. [26]

    Presslight: Learning max pressure control to coordinate traffic signals in arterial network,

    H. Wei, C. Chen, G. Zheng, K. Wu, V . Gayah, K. Xu, and Z. Li, “Presslight: Learning max pressure control to coordinate traffic signals in arterial network,” inProceedings of the 25th ACM SIGKDD Inter- national Conference on Knowledge Discovery & Data Mining (KDD), 2019, pp. 1290–1298

    2019

  27. [27]

    Colight: Learning network-level cooperation for traffic signal control,

    H. Wei, N. Xu, H. Zhang, G. Zheng, X. Zang, C. Chen, W. Zhang, Y . Zhu, K. Xu, and Z. Li, “Colight: Learning network-level cooperation for traffic signal control,” inProceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM), 2019, pp. 1913–1922

    2019

  28. [28]

    Multi-agent reinforcement learning for traffic signal control: Algorithms and robustness analysis,

    C. Wu, Z. Ma, and I. Kim, “Multi-agent reinforcement learning for traffic signal control: Algorithms and robustness analysis,” inProceedings of the 23rd IEEE International Conference on Intelligent Transportation Systems (ITSC), 2020, pp. 1–7

    2020

  29. [29]

    A robust and efficient multi-agent reinforcement learning framework for traffic signal control,

    S.-Y . Huang, H.-C. Chang, Y .-C. Chen, T.-H. Wei, I.-H. Yeh, S.-Y . Kuan, C.-Y . Wang, H.-H. Lee, and I.-C. Wu, “A robust and efficient multi-agent reinforcement learning framework for traffic signal control,” arXiv:2603.12096, 2026

    arXiv 2026

  30. [30]

    Robust multi-agent reinforcement learning with model uncertainty,

    K. Zhang, T. Sun, Y . Tao, S. Genc, S. Mallya, and T. Basar, “Robust multi-agent reinforcement learning with model uncertainty,” arXiv:2010.05445, 2020

    Pith/arXiv arXiv 2010

  31. [31]

    Safe and robust reinforcement learning: Principles and practice,

    T. Yamagata and R. Santos-Rodriguez, “Safe and robust reinforcement learning: Principles and practice,”arXiv:2403.18539, 2024

    Pith/arXiv arXiv 2024

  32. [32]

    Robust dynamic programming,

    G. Iyengar, “Robust dynamic programming,”Mathematics of Operations Research, vol. 30, no. 2, pp. 257–280, 2005

    2005

  33. [33]

    Robust control of markov decision pro- cesses with uncertain transition matrices,

    A. Nilim and L. El Ghaoui, “Robust control of markov decision pro- cesses with uncertain transition matrices,”Operations Research, vol. 53, no. 5, pp. 780–798, 2005

    2005

  34. [34]

    Policy gradients with variance related risk criteria,

    A. Tamar, Y . Glassner, and S. Mannor, “Policy gradients with variance related risk criteria,” inAdvances in Neural Information Processing Systems (NeurIPS), 2015, pp. 1653–1661

    2015

  35. [35]

    Robust adver- sarial reinforcement learning,

    L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta, “Robust adver- sarial reinforcement learning,” inProceedings of the 34th International Conference on Machine Learning (ICML), 2017

    2017

  36. [36]

    EPOpt: Learning robust neural network policies using model ensembles,

    A. Rajeswaran, K. Lowrey, E. Todorov, and S. Kakade, “EPOpt: Learning robust neural network policies using model ensembles,” in Proceedings of the 5th International Conference on Learning Repre- sentations (ICLR), 2017

    2017

  37. [37]

    Uncertainty-aware counterfactual traffic signal control with predictive safety and starvation-avoidance constraints using vision-based sensing,

    J. Bodagala and B. Bodagala, “Uncertainty-aware counterfactual traffic signal control with predictive safety and starvation-avoidance constraints using vision-based sensing,”arXiv:2602.07784, 2026

    Pith/arXiv arXiv 2026

  38. [38]

    Distributionally robust optimization,

    D. Kuhn, S. Shafiee, and W. Wiesemann, “Distributionally robust optimization,”Acta Numerica, vol. 34, pp. 579–804, 2025

    2025

  39. [39]

    Epoch-greedy algorithm for contextual ban- dits,

    J. Langford and T. Zhang, “Epoch-greedy algorithm for contextual ban- dits,” inAdvances in Neural Information Processing Systems (NeurIPS), 2008

    2008

  40. [40]

    Doubly robust policy evaluation and learning,

    M. Dud ´ık, J. Langford, and L. Li, “Doubly robust policy evaluation and learning,” inProceedings of the 28th International Conference on Machine Learning (ICML), 2011

    2011

  41. [41]

    Taming the monster: A fast and simple algorithm for contextual ban- dits,

    A. Agarwal, D. Hsu, S. Kale, J. Langford, L. Li, and R. E. Schapire, “Taming the monster: A fast and simple algorithm for contextual ban- dits,” inProceedings of the 31st International Conference on Machine Learning (ICML), 2014

    2014

  42. [42]

    Sample-efficient distributionally robust multi-agent reinforcement learning via online interaction,

    Z. U. Farhat, D. Ghosh, G. K. Atia, and Y . Wang, “Sample-efficient distributionally robust multi-agent reinforcement learning via online interaction,” inProceedings of the 14th International Conference on Learning Representations (ICLR), 2026

    2026

  43. [43]

    Recent de- velopment and applications of SUMO – simulation of urban MObility,

    D. Krajzewicz, J. Erdmann, M. Behrisch, and L. Bieker, “Recent de- velopment and applications of SUMO – simulation of urban MObility,” International Journal on Advances in Systems and Measurements, vol. 5, no. 3 & 4, pp. 128–138, 2012

    2012

  44. [44]

    Adaptive traffic signal control with actor-critic methods in a real-world traffic network with different traffic disruption events,

    M. Aslani, M. S. Mesgari, and M. Wiering, “Adaptive traffic signal control with actor-critic methods in a real-world traffic network with different traffic disruption events,”Transportation Research Part C: Emerging Technologies, vol. 85, pp. 732–752, 2017

    2017

  45. [45]

    Gradient-based learning applied to document recognition,

    Y . LeCun, L. Bottou, Y . Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,”Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 2002

    2002

  46. [46]

    Semi-supervised classification with graph convolutional networks,

    T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,”arXiv:1609.02907, 2016

    Pith/arXiv arXiv 2016

  47. [47]

    A survey on traffic signal control methods,

    H. Wei, G. Zheng, V . Gayah, and Z. Li, “A survey on traffic signal control methods,”arXiv:1904.08117, 2019

    Pith/arXiv arXiv 1904

  48. [48]

    Learning phase competition for traffic signal control,

    G. Zheng, Y . Xiong, X. Zang, J. Feng, H. Wei, H. Zhang, Y . Li, K. Xu, and Z. Li, “Learning phase competition for traffic signal control,” in Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM), 2019, pp. 1963–1972. Shuwei Peiis currently pursuing the Ph.D. degree with the Faculty of Science and Engineering...

    2019

  49. [49]

    He was a Postdoctoral Associate with the Laboratory for Information and Decision Systems, Massachusetts Institute of Technology, Cambridge, MA, USA

    He is an Assistant Professor with the De- partment of Electrical and Electronics Engineering, Bilkent University. He was a Postdoctoral Associate with the Laboratory for Information and Decision Systems, Massachusetts Institute of Technology, Cambridge, MA, USA. His research interests include developing the theoretical foundation of learning and autonomy ...

    2018