Large-Scale Traffic Signal Control Using a Novel Multi-Agent Reinforcement Learning

Liangjun Ke; Xiaoqiang Wang; Xinghua Chai; Zhimin Qiao

REVIEW 5 major objections 6 minor 49 references

Large-Scale Traffic Signal Control Using a Novel Multi-Agent Reinforcement Learning

T0 review · 5 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A decentralized multi-agent reinforcement learning algorithm, Co-DQL, makes each traffic signal model its neighbors' average action and provably converges to Nash Q-values, while simulator tests show much lower vehicle delays than four…

desk verdict Co-DQL is a sensible empirical MARL recipe for traffic signal control with large reported gains, but its convergence theorem is a tabular proof that does not cover the deep algorithm actually evaluated. read the letter →

arxiv 1908.03761 v2 pith:3QX237ZV submitted 2019-08-10 cs.LG cs.MAstat.ML

Xiaoqiang Wang , Liangjun Ke , Zhimin Qiao , Xinghua Chai This is my paper · ORCID

classification cs.LGcs.MAstat.ML MSC 68T0591A1590B20

keywords trafficsignalcontrolmulti-agentreinforcementlearningmeanfieldapproximationdoubleQ-learningupperconfidencebounddecentralizedconvergenceanalysiscooperativeagents

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that large-scale traffic signal control can be handled by many independent learning agents that still cooperate, without the exponential blow-up of centralized control. It introduces Co-DQL, a multi-agent reinforcement learning algorithm in which each signal learns two Q-value estimates, models neighboring intersections through their average action, and receives a reward that mixes its own delay with its neighbors' delays. The authors prove that, under conditions on the one-step games, both Q-estimates converge to Nash Q-values with probability one. On grid traffic simulators and a realistic road network, Co-DQL produces much shorter average vehicle delays and higher trip-completion rates than independent Q-learning, independent double Q-learning, DDPG, and a multi-agent A2C baseline. If correct, the method offers a scalable route to cooperative signal timing that needs only local communication.

What carries the argument

The central object is the mean-field action $\bar{a}_k$, the average one-hot action vector of agent $k$'s neighbors, which reduces the joint action-value function $Q_k(s_k, a_k, a_{-k})$ to $Q_k(s_k, a_k, \bar{a}_k)$, cutting the input dimension from $C^{N_k}$ to $C^2$. This decomposition is justified by a Taylor expansion whose remainder is treated as negligible. Around that core, the algorithm uses double estimators $Q^a$ and $Q^b$ to avoid over-estimation, a UCB policy for exploration, reward reallocation $\hat{r}_k = r_k + \alpha \sum_{i \in \mathcal{N}(k)} r_i$, and state sharing $\hat{s}_k = \langle s_k, \frac{1}{N_k}\sum_{i \in \mathcal{N}(k)} s_i\rangle$. The convergence proof works by showing that the difference $\Delta^{ba}_t = Q^b_t - Q^a_t$ converges to zero, so the double-estimator update inherits the mean-field Q-learning bound.

What would settle it

Record the joint payoff matrix for a fixed intersection state during training; if any stage game has a Nash equilibrium that is neither a global optimum nor a saddle point—for example a mixed equilibrium better than the pure global optimum—then Assumption 3 fails and the convergence theorem no longer applies. A second check is to track $Q^a$ and $Q^b$ during training: if their difference does not shrink to zero, the proof's central step fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that the Co-DQL update rule, Eq. 17, makes both Q-functions of every agent converge with probability one to the Nash Q-value of the underlying stochastic game, provided each state-action pair is visited infinitely often, rewards are bounded, the policy is greedy in the limit with infinite exploration, and every stage-game Nash equilibrium is either a global optimum or a saddle point. The same machinery—mean-field action averaging, double estimators, reallocated rewards, and shared local states—is then claimed to translate into a practical large-scale traffic signal controller: in three grid-flow scenarios and one realistic asymmetric road network, Co-DQL reports the lowest average delay, the shortest queues, and the highest trip-arrival rate among the five tested algorithms.

Load-bearing premise

The convergence theorem rests on Assumption 3, which requires every one-step game encountered in training to have a Nash equilibrium that is either a global optimum or a saddle point; the paper does not show that traffic signal control satisfies this condition.

Editorial extensions

If this is right

An intersection's Q-network input size stays constant as the road network grows, because only the average neighbor action and average neighbor state are fed in.
The double-estimator structure should prevent the positive bias of independent Q-learning; in the reported experiments independent double Q-learning always beats independent Q-learning and Co-DQL beats both.
If Theorem 1 holds, Co-DQL carries a per-agent convergence guarantee to Nash Q-values, which plain independent Q-learning does not offer.
The method can be executed with local communication only, since the mean action and shared state require no global coordination at run time.
The reported simulator results imply average vehicle delay about half that of the multi-agent A2C baseline in the three grid-flow scenarios, with a higher trip-arrival rate on the realistic road network.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same design—mean-field opponent averaging plus double estimators—could transfer to other cooperative multi-agent settings with many homogeneous agents, such as automated fleet dispatch or elevator coordination, where the linear reward-mixing assumption would need re-testing.
A direct test of Assumption 3 would be to record the one-step payoff matrix at several traffic states and check whether every Nash equilibrium is a global optimum or a saddle point; the paper does not perform this check.
The paper's own closing note suggests that linear reward allocation may bias control toward intersections with more legs; a natural extension is a degree-normalized or nonlinear reward-sharing rule.
Because the mean-field approximation treats neighbor actions only through their average, its quality should degrade when neighboring intersections are highly heterogeneous; comparisons on strongly asymmetric networks would clarify that boundary.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is an engineering paper, not a theory paper. The empirical recipe is coherent and the reported gains over MA2C are large, but the convergence proof is a tabular proof that does not cover the deep algorithm you actually run, and the evaluation has some selection bias. The paper is worth reading for the TSC application, not for the theory.

What's new: Co-DQL is a specific combination of known pieces—double Q-learning with UCB exploration, mean field action approximation, a linear reward-sharing term, and local state averaging. The particular assembly is new, and the experiments on the grid simulator and on a 49-intersection SUMO network show consistent improvements over IQL, IDQL, DDPG, and MA2C across several metrics. The authors also openly list their hyperparameters and describe the simulators well enough to reproduce roughly. They even flag in the conclusion that the linear reward allocation can bias solutions for intersections with different degrees, which is honest.

Soft spots: the convergence claim is the biggest. Theorem 1 states convergence for the update in Eq. 17, which is a tabular synchronous Q-learning update with two estimators. Algorithm 2 actually trains neural networks with a replay buffer, minibatch SGD, soft target updates, UCB exploration, and modified reward/state inputs. None of these appear in the theorem. So the proof does not support the algorithm as implemented. There is also a technical slip in the proof: the bound on the auxiliary process Delta^ba comes out as ||Delta^ba|| without the gamma factor, which fails the strict contraction condition in Lemma 1. And the mean-field contraction is cited to [17], but that is Claus and Boutilier (1998); the result the authors need is in [27]. So the theory section is not reliable.

Assumption 3 is also strong: it says every stage game's Nash equilibrium must be a global optimum or a saddle point, which is not derived from traffic dynamics or checked in the experiments. If that assumption fails, the theorem does not apply. The experiments are also less controlled than they could be: the authors select the best model from training for each method, evaluate over 100 episodes in the simplified simulator but only 10 in the SUMO network, and do not report significance tests or any non-RL baselines like fixed-time or max pressure. That said, the differences are large enough that the qualitative conclusion—Co-DQL outperforms these four MARL baselines on these scenarios—is plausible.

Who is this for: someone working on MARL for traffic signal control who wants a practical baseline and a clear description of one way to combine mean field with double Q-learning. A theorist will be disappointed. Should it be refereed? Yes, as an applied paper, with requests to fix the theory section (or remove it), add non-RL baselines, and use a proper evaluation protocol. I would not desk-reject it.

Referee Report

5 major / 6 minor

Summary. The paper proposes Co-DQL, a decentralized multi-agent reinforcement learning method for large-scale traffic signal control. The method combines independent double Q-learning with UCB exploration, mean-field approximation of neighboring agents' actions, a neighborhood-based reward reallocation rule, and local state sharing. The authors provide a convergence analysis intended to show that the two Q-functions converge to Nash Q-values, and they evaluate Co-DQL in a simplified grid simulator and a 49-intersection SUMO network against IQL, IDQL, DDPG, and MA2C. The central claims are that Co-DQL converges to Nash Q-values under stated assumptions and empirically outperforms state-of-the-art decentralized MARL algorithms on delay and related traffic metrics.

Significance. If the claims are established, Co-DQL would be a practically relevant, scalable decentralized MARL method for TSC: the paper combines double estimators, mean-field opponent modeling, and local information sharing in a way that is natural for large networks, and it reports substantial delay reductions (e.g., 36.98 vs. 71.55 time steps against MA2C in the global-random scenario, Table II). The paper also ships an open-source implementation and evaluates on several scenarios, which is a strength. However, the convergence theorem as stated does not cover the deep, replay-buffer-based algorithm that is actually implemented and tested, the proof has a gap in satisfying the contraction condition, and the empirical evaluation selects the best training model and lacks significance tests and non-RL baselines. These issues are substantial but appear repairable within the manuscript's scope.

major comments (5)

[Section III-C, Theorem 1 and Eq. (17)] The theorem states that Q^a and Q^b 'as updated by the rule of Algorithm 2 in Eq. 17' converge to the Nash Q-values, but Eq. 17 is a tabular, synchronous-style update with a scalar learning rate. Algorithm 2, by contrast, updates deep neural networks via minibatch SGD, a replay buffer, target networks, and soft target updates (Eqs. 15-16), and it uses UCB exploration and the modified state/reward of Eqs. 13-14. None of these components appear in the theorem's model or proof, so the convergence guarantee does not cover the implemented Co-DQL that is evaluated in Section V. The authors should either provide a convergence result for the deep variant under explicit assumptions, or clearly restrict Theorem 1 to the tabular algorithm and describe the deep implementation as a heuristic approximation.
[Section III-C, proof of Theorem 1] In the bounding of E[F^ba_t], both Case 1 and Case 2 conclude with |E[F^ba_t]| <= ||Delta^ba_t||. However, Lemma 1's condition 3 requires a contraction with gamma < 1, namely ||E[F_t|I_t]||_W <= gamma ||Delta_t||_W + c_t. The displayed derivation actually has a gamma factor in front of the expectation and should yield gamma ||Delta^ba_t||; as written, the proof does not establish the strict contraction required by the lemma. This is a load-bearing gap, although it appears to be repairable.
[Section III-C, Assumption 3] Assumption 3 restricts every stage-game Nash equilibrium to be either a global optimum or a saddle point, but this property is neither derived from the traffic dynamics of Section IV nor verified in the experiments. For reward functions with congestion terms, such as Eq. 26, it is plausible that some stage games have equilibria outside these two classes, and the theorem's conclusion collapses if Assumption 3 fails. The paper should explain how Assumption 3 can be checked for the TSC model, or weaken the convergence claim to hold only when the assumption is separately established.
[Section III-B, Eq. (12) and Section IV-A] The mean-field approximation drops the Taylor remainder R_k(a_l) by citing prior work, but no argument is given that this remainder is small for the specific TSC Q-functions used here. Similarly, the claim that the compressed state (shared local state plus mean action) is sufficient is asserted around Eq. (21) rather than proved. These two approximations are load-bearing for the reduction from joint action space C^{N_k} to C^2 and for the Markov assumption, so the paper should justify them for the traffic setting or explicitly label them as empirical assumptions.
[Section V-B and Tables II-V] The evaluation protocol selects the best model obtained during training for each algorithm, reports means and standard deviations without significance tests, and does not include multiple independent training runs. In addition, no non-RL baselines (e.g., fixed-time control or max-pressure) are included. This protocol makes the headline claim that Co-DQL 'outperforms the state-of-the-art decentralized MARL algorithms' less robust than the tables suggest. Please add paired statistical tests across seeds, report results from all training runs or a fixed checkpoint selection rule, and compare with standard non-RL TSC methods.

minor comments (6)

[Algorithm 2, line 11] The target-network update is printed as 'φ−,k ← τφk + (1−τ)φ,k'; the last term should be φ−,k.
[Eq. (8) and Algorithm 2] The soft target update is written with θ and θ′ in Eq. (8) but with φ and φ− in Algorithm 2; please unify the notation.
[Eq. (9)] The UCB exploration rule uses visit counts R_{s_k} and R_{s_k,c} for discrete states, but in the deep implementation the state becomes continuous after state sharing; please clarify how these counts are computed in that setting.
[Section VI, Conclusion] The acknowledged limitation that a constant α in Eq. (13) cannot capture nonlinear interactions and may overweight intersections with more legs is directly relevant to the reward-allocation claim and deserves a more prominent place in the evaluation discussion.
[Section V-B, Figs. 10 and 11] The bar charts would be easier to interpret if they included error bars or confidence intervals, since the text discusses differences across scenarios that may be within noise.
[Section IV-A] The open-source code link contains a space ('larger real net') and appears not to be a valid URL; please provide the correct repository address.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Co-DQL's empirical comparison is not fitted to the target, and the theoretical claim is a conditional theorem supported by external citations, not by self-citation.

full rationale

The derivation chain of Co-DQL is not circular. The algorithm's components—independent double Q-learning, mean-field approximation, reward allocation, and local state sharing—are defined from standard RL/MARL primitives, and the empirical claims are evaluated against independent baselines (IQL, IDQL, DDPG, MA2C) on fixed simulator settings, with no target delay value fitted into the method. The convergence analysis (Section III-C) is explicitly conditional: Theorem 1 assumes Assumptions 1-3 and imports the mean-field contraction from [27] and the double-Q framework from [24]; neither citation is to the present authors, so the self-citation patterns do not apply. Assumption 3 is a stated condition on the stage game, not a conclusion obtained from Co-DQL itself. The mean-field Taylor remainder is imported as an approximation from [27], which is an external published source. There are correctness concerns—notably that Eq. 17 is a tabular update while Algorithm 2 uses neural networks, replay buffers, and target networks, so Theorem 1 does not cover the implemented deep variant, and the proof of Delta^ba lacks the required gamma contraction factor—but these are proof gaps and unsupported generalizations, not instances of a prediction reducing to its input by construction. The appended limitation about linear reward allocation reinforces that the reward mechanism is a fixed heuristic, not a fitted target. Hence no significant circularity.

Assumptions & free parameters 9 free parameters · 6 assumptions · 0 invented entities

The empirical claims rest on tuned hyperparameters and simulated environments. The convergence theorem rests on stochastic approximation conditions, on Assumption 3, and on an imported mean-field contraction; none of these are verified against the actual TSC dynamics. No new physical entities are introduced.

free parameters (9)

Reward allocation factor alpha in Eq. 13 = 1/n where n is the number of neighboring agents
Hand-set in Section V-A to balance selfishness and cooperation; no sensitivity analysis is reported.
Regularization rate beta in reward Eq. 26 = 0.2 veh/s
Set in Section V-C to scale delay and queue terms in the reward.
Discount factor gamma = 0.95
Set in Section V-A; affects all Q-value targets.
Soft target update rate tau = 0.01
Set in Section V-A for target network smoothing.
Q-network learning rate = 0.0001 (Adam)
Set in Section V-A.
Mini-batch size = 1024
Set in Section V-A.
Replay buffer size = 5e5
Set in Section V-A.
Wave, wait and reward normalization factors = 5 veh, 100 s, 2000 veh
Listed in Section V-C for scaling state and reward quantities.
Neighborhood size N(k) = All agents in simplified setting; spatial neighborhood in SUMO setting
Declared user-specific in footnote 1; controls mean-field average and reward reallocation scope.

assumptions (6)

standard math Lemma 1 stochastic approximation conditions (weighted maximum norm, square-summable step sizes)
Stated in Section III-C as the convergence backbone for Theorem 1.
domain assumption Assumption 1: every action-value pair is visited infinitely often and rewards are bounded
Stated in Section III-C; standard but unverified for the neural network implementation on continuous TSC states.
domain assumption Assumption 2: agent policy is Greedy in the Limit with Infinite Exploration (GLIE)
Stated in Section III-C; UCB is expected to become greedy in the limit but is not formally proven GLIE here.
ad hoc to paper Assumption 3: at every stage game the Nash equilibrium is a global optimum or a saddle point
Section III-C; the key structural condition for the convergence proof, not derived from traffic dynamics.
domain assumption Mean-field Taylor expansion remainder is negligible
Invoked in Eq. 12 in Section III-B and imported from [27]; not verified for TSC value functions.
ad hoc to paper Global traffic state is Markov and the compressed state (shared local state, mean action) is sufficient per agent
Eq. 21 in Section IV-A asserts global Markovianity; the sufficiency of the compressed observation is assumed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Large-Scale Traffic Signal Control Using a Novel Multi-Agent Reinforcement Learning." pith.science (2026). https://pith.science/paper/3QX237ZV

@misc{pith2026190803761,
  author       = {Pith},
  title        = {Pith review of: Large-Scale Traffic Signal Control Using a Novel Multi-Agent Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3QX237ZV}},
  note         = {Machine review of arXiv:1908.03761}
}

read the original abstract

Finding the optimal signal timing strategy is a difficult task for the problem of large-scale traffic signal control (TSC). Multi-Agent Reinforcement Learning (MARL) is a promising method to solve this problem. However, there is still room for improvement in extending to large-scale problems and modeling the behaviors of other agents for each individual agent. In this paper, a new MARL, called Cooperative double Q-learning (Co-DQL), is proposed, which has several prominent features. It uses a highly scalable independent double Q-learning method based on double estimators and the UCB policy, which can eliminate the over-estimation problem existing in traditional independent Q-learning while ensuring exploration. It uses mean field approximation to model the interaction among agents, thereby making agents learn a better cooperative strategy. In order to improve the stability and robustness of the learning process, we introduce a new reward allocation mechanism and a local state sharing method. In addition, we analyze the convergence properties of the proposed algorithm. Co-DQL is applied on TSC and tested on a multi-traffic signal simulator. According to the results obtained on several traffic scenarios, Co- DQL outperforms several state-of-the-art decentralized MARL algorithms. It can effectively shorten the average waiting time of the vehicles in the whole road system.

Figures

Figures reproduced from arXiv: 1908.03761 by the authors.

**Figure 2.** Illustration of the grid traffic signal system simulator. (a) global random traffic flow, (b) double-ring traffic flow, and (c) four-ring traffic flow. Each [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** Overall view of the realistic road network with asymmetric geometry. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Local view of two adjacent intersections of the realistic road network. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 5.** Figure 5: Illustration of mean reward change curve of signal agents using random strategy in various traffic flows scenarios. (a) global random traffic flow [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

**Figure 6.** Figure 6: Reward curve of signal agent during training in the global random [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

**Figure 7.** Figure 7: Reward curve of signal agent during training in the double-ring traffic [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

**Figure 8.** Figure 8: Reward curve of signal agent during training in the four-ring traffic [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

**Figure 10.** Figure 10: Mean episode reward comparison for testing the corresponding model [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]

**Figure 11.** Figure 11: Mean delay time comparison for testing the corresponding model in [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 29 canonical work pages

[17]

The dynamics of reinforcement learning in cooperative multiagent systems,

C. Claus and C. Boutilier, “The dynamics of reinforcement learning in cooperative multiagent systems,” AAAI/IAAI, vol. 1998, no. 746-752, p. 2, 1998

1998
[27]

Mean ﬁeld multi-agent reinforcement learning,

Y . Yang, R. Luo, M. Li, M. Zhou, W. Zhang, and J. Wang, “Mean ﬁeld multi-agent reinforcement learning,” arXiv preprint arXiv:1802.05438 , 2018

arXiv 2018
[1]

A survey on reinforcement learning models and algorithms for trafﬁc signal control,

K.-L. A. Yau, J. Qadir, H. L. Khoo, M. H. Ling, and P. Komisarczuk, “A survey on reinforcement learning models and algorithms for trafﬁc signal control,” ACM Computing Surveys (CSUR) , vol. 50, no. 3, p. 34, 2017

2017
[2]

Optimal bidding strategies in electricity markets using reinforcement learning,

Q. Wu and J. Guo, “Optimal bidding strategies in electricity markets using reinforcement learning,” Electric Power Components and Systems, vol. 32, no. 2, pp. 175–192, 2004

work page 2004
[3]

Trafﬁc network micro-simulation model and control algorithm based on approximate dynamic program- ming,

B. Yin, M. Dridi, and A. El Moudni, “Trafﬁc network micro-simulation model and control algorithm based on approximate dynamic program- ming,” IET Intelligent Transport Systems , vol. 10, no. 3, pp. 186–196, 2016

work page 2016
[4]

Trafﬁc signal timing manual

P. Koonce and L. Rodegerdts, “Trafﬁc signal timing manual.” United States. Federal Highway Administration, Tech. Rep., 2008

work page 2008
[5]

Trafﬁc signal timing optimisation based on genetic algorithm approach, including drivers routing,

H. Ceylan and M. G. Bell, “Trafﬁc signal timing optimisation based on genetic algorithm approach, including drivers routing,” Transportation Research Part B: Methodological , vol. 38, no. 4, pp. 329–342, 2004

work page 2004
[6]

Swarm intelligence for trafﬁc light scheduling: Application to real urban areas,

J. Garc ´ıa-Nieto, E. Alba, and A. C. Olivera, “Swarm intelligence for trafﬁc light scheduling: Application to real urban areas,” Engineering Applications of Artiﬁcial Intelligence, vol. 25, no. 2, pp. 274–283, 2012

work page 2012

Show all 49 references

[7]

Two-stage fuzzy logic controller for signalized intersection,

J. Qiao, N. Yang, and J. Gao, “Two-stage fuzzy logic controller for signalized intersection,” IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans , vol. 41, no. 1, pp. 178–184, 2010

2010
[8]

Neural networks for real- time trafﬁc signal control,

D. Srinivasan, M. C. Choy, and R. L. Cheu, “Neural networks for real- time trafﬁc signal control,” IEEE Transactions on intelligent transporta- tion systems, vol. 7, no. 3, pp. 261–272, 2006

2006
[9]

R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction . MIT press, 2018

2018
[10]

Intelligent trafﬁc light control,

M. Wiering, J. v. Veenen, J. Vreeken, and A. Koopman, “Intelligent trafﬁc light control,” 2004

2004
[11]

Reinforcement learning with function approximation for trafﬁc signal control,

L. Prashanth and S. Bhatnagar, “Reinforcement learning with function approximation for trafﬁc signal control,” IEEE Transactions on Intelli- gent Transportation Systems, vol. 12, no. 2, pp. 412–421, 2010

2010
[12]

Deep learning,

Y . LeCun, Y . Bengio, and G. Hinton, “Deep learning,” nature, vol. 521, no. 7553, p. 436, 2015

2015
[13]

Human-level control through deep reinforcement learning,

V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al. , “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, p. 529, 2015

2015
[14]

Continuous control with deep reinforcement learning,

T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” arXiv preprint arXiv:1509.02971 , 2015

2015 arXiv
[15]

Intellilight: A reinforcement learning approach for intelligent trafﬁc light control,

H. Wei, G. Zheng, H. Yao, and Z. Li, “Intellilight: A reinforcement learning approach for intelligent trafﬁc light control,” in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . ACM, 2018, pp. 2496–2505

2018
[16]

Deep deterministic policy gradient for urban trafﬁc light control,

N. Casas, “Deep deterministic policy gradient for urban trafﬁc light control,” arXiv preprint arXiv:1703.09035 , 2017

2017 arXiv
[18]

A distributed approach for coordination between trafﬁc lights based on game theory

S. Shamshirband, “A distributed approach for coordination between trafﬁc lights based on game theory.” Int. Arab J. Inf. Technol. , vol. 9, no. 2, pp. 148–153, 2012

2012
[19]

Reinforcement learning-based multi-agent system for network trafﬁc signal control,

I. Arel, C. Liu, T. Urbanik, and A. Kohls, “Reinforcement learning-based multi-agent system for network trafﬁc signal control,” IET Intelligent Transport Systems, vol. 4, no. 2, pp. 128–135, 2010

2010
[20]

Trafﬁc light control in non-stationary environments based on multi agent q-learning,

M. Abdoos, N. Mozayani, and A. L. Bazzan, “Trafﬁc light control in non-stationary environments based on multi agent q-learning,” in 2011 14th International IEEE conference on intelligent transportation systems (ITSC). IEEE, 2011, pp. 1580–1585

2011
[21]

Multi-agent reinforcement learning: Independent vs. cooper- ative agents,

M. Tan, “Multi-agent reinforcement learning: Independent vs. cooper- ative agents,” in Proceedings of the tenth international conference on machine learning, 1993, pp. 330–337

1993
[22]

Multiagent rein- forcement learning for urban trafﬁc control using coordination graphs,

L. Kuyer, S. Whiteson, B. Bakker, and N. Vlassis, “Multiagent rein- forcement learning for urban trafﬁc control using coordination graphs,” in Joint European Conference on Machine Learning and Knowledge Discovery in Databases . Springer, 2008, pp. 656–671

2008
[23]

Multi-agent deep reinforcement learning for large-scale trafﬁc signal control,

T. Chu, J. Wang, L. Codec `a, and Z. Li, “Multi-agent deep reinforcement learning for large-scale trafﬁc signal control,” IEEE Transactions on Intelligent Transportation Systems, 2019

2019
[24]

Double q-learning,

H. V . Hasselt, “Double q-learning,” in Advances in Neural Information Processing Systems, 2010, pp. 2613–2621

2010
[25]

Finite-time analysis of the multiarmed bandit problem,

P. Auer, N. Cesa-Bianchi, and P. Fischer, “Finite-time analysis of the multiarmed bandit problem,” Machine learning , vol. 47, no. 2-3, pp. 235–256, 2002

2002
[26]

H. E. Stanley, Phase transitions and critical phenomena . Clarendon Press, Oxford, 1971

1971
[28]

Deep reinforcement learning with double q-learning,

H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning,” in Thirtieth AAAI conference on artiﬁcial intelligence, 2016

2016
[29]

The optimizers curse: Skepticism and postdecision surprise in decision analysis,

J. E. Smith and R. L. Winkler, “The optimizers curse: Skepticism and postdecision surprise in decision analysis,” Management Science , vol. 52, no. 3, pp. 311–322, 2006

2006
[30]

Stochastic games,

L. S. Shapley, “Stochastic games,” Proceedings of the national academy of sciences, vol. 39, no. 10, pp. 1095–1100, 1953

1953
[31]

Markov games as a framework for multi-agent rein- forcement learning,

M. L. Littman, “Markov games as a framework for multi-agent rein- forcement learning,” in Machine learning proceedings 1994 . Elsevier, 1994, pp. 157–163

1994
[32]

Multi-agent reinforce- ment learning for trafﬁc signal control,

K. Prabuchandran, H. K. AN, and S. Bhatnagar, “Multi-agent reinforce- ment learning for trafﬁc signal control,” in 17th International IEEE Conference on Intelligent Transportation Systems (ITSC) . IEEE, 2014, pp. 2529–2534

2014
[33]

Analyzing and visualizing multiagent rewards in dynamic and stochastic domains,

A. K. Agogino and K. Tumer, “Analyzing and visualizing multiagent rewards in dynamic and stochastic domains,” Autonomous Agents and Multi-Agent Systems, vol. 17, no. 2, pp. 320–338, 2008

2008
[34]

Multi-agent actor-critic for mixed cooperative-competitive environ- ments,

R. Lowe, Y . Wu, A. Tamar, J. Harb, O. P. Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environ- ments,” in Advances in Neural Information Processing Systems , 2017, pp. 6379–6390. Copyright (c) 2020 IEEE. Personal use is permitted. For a...

2017
[35]

Efﬁcient ridesharing order dispatching with mean ﬁeld multi-agent reinforcement learning,

M. Li, Z. Qin, Y . Jiao, Y . Yang, J. Wang, C. Wang, G. Wu, and J. Ye, “Efﬁcient ridesharing order dispatching with mean ﬁeld multi-agent reinforcement learning,” in The World Wide Web Conference . ACM, 2019, pp. 983–994

2019
[36]

Convergence of stochastic iterative dynamic programming algorithms,

T. Jaakkola, M. I. Jordan, and S. P. Singh, “Convergence of stochastic iterative dynamic programming algorithms,” in Advances in neural information processing systems , 1994, pp. 703–710

1994
[37]

A uniﬁed analysis of value-function- based reinforcement-learning algorithms,

C. Szepesv ´ari and M. L. Littman, “A uniﬁed analysis of value-function- based reinforcement-learning algorithms,” Neural Computation, vol. 11, no. 8, pp. 2017–2060, 1999

2017
[38]

Adaptive trafﬁc signal control with actor-critic methods in a real-world trafﬁc network with different trafﬁc disruption events,

M. Aslani, M. S. Mesgari, and M. Wiering, “Adaptive trafﬁc signal control with actor-critic methods in a real-world trafﬁc network with different trafﬁc disruption events,” Transportation Research Part C: Emerging Technologies, vol. 85, pp. 732–752, 2017

2017
[39]

Multiagent rein- forcement learning for integrated network of adaptive trafﬁc signal controllers (marlin-atsc): methodology and large-scale application on downtown toronto,

S. El-Tantawy, B. Abdulhai, and H. Abdelgawad, “Multiagent rein- forcement learning for integrated network of adaptive trafﬁc signal controllers (marlin-atsc): methodology and large-scale application on downtown toronto,” IEEE Transactions on Intelligent Transportation Systems...

2013
[40]

Adaptive group-based signal control by reinforcement learning,

J. Jin and X. Ma, “Adaptive group-based signal control by reinforcement learning,” Transportation Research Procedia , vol. 10, pp. 207–216, 2015

2015
[41]

Cooperative deep reinforcement learning for large-scale trafﬁc grid signal control,

T. Tan, F. Bao, Y . Deng, A. Jin, Q. Dai, and J. Wang, “Cooperative deep reinforcement learning for large-scale trafﬁc grid signal control,” IEEE Transactions on Cybernetics , 2019

2019
[42]

Large-scale trafﬁc grid signal control with regional reinforcement learning,

T. Chu, S. Qu, and J. Wang, “Large-scale trafﬁc grid signal control with regional reinforcement learning,” in 2016 American Control Conference (ACC). IEEE, 2016, pp. 815–820

2016
[43]

Monaco sumo trafﬁc (most) scenario: A 3d mobility scenario for cooperative its,

L. Codeca and J. H ¨arri, “Monaco sumo trafﬁc (most) scenario: A 3d mobility scenario for cooperative its,” in SUMO 2018, SUMO User Conference, Simulating Autonomous and Intermodal Transport Systems , 2018

2018
[44]

Agent-based trafﬁc ﬂow optimization at multiple signalized intersec- tions,

K. T. K. Teo, K. B. Yeo, Y . K. Chin, H. S. E. Chuo, and M. K. Tan, “Agent-based trafﬁc ﬂow optimization at multiple signalized intersec- tions,” in 2014 8th Asia Modelling Symposium. IEEE, 2014, pp. 21–26

2014
[45]

Openai gym,

G. Brockman, V . Cheung, L. Pettersson, J. Schneider, J. Schul- man, J. Tang, and W. Zaremba, “Openai gym,” arXiv preprint arXiv:1606.01540, 2016

2016 arXiv
[46]

Sim2real viewpoint invariant visual servoing by recurrent control,

F. Sadeghi, A. Toshev, E. Jang, and S. Levine, “Sim2real viewpoint invariant visual servoing by recurrent control,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 4691–4699

2018
[47]

Feudal networks for hierarchical rein- forcement learning,

A. S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg, D. Silver, and K. Kavukcuoglu, “Feudal networks for hierarchical rein- forcement learning,” inProceedings of the 34th International Conference on Machine Learning-Volume 70 . JMLR. org, 2017, pp. 3540–3549

2017
[48]

Max pressure control of a network of signalized intersec- tions,

P. Varaiya, “Max pressure control of a network of signalized intersec- tions,” Transportation Research Part C: Emerging Technologies, vol. 36, pp. 177–195, 2013

2013
[49]

Distributed trafﬁc signal control using the cell transmission model via the alternating direction method of multipliers,

S. Timotheou, C. G. Panayiotou, and M. M. Polycarpou, “Distributed trafﬁc signal control using the cell transmission model via the alternating direction method of multipliers,” IEEE Transactions on Intelligent Transportation Systems, vol. 16, no. 2, pp. 919–933, 2014. Copyrigh...

2014

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [17]

The dynamics of reinforcement learning in cooperative multiagent systems,

C. Claus and C. Boutilier, “The dynamics of reinforcement learning in cooperative multiagent systems,” AAAI/IAAI, vol. 1998, no. 746-752, p. 2, 1998

1998

[2] [27]

Mean ﬁeld multi-agent reinforcement learning,

Y . Yang, R. Luo, M. Li, M. Zhou, W. Zhang, and J. Wang, “Mean ﬁeld multi-agent reinforcement learning,” arXiv preprint arXiv:1802.05438 , 2018

arXiv 2018

[3] [1]

A survey on reinforcement learning models and algorithms for trafﬁc signal control,

K.-L. A. Yau, J. Qadir, H. L. Khoo, M. H. Ling, and P. Komisarczuk, “A survey on reinforcement learning models and algorithms for trafﬁc signal control,” ACM Computing Surveys (CSUR) , vol. 50, no. 3, p. 34, 2017

2017

[4] [2]

Optimal bidding strategies in electricity markets using reinforcement learning,

Q. Wu and J. Guo, “Optimal bidding strategies in electricity markets using reinforcement learning,” Electric Power Components and Systems, vol. 32, no. 2, pp. 175–192, 2004

work page 2004

[5] [3]

Trafﬁc network micro-simulation model and control algorithm based on approximate dynamic program- ming,

B. Yin, M. Dridi, and A. El Moudni, “Trafﬁc network micro-simulation model and control algorithm based on approximate dynamic program- ming,” IET Intelligent Transport Systems , vol. 10, no. 3, pp. 186–196, 2016

work page 2016

[6] [4]

Trafﬁc signal timing manual

P. Koonce and L. Rodegerdts, “Trafﬁc signal timing manual.” United States. Federal Highway Administration, Tech. Rep., 2008

work page 2008

[7] [5]

Trafﬁc signal timing optimisation based on genetic algorithm approach, including drivers routing,

H. Ceylan and M. G. Bell, “Trafﬁc signal timing optimisation based on genetic algorithm approach, including drivers routing,” Transportation Research Part B: Methodological , vol. 38, no. 4, pp. 329–342, 2004

work page 2004

[8] [6]

Swarm intelligence for trafﬁc light scheduling: Application to real urban areas,

J. Garc ´ıa-Nieto, E. Alba, and A. C. Olivera, “Swarm intelligence for trafﬁc light scheduling: Application to real urban areas,” Engineering Applications of Artiﬁcial Intelligence, vol. 25, no. 2, pp. 274–283, 2012

work page 2012