{"id":"6527d8ae-7e3b-4cc5-b537-8da52ed32427","arxiv_id":"1908.02673","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Deep reinforcement learning for traffic signal control loses its ability to learn when trained at high network densities, while free-flow-trained policies and a two-example supervised policy outperform the longest-queue-first benchmark.","lead":"This paper tests machine learning methods for controlling traffic lights in a large idealized grid, using a simple cellular automaton traffic model. It finds that deep reinforcement learning fails when trained under congested conditions, and that a policy trained on just two extreme examples beats a standard greedy benchmark.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The congested-network-property explanation assumes independent Bernoulli queues that CA rule 184 violates at high density; the observed DRL failure stands, but the general 'nothing to learn' conclusion and the advice to discard all congested data do not follow from the presented derivation.","rationale":"The paper's main empirical finding — that DRL policies trained at high occupancy fail, while free-flow training works best — is directly visible in Figs. 6–7 and does not depend on the theoretical explanation. What is load-bearing is the generalization from that observation to a universal 'congested network property' and the recommendation to discard all congested data. The derivation of the lower bound in the Discussion models cell occupancies as independent Bernoulli and queue sizes as independent binomials, then computes the min of the two. This is not checked against the CA dynamics and is least credible in the congested regime (k > 0.75) where it is used: rule 184 conserves vehicles and produces correlated platoon/hole patterns, and the derivation's own proviso that outgoing approaches do not block traffic is violated under congestion. If the true lower bound is zero or has a different shape, the 'nothing to learn' mechanism is unsupported. That does not overturn the simulator-specific empirical result, but it does mean the paper's broader claims are conditional, exactly as the reader concluded. I also note that the abstract's 'optimal' for free-flow-trained DRL is stronger than the paper's own 'only competitive' wording in Section 5.3; fixing that wording would strengthen the paper.","tokens_in":12341,"tokens_out":11715,"duration_ms":132237,"concrete_test":"Recompute the lower-bound curves in Fig. 10 directly from the CA simulator: for densities k = 0.5, 0.6, 0.7, 0.8, and 0.9, initialize the torus at those densities, run the smallest-queue-first policy for at least 10^4 time steps, and record the empirical distribution of per-intersection discharge per green interval. Compare the 2.5th percentile of this distribution with the min-of-two-independent-binomials lower bound. If the empirical percentile is significantly below the Bernoulli bound (or zero) at k >= 0.75, the congested-network property is an artifact of the independence assumption and the explanation for DRL failure must be revised; if it matches, the property is upheld.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central explanation for DRL failure under congestion is the 'congested network property': even a bad policy yields throughput above a nonzero lower bound, so flow becomes policy-independent and gradients vanish. The lower bound is derived in the Discussion by modeling each cell as an independent Bernoulli(k) and treating queue sizes on the two approaches as independent binomials, so the throughput of a 'smallest-queue-first' policy is a linear transform of the min of two independent binomials. This assumption is not verified against the CA rule-184 dynamics and is least plausible precisely in the regime (k > 0.75) where it is used: rule 184 conserves vehicle number, and steady-state configurations at high density consist of platoons separated by holes, so cell occupancies are strongly correlated; moreover, the derivation explicitly assumes 'provided outgoing approaches do not block traffic,' which fails under congestion, where downstream links are typically full and spillback limits any intersection's discharge. If the true lower bound in the CA network is zero or has a different shape, the 'nothing to learn' mechanism is unsupported, and the practical advice to discard all congested training data (which the authors generalize to all DRL methods) does not follow. The empirical DRL failure in the specific simulator is, however, directly observed and independent of this derivation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies steady-state signal control in a homogeneous toroidal grid network using Cellular Automaton rule 184. It compares supervised learning, random search, and a proposed continuing-task policy-gradient algorithm (REINFORCE-TD) against the longest-queue-first (LQF) benchmark. The main empirical findings are that a supervised policy trained on only two extreme examples outperforms LQF, random search can find competitive policies, and the occupancy level used during training is decisive: DRL policies trained at occupancies above about 75% perform very poorly and appear unable to learn. The paper proposes a 'congested network property'—a nonzero lower bound on intersection throughput that becomes policy-independent under congestion—as the explanation, and recommends that DRL practitioners discard congested training data. The theoretical explanation and the broad practical advice are the main points in need of scrutiny.","tokens_in":12654,"tokens_out":7851,"duration_ms":87893,"significance":"If the central mechanism were established, the paper would make an important contribution by identifying training occupancy as a decisive factor for DRL traffic-signal control on large networks. The manuscript has clear strengths: the simulation protocol is described in enough detail to be reproduced, the REINFORCE-TD algorithm is explicitly specified, and MFD comparisons are reported with confidence intervals. The supervised-learning result with two examples is striking and well supported within the paper's setting. However, the significance of the paper currently rests on two unsupported extrapolations: the heuristic derivation of the congested network property, which is not verified against the actual CA dynamics, and the generalization of the empirical failure to all DRL methods. The paper offers a falsifiable prediction about the shape of MFD lower bounds, but that prediction is not yet validated; the derivation also does not establish a true lower bound for all policies.","major_comments":[{"comment":"The nonzero lower bound that drives the 'congested network property' is derived under an unverified stochastic assumption: cell occupancies are modeled as Bernoulli(k) and the queue lengths on NS and EW approaches as independent binomials. CA rule 184 (Eq. 5) is conservative and deterministic, and at k>0.75 stationary configurations contain strongly correlated platoons and holes, so the independence assumption is least plausible in the regime where it is used. The derivation also assumes 'provided outgoing approaches do not block traffic,' which fails under congestion because downstream spillback limits discharge. Finally, the percentile-of-the-min-of-two-binomials curve is not a lower bound for all policies; it is a quantile of one heuristic policy's throughput. Without verification against the actual CA dynamics or a rigorous bound, the 'nothing to learn' mechanism is unsupported.","section":"§5 (Discussion), 'congested network property' derivation and Fig. 10"},{"comment":"The abstract claims that training under free-flow conditions yields DRL policies that are 'optimal for all traffic conditions', but §4.3.1 describes the best free-flow-trained policies as 'albeit only competitive', i.e. only overlapping LQF. Under the paper's own terminology in §4, 'optimal' means strictly outperforming LQF. These statements need to be reconciled, with the abstract's claim either weakened or supported by the relevant MFD comparisons.","section":"Abstract and §4.3.1 ('Constant demand')"},{"comment":"The central empirical claim that DRL cannot learn for training occupancies above about 75% is supported by single learning trajectories for π(s1) and π(s2); no confidence intervals or multiple-seed summaries are given for these curves, unlike the MFD evaluations. Since this claim motivates the practical recommendation to discard congested data, repeated-seed results and a quantitative definition of 'failure to learn' should be provided.","section":"§4.3, Figs. 6-8"},{"comment":"The recommendation to discard all congested training data for current DRL methods goes beyond the evidence. The experiments use one policy class (an MLP with REINFORCE-TD), one network topology (torus with one lane per direction and constant demand), and one benchmark. The statement that 'all the DRL methods proposed in the literature to date are unable to learn' is a conjecture, not a demonstrated result; the advice should be scoped to the tested setting.","section":"§5, 'In the meantime' paragraph"}],"minor_comments":[{"comment":"The Boolean update formula is printed as 'ci := ci−1∨ ci−1∧ ci∨ ci∧ ci+1', which appears to omit operators or parentheses; please correct and verify that it matches rule 184.","section":"§3, Eq. (5)"},{"comment":"The quantities λ, ρ, and δ are not defined in the manuscript; since the consistency check w≈2/3 and λ≈1 depends on equation (17b) of [17], please reproduce the needed definitions or equation.","section":"§5, Eq. (9)"},{"comment":"There is a stray phrase 'busy working so' after Eq. (9) and an ungrammatical phrase 'which is deterministic in which explains' in the paragraph about extreme congestion; the manuscript needs proofreading.","section":"§5"},{"comment":"The right panel of Fig. 10 appears to be unused and unlabeled in the text; either make it part of the argument or remove it.","section":"§5 / Fig. 10"},{"comment":"Several sentences state that results were verified for other cell lengths, green times, network sizes, and training algorithms, but no supporting data or appendix is provided; please document these checks or soften the statements.","section":"§4.3.1 and §5"}],"recommendation":"major_revision","confidential_remarks":"The empirical findings are potentially valuable, but the paper currently overclaims in both the abstract and the practical recommendations. The authors should either substantially strengthen the theoretical lower-bound argument or reframe the manuscript as an empirical report confined to the tested idealized setting. A useful revision would also add multi-seed learning curves and a quantitative failure criterion for the DRL experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a look for two results: the density threshold around 0.75 where policies become throughput-independent, and the striking finding that supervised learning with just two extreme examples outperforms longest-queue-first. The empirical core is honest and fairly careful for an idealized CA grid on a torus. The authors also make a reasonable extension of REINFORCE to continuing problems, though they note other methods behave similarly.\n\nWhat is genuinely new: nobody seems to have trained DRL policies under controlled density levels and shown that training occupancy is the dominant factor. The deterioration above 0.75 is directly observed, and the two-example supervised result is a useful practical insight, even if its mechanism is unexplained.\n\nSoft spots, in proportion. First, \"optimal\" means \"outperforms LQF.\" That is a weak benchmark, and the paper says so, but it is still worth flagging. Second, confidence intervals are shown for the MFDs but not for the learning curves, so some of the claims about convergence rest on visual inspection. Third, the explanatory \"congested network property\" is a conjecture. The Bernoulli/binomial derivation in the Discussion is explicitly heuristic, and the stress-test note is right: rule 184 creates strong correlations at high density, and the derivation assumes outgoing approaches do not block traffic, which is exactly what fails under spillback. So the lower bound shape is unverified, and the practical advice to discard all congested data from all DRL methods goes beyond the evidence. The authors are candid that this is a conjecture, but the abstract and outlook lean on it more than the derivation supports.\n\nThe empirical failure of DRL under congested training stands on its own, independent of the mechanism. I would not accept the general \"nothing to learn\" conclusion without a verified bound, and the recommendation to discard congested data should be limited to the conditions tested here. The paper also provides no code or data, which is a real limitation for reproducibility.\n\nWho is this for? Researchers in traffic signal control using RL, and people studying why policy-gradient methods fail in congested, continuing environments. It deserves a serious referee: the density threshold result could affect a large body of work, and the empirical setup is clear enough to reproduce or refute. I would send it to review, but ask the authors either to verify the lower bound mechanism or to label it more explicitly as a plausible explanation, and to release the simulator code or at least detailed experiment parameters.\n\nFor your reading group: maybe. It is not deep theory, but it is a clean counterexample to the common habit of training DRL on whatever traffic data is available.","headline":"A clean empirical demonstration that DRL signal control fails under congested training data, with a heuristic explanation that is more speculative than the data.","tokens_in":13134,"tokens_out":1599,"would_cite":false,"duration_ms":20970,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Traffic-signal DRL fails to learn above 75% network occupancy because signal choice stops affecting throughput.","keywords":["traffic signal control","deep reinforcement learning","macroscopic fundamental diagram","cellular automaton rule 184","congested network property","policy gradient reinforcement learning","supervised learning","traffic density"],"falsifier":"Use the same torus-grid simulator but replace the Bernoulli initialization with platoon or bursty arrivals: if a deliberately bad policy's throughput at 75-85% occupancy drops below the LQF benchmark, or if DRL trained under that congestion learns a policy that beats LQF, then the congested network property is refuted. A simpler observational check is to measure the MFD lower bound directly with probe vehicles on a real grid and see whether it rises toward the upper bound above 75% occupancy.","tokens_in":12151,"feed_emoji":"🚦","tokens_out":5423,"duration_ms":54943,"temperature":0.7,"pith_summary":"This paper asks why deep reinforcement learning (DRL) methods for traffic signal control have struggled in large networks, and answers with a traffic-flow property rather than an algorithmic weakness. Using a boundary-free grid on a torus and a cellular-automaton traffic model, it shows that a policy trained only under free-flow conditions is near-optimal at every density, while training under congestion degrades the policy and training above about 75% occupancy produces policies that fail everywhere. The cause is a 'congested network property': under heavy congestion, even a deliberately bad signal policy produces more throughput than the downstream links can absorb, so total network flow becomes independent of signal choice and the learning gradient carries no useful information. A supervised policy trained on just two extreme examples outperforms the standard longest-queue-first benchmark, and random search already finds competitive policies. The practical conclusion is that DRL for signal control should train on free-flow data only and discard congested observations.","feed_headline":"Traffic-signal DRL fails to learn above 75% occupancy","feed_subtitle":"Training on free-flow data alone gives near-optimal signal policies at all densities; congested data corrupt learning.","key_machinery":"The load-bearing object is the congested network property, formalized as a nonzero lower bound on the MFD in congestion. For a 'bad' policy that always serves the shortest queue, the number of vehicles on each pair of approaches is modeled as independent binomial random variables with parameters proportional to the approach length and the prevailing density; the intersection outflow is then a scaled version of the minimum of these two variables. The percentile curves of this minimum give the MFD lower bound, which rises with density and meets the upper bound (from the method of cuts) near 75% occupancy. This collision is the mechanism: it makes network throughput policy-independent in congestion, which strips the reward signal that a policy-gradient trainer needs.","core_discovery":"The paper's central claim is that the effectiveness of a learning-based signal controller is governed by the network's macroscopic fundamental diagram (MFD)—the plot of average flow versus average density—and specifically by the gap between its upper and lower bounds. In free flow the lower bound is zero and the upper bound leaves room for policy choice, so gradient information is meaningful. As density rises, the lower bound rises too: because queues are nearly always present, whichever approach receives green can discharge at high rate, and the intersection throughput exceeds downstream capacity. The MFD's upper and lower bounds collide around 75% density, and beyond that point signal policy has no measurable effect on network throughput. The paper establishes this by deriving the lower bound as the scaled percentile of the minimum of two binomial variables, one for each approach pair, under a Bernoulli occupancy assumption, and by using the method of cuts for the upper bound. It then shows empirically that DRL training under such conditions cannot recover sensible policies, even when initialized from a good supervised policy, while supervised learning from two extreme states succeeds.","pith_inferences":["If the congested network property transfers to real street networks, average network occupancy becomes a simple predictor for when adaptive signal control stops mattering; controllers could monitor the MFD bound gap and switch to fixed or offset-based timing beyond a threshold.","The paper's binomial-min lower bound assumes spatially independent arrivals; a testable extension is to run the same experiments with correlated platoon arrivals, which should either widen the learning window or invalidate the lower bound.","The supervised two-example result suggests a curriculum: train a policy from extreme states in free flow, freeze it, and only fine-tune on data from densities below the critical value, avoiding the degenerate gradient region.","The conjecture that optimal green time is proportional to approach length could be turned into a direct design rule and validated by comparing the MFD of proportional-timing controllers against learned policies on heterogeneous grids."],"forward_implications":["Current DRL signal-control methods should filter out congested training samples; doing so should improve their performance at all traffic densities.","A free-flow-only training regime yields controllers that remain near-optimal in congestion, so congestion data is not needed for good control.","Supervised learning with only two extreme-state examples beats the longest-queue-first greedy benchmark, suggesting that expert-designed extreme states can bootstrap control.","Random search, not just deep learning, can find competitive signal policies in these networks, so the policy landscape is smooth enough for simple search.","REINFORCE-TD provides a practical way to apply policy-gradient training directly to the continuing (infinite-horizon) signal-control problem without episodic resets."],"supporting_citations":[{"why":"Defines the longest-queue-first greedy algorithm used as the benchmark every policy in the paper must beat or match.","marker":"[1]"},{"why":"Provides the method-of-cuts construction used for the upper bound of the MFD feasibility region.","marker":"[16]"},{"why":"Supplies the stochastic MFD approximation and the lambda-rho parameters used to derive the congested-network slope and the green-time conjecture.","marker":"[17]"},{"why":"Earlier random-search and RL experiments across densities up to 0.75, whose results the paper cites as consistent with its 0.75 threshold.","marker":"[31]"},{"why":"Establishes the symmetry that makes the kinematic wave model parameter-free, justifying the isosceles fundamental diagram.","marker":"[33]"},{"why":"Introduces cellular automata and Rule 184, the traffic model used as the simulation environment.","marker":"[34]"},{"why":"The only earlier mention of DRL-style agents struggling under congested non-stationary traffic, cited to support the novelty of the congested network property.","marker":"[39]"},{"why":"REINFORCE with baseline, the basis of the proposed REINFORCE-TD training algorithm for continuing problems.","marker":"[11]"}],"fun_headline_variants":["DRL traffic control fails above 75% occupancy","Supervised learning beats DRL for signal control","Free-flow training yields optimal signal policies","DRL can't learn under congested traffic conditions","Signal policies matter only below 75% occupancy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The lower-bound derivation assumes that, at any time, each cell is occupied independently with probability equal to the average density, so the two approach queues are independent binomial counts; if real arrivals are correlated or queues interact across intersections, the nonzero lower bound—and with it the claim that congestion leaves nothing for DRL to learn—can fail.","fun_headline_variants_meta":{"raw":{"variants":["DRL traffic control fails above 75% occupancy","Supervised learning beats DRL for signal control","Free-flow training yields optimal signal policies","DRL can't learn under congested traffic conditions","Signal policies matter only below 75% occupancy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000454,"raw_usage":{"total_tokens":2343,"prompt_tokens":1069,"completion_tokens":1274,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":685,"completion_tokens_details":{"reasoning_tokens":1217}},"tokens_in":685,"tokens_out":1274,"duration_ms":11161,"temperature":1.0,"reasoning_tokens":1217,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:38:50.810297+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use the same torus-grid simulator but replace the Bernoulli initialization with platoon or bursty arrivals: if a deliberately bad policy's throughput at 75-85% occupancy drops below the LQF benchmark, or if DRL trained under that congestion learns a policy that beats LQF, then the congested network property is refuted. A simpler observational check is to measure the MFD lower bound directly with probe vehicles on a real grid and see whether it rises toward the upper bound above 75% occupancy.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the longest-queue-first greedy algorithm used as the benchmark every policy in the paper must beat or match."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the method-of-cuts construction used for the upper bound of the MFD feasibility region."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the stochastic MFD approximation and the lambda-rho parameters used to derive the congested-network slope and the green-time conjecture."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Earlier random-search and RL experiments across densities up to 0.75, whose results the paper cites as consistent with its 0.75 threshold."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the symmetry that makes the kinematic wave model parameter-free, justifying the isosceles fundamental diagram."},{"cited_title":"Nature, V ol","cited_arxiv_id":null,"evidence_quote":"Introduces cellular automata and Rule 184, the traffic model used as the simulation environment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The only earlier mention of DRL-style agents struggling under congested non-stationary traffic, cited to support the novelty of the congested network property."},{"cited_title":"Technical Report NU-CCS-88-3, Northeastern University, 1988","cited_arxiv_id":null,"evidence_quote":"REINFORCE with baseline, the basis of the proposed REINFORCE-TD training algorithm for continuing problems."}],"review_version":1}