REVIEW 5 major objections 6 minor 15 references
Multi-Agent Reinforcement Learning for Online Traffic Scheduling in Time-Sensitive Application
T0 review · 5 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read This paper claims that a multi-agent reinforcement learning scheduler, with one autonomous agent per TSN queue and trained by HAPPO, cuts average frame waiting times by up to 26.8% and worst-case delays by about 16.8% in dynamic XR-driven…
desk verdict A plausible new application of MARL to TSN scheduling, but the reported 26.8%/16.8% gains are not traceable in the results as written. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the per-queue agent decomposition trained by Heterogeneous-Agent Proximal Policy Optimization (HAPPO). Each queue agent observes only its own backlog, average waiting time, and oldest-frame age, then outputs a scalar score in $[0,1]$; the scheduled queue is the argmax of the agents' scores. A global critic sees the joint observation and all scores to produce a coordinated training signal, and agents are updated sequentially in a random permutation to damp non-stationarity. The team reward is the minimum over agents of per-queue rewards based on $\lambda_i$, the ratio $g_{i,k}/\text{deadline}_i$, with an extra constant penalty $\alpha=1.5$ when $\lambda_i > 1$; this forces the worst-off queue to dominate learning and is what prevents the lower-load semantic queue from being starved.
What would settle it
Re-run the same three-queue experiment with recorded real XR downlink traces and an actual TSN NIC at 1 Gbps; if HAPPO's $p_{99}$ waiting times no longer stay at or below the 11.11, 13.88, and 16.75 ms deadlines while PPO and A2C miss them by the reported margins, then the claimed 26.8% and 16.8% improvements are artifacts of the simulated arrival process.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that a cooperative multi-agent formulation of the TSN time-aware shaper outperforms both centralized learning and rule-based heuristics in a dynamic MEC scenario. Each queue is an agent with local features $(b_{i,k}, w_{i,k}, g_{i,k})$ -- backlog, average waiting time, and age of the oldest frame -- and at every slot each agent emits a preference score; the queue with the highest score transmits. Agents are trained with HAPPO, which updates policies sequentially in a random order under a global critic, using a team reward $r_t = \min_i R_i$ where $R_i = -\alpha \lambda_i$ if $\lambda_i > 1$ and $R_i = -\lambda_i$ otherwise, with $\lambda_i$ the ratio of the oldest frame's age to the application deadline. In simulation, this yields $p_{99}$ average waiting times below all three deadlines, whereas single-agent PPO and A2C exceed the deadlines in the two AR queues about 48.51% and 41.06% of the time, and backlog-aware and AOF-aware heuristics meet deadlines by sacrificing the lower-load semantic queue. The measured headline results are a reduction of up to 26.8% in average frame waiting time and roughly 16.8% in worst-case delays relative to the baselines.
Load-bearing premise
The load-bearing premise is that the synthetic XR traffic generated from the Johnson SU models, with randomly chosen active-user counts and a fixed 500-microsecond slot, represents real co-located XR/MEC dynamics closely enough that the measured delay reductions would transfer to practice.
Editorial extensions
If this is right
- Deploying the scheduler requires no changes to TSN switch hardware, because execution uses queue-local observations and only the training phase needs a central critic.
- Adding a new XR flow would mean adding another queue agent rather than recomputing a static schedule, since coordination among agents is learned rather than hand-coded.
- The scheduler keeps $p_{99}$ waiting times below each application deadline in all tested queues, which is the operating condition that matters for perceived XR quality.
- In comparable dynamic MEC settings with mixed high-rate video and low-rate semantic streams, the expected gains are up to 26.8% lower average waiting times and roughly 16.8% lower worst-case delays.
Reading between the lines
- Because the team reward is the minimum of per-queue rewards, the queue with the tightest deadline relative to its load will dominate the learned policy; a useful test the paper does not run is to vary $\alpha$ and the deadline margins and check whether the reported gains shift toward whichever queue is most stressed.
- The fixed 500-microsecond slot was chosen to shrink the action space; letting agents also propose slot durations, which the paper lists as future work, is the most direct extension and could reduce the residual deadline misses visible in the reaction-delay curves.
- The three-queue, all-XR setting is the simplest stress case; scaling to more queues or mixing control traffic with video would show whether HAPPO's higher seed sensitivity, reported as 28.41% above A2C and 76.69% above PPO, becomes a practical obstacle.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a multi-agent reinforcement learning scheduler for TSN queues at a MEC server serving co-located XR applications. Each queue is an agent in a HAPPO-trained cooperative policy, and the reward is derived from the age-of-oldest-frame (AOF) relative to application deadlines. The evaluation compares HAPPO with single-agent PPO/A2C and with backlog- and AOF-aware heuristics in a simulated three-queue XR scenario. The abstract and introduction claim a 26.8% reduction in average frame waiting time and a 16.8% reduction in worst-case delay.
Significance. The topic is timely and the architecture choice is reasonable: per-queue agents with centralized training and sequential updates directly address the non-stationarity typical of shared-medium scheduling. The comparison set (two RL baselines and two heuristics) is appropriate. If the reported gains were backed by a traceable summary table, the paper would be a useful empirical contribution. At present, however, the headline numbers are not derivable from the presented figures or tables, the training-reward values are internally contradictory, and the evaluation metric overlaps with the optimized reward. These issues are fixable, but they currently prevent verification of the central claim.
major comments (5)
- [IV-B, Table I] Section IV-B contains contradictory reward statements. The text says 'PPO achieved the highest average reward post-convergence (-20.77), whereas A2C exhibited the lowest terminal performance (-20.45)', but Table I lists A2C as -23.33 and PPO as -20.77; -20.45 is neither the tabulated A2C value nor the lowest reward. The later sentence 'A2C achieved the highest final reward' adds a third incompatible statement. Please correct the numbers and make the prose consistent with the table.
- [IV-B, abstract/introduction] The claimed reductions of 26.8% (average waiting time) and 16.8% (worst-case delay) are never computed or located in the results section. Fig. 3 reports p99 waiting times, Fig. 4 reports reaction-delay ECDFs, and the network-performance paragraph contains the broken references 'Fig.?? and Fig.??'. No table or passage reports the mean or worst-case values from which those percentages follow. Please add a summary table with per-queue and aggregate average, p99, and worst-case values for every method, and derive the headline percentages explicitly from that table.
- [IV-B] The text makes quantitative claims without a supporting figure or table: single-agent methods exceed deadlines in q0 and q1 by 48.51% and 41.06%, and HAPPO keeps q1's oldest frame 25.50% above the deadline versus 160.01% and 128.96% for PPO and A2C. These are tail-statistic claims that should be reported with run-to-run variability, not only in prose.
- [III-A, Eq. (7) and Eq. (6a)] The reward function and the optimization objective are both defined directly on AOF/deadline ratios, and the main evaluation metrics (average waiting time, reaction delay) are AOF-based. As a result, the reported improvements are partially by construction relative to heuristics that do not optimize this reward. Please report an independent, reward-blind metric such as deadline-miss ratio, frame loss, or per-flow jitter, or explicitly discuss the alignment of the objective and the evaluation metric and its consequences for interpreting the percentages.
- [IV-A, IV-B] The evaluation uses only five runs, and Figs. 3 and 4 contain no confidence intervals or per-run markers. Since the claims concern p99 and worst-case tail statistics, the variance matters: five runs can produce noisy tail estimates. Please provide error bars, confidence intervals, or per-run scatter so the reader can assess whether the observed gaps exceed run-to-run variability.
minor comments (6)
- [II-A] The set of queues is written as Q = {q0, q1, ..., qq}; the final index should presumably be qn, and the equation needs formatting cleanup.
- [IV-B, Fig. 4 caption] The caption states that vertical lines indicate p99 values and deadlines, but the lines are not labeled in the legend; please distinguish the two types of lines.
- [IV-A] The text says 'all results are reported as average and standard deviation across five different runs,' but neither Fig. 3 nor Fig. 4 displays standard deviations; please state where the variability is shown or remove the claim.
- [III-B, Algorithm 1] The loop at line 9 iterates i = 1,...,M, but line 10 updates theta_{sigma(m)}; the index m is undefined there. Please use a consistent index.
- [II-B] The traffic model uses Johnson SU distributions from [13] without reporting the fitted parameters; a brief summary of the parameters or a pointer to the dataset would improve reproducibility.
- [References] Reference [12] is an arXiv preprint; please cite the published version if one is available.
Circularity Check
No significant circularity: the reported AOF/waiting-time improvements align with the stated reward objective, but the comparisons against same-reward RL baselines and non-learning heuristics provide independent content.
full rationale
The paper's derivation chain is not circular. The traffic model is taken from an external dataset [13]; the latency model defines AOF and waiting time independently in Eqs. (3)-(5); the optimization objective in Eq. (6a) and reward in Eq. (7) are explicitly designed to minimize AOF relative to deadlines. The central results then compare HAPPO against PPO, A2C, backlog-aware, and AOF-aware baselines on the same waiting-time and AOF-derived metrics. Because all learning baselines optimize the same reward and the heuristics are not fitted to the reported numbers, the improvement is not forced by construction. The single self-citation [2] is used only as background for the TSN-MEC architecture and is not load-bearing for the central claim. The abstract's specific 26.8% and 16.8% numbers are not derivable from the figures or tables shown, and Section IV-B contains internal contradictions and broken references, but those are reproducibility and reporting problems, not circularity. No step reduces, by the paper's own equations or by self-citation, to its own inputs.
Assumptions & free parameters
free parameters (4)
- alpha (eq. 7 additional deadline-violation penalty) =
1.5
- time-slot duration D =
500 microseconds
- RL hyperparameters (learning rate, discount, hidden size, layers, episode length) =
1e-4, 0.99, 64, 2, 150
- number of queues/agents and per-queue user counts =
3 queues; q0 and q1 up to 4 users, q2 with 15 users
assumptions (5)
- domain assumption TSN egress model with one queue transmitting per fixed time slot (eq. 6b) is the correct abstraction for IEEE 802.1Qbv.
- domain assumption The Johnson SU traffic model from [13], with random active users, represents realistic co-located XR flows.
- domain assumption AOF relative to deadline is a valid proxy for XR quality of experience.
- standard math HAPPO's theoretical guarantees from [12] hold in this three-agent POMDP with scalar score actions.
- domain assumption A fixed 500 microsecond slot can transmit multiple frames over a 1 Gbps link, so per-slot transmission time can be ignored.
Cite this review
Pith. "Pith review of Multi-Agent Reinforcement Learning for Online Traffic Scheduling in Time-Sensitive Application." pith.science (2026). https://pith.science/paper/KJXDC67T
@misc{pith2026260805346,
author = {Pith},
title = {Pith review of: Multi-Agent Reinforcement Learning for Online Traffic Scheduling in Time-Sensitive Application},
year = {2026},
howpublished = {\url{https://pith.science/paper/KJXDC67T}},
note = {Machine review of arXiv:2608.05346}
}
read the original abstract
Time-sensitive networking (TSN) is increasingly integrated into mobile edge computing (MEC) to support applications with stringent latency requirements, such as extended reality (XR). However, existing TSN scheduling solutions predominantly rely on static optimization techniques or centralized learning models that are based on fixed traffic patterns, limiting their effectiveness in dynamic environments. In practice, MEC environments often host multiple co-located XR traffic flows whose characteristics evolve over time, creating complex inter-queue dependencies that current schedulers fail to capture. Addressing these challenges requires adaptive, decentralized scheduling mechanisms capable of coordinating multiple TSN queues under varying traffic conditions. To this end, this paper proposes a multi-agent reinforcement learning (MARL) framework for TSN scheduling, where each TSN queue is modeled as an autonomous agent. The Heterogeneous-Agent Proximal Policy Optimization (HAPPO) algorithm is employed to explicitly model inter-agent dependencies and jointly optimize service delivery across queues. The simulation results demonstrate that the proposed approach reduces average frame waiting times by up to 26.8% and worst-case delays by approximately 16.8%, highlighting its effectiveness in dynamic XR-driven MEC scenarios.
Figures
Reference graph
Works this paper leans on
-
[1]
“IEEE Standard for local and metropolitan area networks—bridges and bridged networks—amendment 25: Enhancements for scheduled traffic,IEEE Standard 802.1qbv-2015, 2016, pp. 1–57,” 2016
work page 2015
-
[2]
Performance analysis of the integra- tion of dynamic cloud computing environments and tsn networks,
M. Carvalho and D. F. Macedo, “Performance analysis of the integra- tion of dynamic cloud computing environments and tsn networks,” in 2025 IEEE 11th International Conference on Network Softwarization (NetSoft). IEEE, 2025, pp. 406–414
work page 2025
-
[3]
A survey of schedul- ing algorithms for the time-aware shaper in time-sensitive networking (tsn),
T. St ¨uber, L. Osswald, S. Lindner, and M. Menth, “A survey of schedul- ing algorithms for the time-aware shaper in time-sensitive networking (tsn),”Ieee Access, vol. 11, pp. 61 192–61 233, 2023
work page 2023
-
[4]
Time- sensitive networking (tsn) for industrial automation: Current advances and future directions,
T. Zhang, G. Wang, C. Xue, J. Wang, M. Nixon, and S. Han, “Time- sensitive networking (tsn) for industrial automation: Current advances and future directions,”ACM Computing Surveys, vol. 57, no. 2, pp. 1–38, 2024
2024
-
[5]
Reinforce- ment learning based routing for time-aware shaper scheduling in time- sensitive networks,
J. Min, Y . Kim, M. Kim, J. Paek, and R. Govindan, “Reinforce- ment learning based routing for time-aware shaper scheduling in time- sensitive networks,”Computer Networks, vol. 235, p. 109983, 2023
work page 2023
-
[6]
Deepscheduler: En- abling flow-aware scheduling in time-sensitive networking,
X. He, X. Zhuge, F. Dang, W. Xu, and Z. Yang, “Deepscheduler: En- abling flow-aware scheduling in time-sensitive networking,” inIEEE IN- FOCOM 2023-IEEE Conference on Computer Communications. IEEE, 2023, pp. 1–10
work page 2023
-
[7]
Ai-based dynamic schedule calculation in time sensitive networks using gcn-td3,
S. T. Islam and A. B. Muslim, “Ai-based dynamic schedule calculation in time sensitive networks using gcn-td3,” in2024 IFIP Networking Conference (IFIP Networking). IEEE, 2024, pp. 714–719
work page 2024
-
[8]
Deterministic scheduling for asymmetric flows in future wireless networks,
H. Dou, T. Zhu, F. Li, C. Liu, and L. Wang, “Deterministic scheduling for asymmetric flows in future wireless networks,”Symmetry, vol. 17, no. 8, p. 1246, 2025
work page 2025
Show all 15 references
-
[9]
Configuring the ieee 802.1 q time-aware shaper with deep reinforcement learning,
A. Roberty, Q. Besnard, S. B. H. Said, F. Ridouard, H. Bauer, and A. Geniet, “Configuring the ieee 802.1 q time-aware shaper with deep reinforcement learning,” inNOMS 2024-2024 IEEE Network Operations and Management Symposium. IEEE, 2024, pp. 1–7
2024
-
[10]
Mitigation of scheduling violations in time- sensitive networking using deep deterministic policy gradient,
B. Zhou and L. Cheng, “Mitigation of scheduling violations in time- sensitive networking using deep deterministic policy gradient,” inPro- ceedings of the 4th FlexNets Workshop on Flexible Networks Artificial Intelligence Supported Network Flexibility and Agility, 2021, pp. 32–37
2021
-
[11]
Convergence of reinforcement learning and time-sensitive networking for future industrial ai agent communication: Fundamentals, challenges, and opportunities,
N. Zhou, Y . Yao, X. Liang, S. Yao, J. Wan, C. Yang, W. Gui, and W. Gao, “Convergence of reinforcement learning and time-sensitive networking for future industrial ai agent communication: Fundamentals, challenges, and opportunities,”IEEE Communications Surveys & Tutorials, 2026
2026
-
[12]
Trust region policy optimisation in multi-agent reinforcement learning,
J. G. Kuba, R. Chen, M. Wen, Y . Wen, F. Sun, J. Wang, and Y . Yang, “Trust region policy optimisation in multi-agent reinforcement learning,” arXiv preprint arXiv:2109.11251, 2021
2021 arXiv
-
[13]
An extended reality offloading ip traffic dataset and models,
D. G. Mor ´ın, D. Medda, A. Iossifides, P. Chatzimisios, A. G. Armada, A. Villegas, and P. Per ´ez, “An extended reality offloading ip traffic dataset and models,”IEEE Transactions on Mobile Computing, vol. 23, no. 6, pp. 6820–6834, 2023
2023
-
[14]
From pixels to packets: Traffic classification of augmented reality and cloud gaming,
A. Shirmarz, F. L. Verdi, S. K. Singh, and C. E. Rothenberg, “From pixels to packets: Traffic classification of augmented reality and cloud gaming,” in2024 IEEE 10th International Conference on Network Softwarization (NetSoft). IEEE, 2024, pp. 195–203
2024
-
[15]
Asynchronous methods for deep rein- forcement learning,
V . Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep rein- forcement learning,” inInternational conference on machine learning. PmLR, 2016, pp. 1928–1937
2016
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.