{"id":"36fc08d3-f5c1-4a50-99e0-55fd2c158648","arxiv_id":"2507.18111","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A percentile-based reward for DRL-based RAN slicing meets delay-violation probability constraints while cutting average delay by 38% versus an average-delay baseline, and a reward-weighted model personalization method outperforms federated averaging.","lead":"This paper proposes a deep reinforcement learning controller that allocates radio resources to mobile virtual network operators while keeping packet delays below a probabilistic threshold, and reports a 38 percent reduction in average delay over an average-delay baseline. It also introduces a way to personalize shared DRL models by weighting each peer model by its measured performance, which outperforms standard federated averaging in the authors' simulations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reward-shaping coefficients in Eq. (11) are undisclosed and optimum-equivalence is demonstrated for only one scenario, so the Table 1 gains are not reproducible or generalizable.","rationale":"The reader's weakest assumption correctly identifies the undisclosed reward-shaping coefficients and the single-scenario validation of the optimum-preservation claim as the key soft spot. This is load-bearing because PDA-DRL is defined by its reward function: if Eq. (11) does not share the argmax of Eq. (22) in a given environment, the learned policy is not solving the stated constrained optimization problem, and the reported QoS and delay metrics become specific to the hand-tuned shaping rather than to the percentile-based formulation. The concrete test directly checks this by comparing argmax locations across the environments used in the paper. The absence of multi-seed confidence intervals compounds the issue but is secondary; even with error bars, an undisclosed and potentially environment-specific shaping would leave the central claim unsupported. The paper's experimental appendix also uses Deep Q-learning on a different action space rather than the policy-gradient PDA-DRL of the simulations, so the Colosseum validation does not directly validate the headline method, but this is a separate concern and does not alter the conditional verdict. Overall, conditional acceptance remains appropriate: the direction is plausible and the personalization idea is useful, but the quantitative claims need the missing parameter disclosure and optimality check before full acceptance.","tokens_in":17017,"tokens_out":4159,"duration_ms":47139,"concrete_test":"Obtain the exact values of ζ_p, ν_p, γ_p, ζ_n, ν_n, γ_n, and R_max used in Eq. (11). For each of the 10 personalization environments of Fig. 6 plus the Table 1 environment, compute the practical reward as a function of the number of PRBs under the actual traffic and channel distributions, and compare its argmax with the argmax of the Lagrangian reward (22) and with the constrained optimum found by exhaustive search. If any argmax differs, the claim that the modified reward preserves the global optimum is contradicted, and the quantitative gains in Table 1 should be re-evaluated as reward-shaping artifacts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The practical reward in Eq. (11) is the training objective for PDA-DRL, yet its coefficients ζ_p, ζ_n, ν_p, ν_n, γ_p, γ_n and the clipping bound R_max are never given. The only evidence that this shaped reward preserves the global optimum of the Lagrangian reward (22) is Fig. 3a, which shows a single scenario. Since the practical reward is a heavily shaped surrogate, any environment in which the shaping changes the location of the argmax will cause the agent to optimize a different objective than the constrained PRB-minimization problem (5)-(6). Consequently, the claimed 1% QoS margin, 38% average-delay reduction, and 33% delay-STD reduction in Table 1 are not guaranteed to hold outside the displayed scenario, and the results cannot be independently reproduced without the parameter values. The paper also reports no multi-seed statistics, so the scalar values in Table 1 cannot be separated from training variance; however, the reward-optimum issue is more fundamental because it concerns the objective being optimized, not just the precision of the reported numbers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses PRB allocation for RAN slicing in O-RAN under probabilistic delay constraints. It derives a Lagrangian-style reward from the constrained optimization problem (Eq. 10), then proposes a shaped percentile-based reward (Eq. 11) and trains a policy-gradient agent (PDA-DRL) that minimizes PRB usage while keeping Pr(d < Dmax) close to 1-epsilon. It also proposes a reward-based personalization scheme (Eq. 16) in which each MVNO aggregates other agents' model weights according to their measured performance in its own environment. Simulation results report that PDA-DRL meets the QoS constraint within a 1% margin while reducing average delay by 38% and delay STD by 33% over an average-delay DRL baseline, and that reward-based personalization outperforms federated averaging and similarity-based aggregation across 10 environments. Additional experiments on the Colosseum/SCOPE testbed are described in Appendix C.","tokens_in":17222,"tokens_out":7535,"duration_ms":77614,"significance":"If the claims hold, the paper is a useful contribution to O-RAN slicing: it targets percentile/probabilistic delay guarantees rather than average delay, identifies a concrete training pathology in Lagrangian reward shaping (the 'unlearnable region'), and introduces a performance-based personalization mechanism that is a reasonable alternative to FedAV when client environments are heterogeneous. The comparison against fixed, heuristic, and average-delay baselines is appropriate, and the hardware-in-the-loop validation is a strength. However, the central empirical claims are currently supported by a single simulation run and by a shaped reward whose coefficients are never disclosed, which limits reproducibility and generality.","major_comments":[{"comment":"The coefficients zeta_p, nu_p, gamma_p, zeta_n, nu_n, gamma_n and the clipping bound R_max are never given, and no tuning procedure is described. Because Eq. (11) is the training objective for PDA-DRL, the results in Table 1 cannot be independently reproduced, and the sensitivity of the reported 1% QoS margin, 38% average-delay reduction, and 33% delay-STD reduction to these coefficients is unknown. Please report the values used and, ideally, a parameter-sensitivity study.","section":"Section 6, Eq. (11)"},{"comment":"The claim that the practical reward in Eq. (11) preserves the global optimum of the Lagrangian reward in Eq. (22) is supported only by a single scenario (Dmax = 5 ms, epsilon = 0.1). The shaping in Eq. (11) is nonlinear and can in principle shift the argmax in other environments. The paper should either prove conditions under which the optimum is preserved or verify equality of the optimizers across the full set of simulated environments, including the 10 environments used in Fig. 6.","section":"Section 6, Fig. 3(a)"},{"comment":"The action-space definition is internally inconsistent. The set {-2^J, ..., -1, 0, 1, ..., 2^J}, read as consecutive integers, contains 2^{J+1}+1 elements, while the power-of-two reading contains 2J+3 elements; the paper states |A| = 2J+1. Since Section 8 sets J = 5, the number of actions and therefore the output-layer dimension of the policy network are ambiguous. Please correct the formula and report the exact action set used in both the simulation and the testbed experiments.","section":"Section 5.2, Eq. (9)"},{"comment":"All headline results are point estimates from a single training run. No seeds, confidence intervals, or significance tests are reported, so the claimed 38% and 33% reductions and the 1% QoS margin cannot be separated from training variance. The paper should report multi-seed statistics (e.g., mean and standard deviation over at least 5 seeds) for all policies in Table 1 and Fig. 6.","section":"Section 8, Table 1"},{"comment":"The Colosseum validation uses Deep Q-learning with a different action set ({-9, -6, -3, 0, 3, 6, 9}), a different control period (250 ms), and a different exploration schedule, whereas the main simulation method is a policy-gradient agent using the action space of Eq. (9). The appendix is therefore a related feasibility study rather than a validation of the specific PDA-DRL algorithm. Please clarify the relationship and, if the appendix is intended as a validation of PDA-DRL, reconcile the algorithm and hyperparameter choices.","section":"Appendix C vs Section 8"}],"minor_comments":[{"comment":"The caption and legend use generic labels 'DRL', 'Fixed', and 'Heuristic', while the text refers to PDA-DRL, MD-DRL, Fixed-Av, and Fixed-Max. Please make the labels consistent.","section":"Figure 4"},{"comment":"The text says that when Delta < 0 the first term contributes linearly to the reward, but the expression is Delta^{gamma_n}; please specify whether gamma_n = 1 (and analogously gamma_p = 1) or explain the discrepancy.","section":"Eq. (11) and surrounding text"},{"comment":"The evaluation procedure for obtaining R^T_i,j is under-specified: it is not stated whether the tested policies are deterministic, how many evaluation trajectories are averaged, or how the positive bias added to the (negative) rewards in Fig. 6 is chosen. This makes the personalization comparison difficult to reproduce.","section":"Section 7.3, Eq. (16)"},{"comment":"The action-space example A = {10, 20, ..., 120} is inconsistent with the later statement that the RAN has 150 total PRBs. Please clarify whether the actions are in units of PRBs or resource block groups and whether the action range depends on the total PRB budget.","section":"Section 5.2 and Section 8"},{"comment":"The duration Tb is defined as a PRB time slot in Section 3 and set to 1 ms in Section 8, but Appendix C states that a PRB has a duration of 0.5 ms. Please use consistent time-scale definitions across the simulation and the testbed sections.","section":"Section 3 and Section 8"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a serious paper with a genuine idea, and the right response is to send it to referees and ask for a revision, not to reject it. The core contribution — a reward function that converts a probabilistic delay constraint into an RL objective and then shapes it so training does not collapse in the 'unlearnable region' — is sensible and reasonably derived. The LLN/Lagrangian construction in Appendix A checks out as a standard dual derivation, and the exponential shaping terms are a pragmatic fix for a real failure mode. The personalization rule in Eq. (16), where each agent weights other agents' models by their measured reward on its own environment, is a clean and useful idea and is properly compared against FedAV, feature-distance, and weight-distance baselines.\n\nThe soft spots are real but, for the most part, fixable. The practical reward (11) has seven parameters (ζ, ν, γ, λ, Rmax) and the paper never states their values or a tuning procedure. That alone makes Table 1 unreproducible, and the claim that the shaped reward preserves the optimum of the Lagrangian reward is only illustrated in one scenario (Fig. 3a). I don't think the claim is wrong — the shaping is designed to be monotone around the crossing — but the evidence is thin. Also, the numbers in Table 1 come from a single run; no seeds, no error bars. For a DRL paper that is not unusual, but it is still a weakness. There is also a small notation bug in Eq. (9): the set from -2^J to 2^J has 2^{J+1}+1 elements, not 2J+1. The experimental appendix (Colosseum) shows the agent learns, but it uses DQN and a different action set, so it does not validate the main comparison.\n\nThe paper does good things: it states the problem cleanly, derives the reward rather than pulling it from thin air, and the personalization comparison is reasonable. My recommendation: accept the premise and send to peer review, but require a revised version that discloses all reward coefficients, adds multi-seed runs with variance, and either proves or empirically checks the optimum-preservation claim across the environments used. With those additions, the quantitative claims would be credible. The core direction is solid; this is not a desk-reject.","headline":"A serious DRL-for-RAN-slicing paper with a genuine reward-shaping idea and a clean personalization scheme, but the headline numbers rest on undisclosed coefficients and single-run statistics; worth refereeing, not desk-rejecting.","tokens_in":17756,"tokens_out":3184,"would_cite":true,"duration_ms":33609,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A percentile-based DRL policy meets RAN delay bounds within 1% while cutting average delay by 38%; a reward-weighted model-sharing method beats federated averaging across ten environments.","keywords":["RAN slicing","O-RAN","deep reinforcement learning","percentile-based delay guarantee","reward shaping","personalized federated learning","PRB allocation","probabilistic QoS"],"falsifier":"Run PDA-DRL in an environment whose satisfaction-probability curve rises slowly or plateaus before the target $(1-\\epsilon)$ is reached, and compare the PRB count at the shaped reward's maximum with the Lagrangian optimum. If the piecewise reward's argmax shifts (or the clipping $R_{max}$ flattens the landscape), the 1% QoS margin or the 38% delay reduction would not reproduce; the paper reports only one such comparison in Fig. 3a.","tokens_in":16762,"feed_emoji":"📡","tokens_out":9353,"duration_ms":79489,"temperature":0.7,"pith_summary":"This paper takes aim at a specific failure mode in deep reinforcement learning (DRL) for radio access network (RAN) slicing: optimizing for average delay does not meet percentile-style quality-of-service (QoS) guarantees. The authors derive a reward function from the law of large numbers that rewards the probability of meeting a per-packet delay deadline, then reshape it into a practical, clipped form that avoids an 'unlearnable region' where the theoretical reward gives no usable gradient. Their agent, PDA-DRL, is claimed to satisfy the probabilistic delay constraint within a 1 percent margin while using roughly the same number of physical resource blocks (PRBs) as a mean-delay-optimized baseline, yet delivering 38 percent lower average delay and 33 percent lower delay standard deviation. The paper also proposes a reward-based personalization method that lets multiple MVNOs share DRL model weights by weighting each other's models by measured performance, which it shows beats federated averaging and similarity-based aggregation across ten environments.","feed_headline":"38% delay cut with a percentile-based DRL reward","feed_subtitle":"A shaped reward meets RAN delay targets within 1 percent while using the same PRB count","key_machinery":"The load-bearing object is the practical reward function of Eq. (11): $r = -\\Delta \\gamma_p + \\exp(\\zeta_p \\Delta + \\nu_p N_{Ts}^2)$ when $\\Delta \\geq 0$ and $r = \\Delta \\gamma_n + \\exp(\\zeta_n \\Delta + \\nu_n N_{Ts})$ when $\\Delta < 0$, with $\\Delta = Pr(d_q < D_{max}) - (1-\\epsilon)$ and the output clipped to $(-R_{max}, 0)$. This shape is designed so that when the satisfaction probability is below target the exponential term grows, forcing the agent to request more PRBs quickly, while near the feasible boundary the linear term in $\\Delta$ dominates and the PRB penalty $N_{Ts}$ prevents over-allocation. The second object is the personalization coefficient of Eq. (16): $\\alpha_{i,j} = \\exp(\\beta \\hat{R}^T_{i,j}) / \\sum_j \\exp(\\beta \\hat{R}^T_{i,j})$, where $\\hat{R}^T_{i,j}$ is the average reward that agent $i$ obtains by running agent $j$'s model on its own environment for $T$ episodes; $\\beta$ interpolates between plain averaging ($\\beta \\approx 0$) and hard selection of the best model ($\\beta$ large). Both objects carry the paper's arguments: the reward function converts a hard probabilistic constraint into a smooth learning signal, and the personalization rule converts measured performance into aggregation weights.","core_discovery":"The central claim is that a percentile-based, reward-shaped DRL controller can satisfy a probabilistic upper bound on packet transmission delay in O-RAN slicing while spending nearly the same radio resources as a baseline that only minimizes average delay. The reward is built from the empirical satisfaction probability $Pr(d_q < D_{max})$ minus the target $(1-\\epsilon)$, multiplied by a trade-off parameter and balanced against PRB usage, which the authors show is equivalent to the Lagrangian dual of the constrained minimization problem. Because the theoretical reward has a flat 'unlearnable region' where no packets meet the deadline, the authors replace it with a piecewise exponential/linear shape whose maximum coincides with the theoretical optimum (shown for one scenario) and clip it to $(-R_{max}, 0)$. In simulation, the resulting PDA-DRL policy meets the QoS constraint within a 1% margin, cuts average delay by 38% and delay standard deviation by 33% versus a mean-delay DRL baseline, and uses about the same PRB count. For model sharing, the paper's reward-based personalization, where each agent weighs other agents' weights by $\\exp(\\beta \\times \\text{average reward})$ on its own environment, outperforms federated averaging and both feature- and weight-similarity aggregation across ten distinct environments.","pith_inferences":["The reward-shaping trick could be applied to any DRL problem with a hard threshold constraint and a failure region where the naive reward is flat, such as power or budget limits in other scheduling domains.","A practical tuning recipe for the coefficients $\\zeta, \\nu, \\gamma, \\lambda, R_{max}$ (e.g., a grid search or a schedule that anneals them) would be needed before the method can be deployed without environment-specific hand-tuning.","The personalization rule's reliance on testing other agents' models on one's own environment assumes a digital twin or emulator is available; in a purely live network that evaluation cost may be prohibitive."],"forward_implications":["Satisfying a probabilistic delay bound with DRL is achievable with a reward that directly tracks the empirical satisfaction probability, not just the mean delay.","The shaped reward gives a principled way to escape a failure mode (zero gradient when no packet meets the deadline) that plagues naive Lagrangian rewards in constraint-heavy wireless tasks.","Reward-based personalization can replace federated averaging when agents operate in very different environments, because each agent evaluates candidate models on its own soil.","Since PRB usage stays nearly constant while delay variance shrinks, the approach promises more predictable latency for time-critical slices without extra radio resources."],"supporting_citations":[{"why":"RL reward design approach that the LLN-based reward derivation in Appendix A follows.","marker":"Raeis et al., 2021"},{"why":"Federated averaging baseline that the reward-based personalization must beat.","marker":"McMahan et al., 2017"},{"why":"Feature-similarity personalization baseline compared in Section 7.1.","marker":"Rezazadeh et al., 2022"},{"why":"Model-weight-similarity personalization baseline compared in Section 7.2.","marker":"Nagib et al., 2023"},{"why":"RF emulation testbed used for the experimental validation in Appendix C.","marker":"Bonati et al., 2021c"},{"why":"Physical-layer slicing framework used to create PRB slices on the testbed.","marker":"Bonati et al., 2021a"}],"fun_headline_variants":["38% delay cut via percentile-based DRL reward shaping","DRL reward tuning meets delay targets at same PRB use","Percentile reward DRL: 38% lower delay, tight QoS margin","Reward-based personalization tops federated averaging for RAN slicing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the practical, clipped reward function in Eq. (11) reaches its maximum at the same number of PRBs as the idealized Lagrangian reward for every environment, so that reshaping and clipping do not move the optimum; the paper only illustrates this for a single scenario and gives no coefficient values or tuning procedure for $\\zeta, \\nu, \\gamma, \\lambda, R_{max}$.","fun_headline_variants_meta":{"raw":{"variants":["38% delay cut via percentile-based DRL reward shaping","DRL reward tuning meets delay targets at same PRB use","Percentile reward DRL: 38% lower delay, tight QoS margin","Reward-based personalization tops federated averaging for RAN slicing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000506,"raw_usage":{"total_tokens":2504,"prompt_tokens":1018,"completion_tokens":1486,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":1412}},"tokens_in":634,"tokens_out":1486,"duration_ms":13092,"temperature":1.0,"reasoning_tokens":1412,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:38:27.308587+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PDA-DRL in an environment whose satisfaction-probability curve rises slowly or plateaus before the target $(1-\\epsilon)$ is reached, and compare the PRB count at the shaped reward's maximum with the Lagrangian optimum. If the piecewise reward's argmax shifts (or the clipping $R_{max}$ flattens the landscape), the 1% QoS margin or the 38% delay reduction would not reproduce; the paper reports only one such comparison in Fig. 3a.","supporting_citations":[{"cited_title":"Queue-Learning: A Reinforcement Learning Approach for Providing Quality of Service","cited_arxiv_id":"2101.04627","evidence_quote":"RL reward design approach that the LLN-based reward derivation in Appendix A follows."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Federated averaging baseline that the reward-based personalization must beat."},{"cited_title":"On the specialization of fdrl agents for scalable and distributed 6g ran slicing orchestration","cited_arxiv_id":null,"evidence_quote":"Feature-similarity personalization baseline compared in Section 7.1."},{"cited_title":"M., Abou-zeid, H., and Hassanein, H","cited_arxiv_id":null,"evidence_quote":"Model-weight-similarity personalization baseline compared in Section 7.2."}],"review_version":1}