{"id":"1705d9f9-c9f9-4ad4-9856-6bde3dbc075c","arxiv_id":"2501.08045","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A continual reinforcement learning scheduler with multi-timescale replay and a resource-constrained actor-critic reduces digital twin state estimation error by up to 55.2% in simulation.","lead":"A base station uses a continual reinforcement learning algorithm to decide which factory sensors transmit data and how many radio resource blocks each gets, so the digital twin stays synchronized with the physical factory. The paper reports up to 55.2% lower estimation error than fixed scheduling rules at the same radio resource cost.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"State omits the current RB budget M, so the policy, cost critic, and multiplier cannot condition on the quantity that changes in Fig. 4; the claimed capacity adaptation is not supported by the printed formulation.","rationale":"The reader identified the stale-mismatch proxy as the weakest assumption. That is a genuine POMDP concern, but the simulation itself uses the same proxy for all compared methods, and for slowly varying physical data a last-observation-plus-age feature can carry substantial information. The omitted M is a different and sharper issue: it is an internal inconsistency in the printed algorithm. The networks are not given the budget, yet the cost they regress and the constraint they enforce depend on that budget. If the actual implementation does append M to S_t, then Section III-B has a notational omission that is easy to fix; if it does not, the central claim of quick adaptation to network-capacity changes has no mechanism in the described architecture. This concern does not by itself disprove the fixed-M NRMSE improvements, but it directly targets the paper's central novelty and should be resolved by the authors. The reader's conditional verdict remains appropriate, so no change in disposition is proposed.","tokens_in":21200,"tokens_out":10973,"duration_ms":125385,"concrete_test":"Run the Fig. 4 experiment twice with identical hyperparameters: (i) the state exactly as printed in Section III-B without an M feature, and (ii) the same state augmented with the current M as a scalar input to the actor, cost critic, and multiplier networks. Report reward curves and final NRMSE for the 30->10->26 RB schedule over at least 10 random seeds. If variant (ii) does not converge faster and reach lower or lower-variance NRMSE than variant (i), the reported capacity adaptation does not require M in the state; if variant (ii) clearly outperforms variant (i), the printed state omits a variable essential to the continual-adaptation claim, and the paper must be revised to include M in the state and replay tuple.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In the component list of Section III-B, the state is S_t = [s_{1,t}, ..., s_{N,t}] with s_{n,t} = [Phi_{n,t}, Y_{n,t}, gamma_t]; no entry encodes the current RB budget M. However, Eq. (15) defines the cost as M when b^T a_t <= M and as b^T a_t otherwise, so the same (S_t, a_t) has different cost labels depending on M. The cost critic Q_{phi_C}(S_t, a_t), the multiplier lambda_zeta(S_t), and the actor policy all take only S_t as input. During the Fig. 4 schedule (M: 30 -> 10 -> 26), the networks therefore receive identical observations with conflicting cost targets and conflicting constraint thresholds; a single function without an M input cannot represent both. Consequently, the convergence advantage attributed to MTR in Fig. 4 cannot be the result of a policy that 'knows' the new capacity; it is at best the result of additional gradient updates after the change, and the claimed continual-learning transfer across capacity changes is unsupported. The same omission undermines the state-wise RB-constraint guarantees in Figs. 10-11, since the constraint threshold M is not part of the state used to learn or evaluate the policy.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies adaptive device scheduling and resource-block allocation for digital twin (DT) synchronization over wireless networks. A base station selectively schedules devices to transmit sensing data, and the problem is formulated as a constrained Markov decision process (CMDP) minimizing long-term weighted mismatch between physical and virtual states under an average RB constraint. The authors transform the constraint into a state-wise constraint, introduce a Lagrangian dual problem, and propose a continual reinforcement learning algorithm (MTR-SAC) that combines a multi-timescale replay buffer, soft actor-critic, and invariant risk minimization. Simulations with Intel Berkeley sensor data and indoor positioning data report that the proposed algorithm reduces NRMSE by up to 55.2% against DP and polling baselines using the same number of RBs, and converges faster than standard SAC when the RB budget changes.","tokens_in":21551,"tokens_out":4643,"duration_ms":48038,"significance":"If the empirical claims hold, the paper offers a practical scheme for resource-efficient DT synchronization in dynamic networks, with comparisons against polling and DP baselines on real-world datasets. The strengths are the clear system model, the inclusion of packet errors and mismatch losses, and the evaluation of state-wise constraint satisfaction. However, the theoretical support for the dual transform and the convergence guarantee is incomplete, and the state representation omits the resource budget M that changes in the key continual-learning experiment. The central contribution is a plausible integration of known components (MTR, SAC, feasible actor-critic, IRM) rather than a fundamentally new method.","major_comments":[{"comment":"The state S_t is defined as [s_{1,t}, ..., s_{N,t}] with s_{n,t} = [Phi_{n,t}, Y_{n,t}, gamma_t], and no component encodes the current RB budget M. Yet the cost in Eq. (15) is M when b^T a_t <= M and b^T a_t otherwise, and the multiplier lambda_zeta(S_t) and cost critic Q_phi_C(S_t, a_t) take only S_t as input. When M changes from 30 to 10 to 26 in Fig. 4, the same state-action pair receives conflicting cost labels and conflicting constraint thresholds, so a single function of S_t cannot represent the required mapping. The claimed quick adaptation to network capacity changes is therefore not supported by the printed formulation; the state should be augmented with M, and M should be an input to the actor, cost critic, and multiplier.","section":"Section III-B, State definition and Eq. (15)"},{"comment":"Theorem 1 states that if the Lagrangian dual problem of L_stw has an optimal scheduling vector sequence u* and multiplier lambda*, then u* is also optimal for the state-wise constrained problem P2. The proof in Appendix A only rewrites the constraint using d0(Z) and shows that L'_stw(u,lambda) is the Lagrangian of the reformulated problem (29). It does not prove strong duality, zero duality gap, or that u* is feasible for the state-wise constraint (10a). Without these steps, the equivalence between P3 and P2 is not established, and the claim that the dual transform refines the RB constraint impact is not rigorously justified.","section":"Theorem 1 and Appendix A"},{"comment":"The convergence proof of Lemma 1 does not establish the claimed guarantee. The argument 'we can always choose pi_new = pi_old in the invariant distribution' only gives a trivial equality, not the required monotonic improvement Q^{pi_new} >= Q^{pi_old}. Moreover, the reduction using Lemma 2 in [40] ignores that the actor loss in Eqs. (22)-(23) contains the IRM term and the multiplier term lambda_zeta(S_t)(Q_phi_C - M), which are not part of the standard SAC loss for which [40] provides convergence conditions. The authors should either prove convergence under explicit conditions for the modified loss or state clearly that convergence is an empirical observation rather than a guarantee.","section":"Lemma 1 and Section III-D1"},{"comment":"The reward in Eq. (14) is defined on the full mismatch vector Z_t, while the agent's state uses only the last received mismatch Y_{n,t} and the time since last successful reception Phi_{n,t}. The paper does not justify that this proxy is a sufficient statistic for near-optimal scheduling in the CMDP. Since the BS cannot observe the current mismatch of unscheduled devices, the learned policy may not track the true objective in deployment; the discrepancy between the full-information reward used in training and the partial-information state used at execution should be analyzed or at least discussed as an approximation.","section":"Section III-B, Reward and State"}],"minor_comments":[{"comment":"The expression for \\bL_stw has an extra closing parenthesis: it should read lambda_zeta(S_t)(c(S_t,a_t) - M) rather than lambda_zeta(S_t)(c(S_t,a_t)) - M).","section":"Eq. (16)"},{"comment":"The stochastic gradient for the cost critic uses Q_phi_C(S_t, a_t) in the target term, but the loss in Eq. (19) defines the target with Q_phi_C(S_{t+1}, a_{t+1}); the gradient expression should be corrected for consistency.","section":"Eq. (20)"},{"comment":"Line 8 writes 'phi <- phi - beta_Q grad ... for i in {1,2,C}', which mixes the reward critics phi_1, phi_2 and the cost critic phi_C; the update rule should distinguish these parameters explicitly.","section":"Algorithm 1"},{"comment":"The subfigure labels are duplicated and inconsistent (for example, '(a)M=9 (a)M=36 (a)M=27 (a)M=18' appears in both figures); the labels should be corrected to a-d with the corresponding M values.","section":"Figures 10 and 11"},{"comment":"The notation bX_{n,t} is used where the hat is presumably meant; please use a consistent notation such as \\hat{X}_{n,t} for the virtual state.","section":"Section II-A"},{"comment":"The text says 'seek a prime-dual solution'; this should be 'primal-dual solution'.","section":"Section III-A"}],"recommendation":"major_revision","confidential_remarks":"The paper is an application-oriented integration of known components: multi-timescale replay from [36], SAC from [40], state-wise Lagrangian constraints from [37], and IRM from [42]. The novelty is moderate but the application to DT synchronization is reasonable. The missing M in the state is a fixable but load-bearing flaw: it directly affects the capacity-adaptation experiment, which is the central continual-learning claim. The theoretical gaps in Theorem 1 and Lemma 1 are also fixable by either adding rigorous arguments or softening the claims. I would be willing to see a revised version, but the revision needs to address the state representation and the theoretical support before the paper can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a readable engineering paper with a genuine integration novelty—multi-timescale replay plus soft actor-critic on a statewise Lagrangian for digital twin scheduling under a variable RB budget. The simulated NRMSE gains over polling and DP are consistently in the right direction, and the paper does not appear to fit constants to produce the 55.2% headline; it is a measured simulation number. That part is honest and worth engaging with.\n\nThe soft spots are real but fixable. The most important one, which the stress-test note gets right: M is absent from the state S_t, yet the cost in Eq. (15) is M when under budget and b^T a_t otherwise. When M changes from 30 to 10 to 26, the same (S,a) pair gets conflicting cost labels and conflicting constraint thresholds, while the cost critic, multiplier network, and actor see only S_t. The networks cannot cleanly condition on the quantity that changed, so Fig. 4's \"quick adaptation to new capacity\" is at best an indirect effect of extra gradient updates, not a policy that knows the new M. The stress-test says a single function 'cannot represent both'; strictly, M could leak through stale-mismatch and last-reception features, but the paper does not show that, and the cost-critic conflict is real. Fix: add M to the state, or argue sufficiency with data.\n\nSecond, Theorem 1 assumes an optimal dual pair and concludes primal optimality without proving zero duality gap; Appendix A only rewrites the constraint. Lemma 1's convergence argument leans on [40, Thm 1] without verifying assumptions. These weaken the theory but do not kill the empirical claim. Third, Eq. (20) has a typo: the target should use Q(St+1, at+1), not Q(St,at). Fourth, no error bars, single traces, and no reproducibility artifacts make the headline number fragile—though not fraudulent.\n\nFor whom: readers working on RL-based DT synchronization or constrained resource allocation will find the formulation and component choices useful. It deserves a serious referee. A major revision should address the M-in-state issue, fix Eq. (20), add multiple-seed results, and either strengthen Theorem 1 or soften its claim.","headline":"A plausible simulation result with a real integration gap: the resource budget M never enters the state, so the continual-learning adaptation claim in Fig. 4 is not supported as written.","tokens_in":21993,"tokens_out":3061,"would_cite":false,"duration_ms":33267,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A continual reinforcement learning scheduler keeps a factory digital twin accurate while using the same wireless resource budget, reducing state-estimation error by up to 55.2%.","keywords":["digital twin","constrained Markov decision process","continual reinforcement learning","device scheduling","resource block allocation","soft actor-critic","multi-timescale replay","invariant risk minimization"],"falsifier":"Run the trained MTR-SAC policy in a simulator where the true mismatch of an unscheduled device evolves quickly between transmissions while its last received mismatch and elapsed time stay unchanged; if the scheduler keeps skipping that device and NRMSE climbs toward the polling baseline, the proxy state is not a sufficient statistic. A cleaner check is to give the agent the true current mismatch vector as state in the same simulator and compare NRMSE; a large gap would indicate the partial-observability assumption is the limiting factor.","tokens_in":21033,"feed_emoji":"📡","tokens_out":4332,"duration_ms":40794,"temperature":0.7,"pith_summary":"The paper claims that a base station can keep a factory digital twin closely matched to physical devices by selecting only a few devices to upload data at each time, and that a continual reinforcement learning algorithm can learn this selection without retraining when network capacity changes. A sympathetic reader would care because wireless resources are too scarce to transmit all sensor data continuously; a scheduler that learns which mismatches matter could cut signaling load while preserving twin fidelity. The paper supports this with a constrained Markov decision process formulation, solves it through a state-wise Lagrangian dual, and simulates with real sensor data, showing up to 55.2% lower normalized root mean square error between physical and virtual states for the same number of resource blocks.","feed_headline":"Scheduler cuts digital-twin error 55.2% at same resource budget","feed_subtitle":"A continual learning algorithm tells the base station which sensors matter, keeping the virtual factory accurate with fewer uploads.","key_machinery":"The central object is the multi-timescale replay (MTR) buffer, a cascade of FIFO sub-buffers plus an overflow buffer that stores experiences with a power-law age distribution, so both fresh and old experiences remain available for training. It is combined with a soft actor-critic (SAC) agent, a Lagrange multiplier network that enforces the state-wise resource-block constraint, and an invariant risk minimization (IRM) penalty in the actor loss that stabilizes the policy across changing environments. The state for each device is $[\\Phi_{n,t}, Y_{n,t}, \\gamma_t]$: time since the last correctly received packet, the last reported mismatch, and the data reception indicator. This machinery carries the argument because it prevents catastrophic forgetting when the resource budget changes, letting the policy re-converge quickly.","core_discovery":"The central discovery is that mismatch minimization for digital twin synchronization can be posed as a state-wise constrained Markov decision process, and its Lagrangian dual can be learned by an SAC variant augmented with a multi-timescale replay buffer and an invariant risk minimization penalty. The paper shows in simulation that the resulting MTR-SAC policy adapts to abrupt changes in the number of available resource blocks (from 30 to 10 to 26), converges faster than standard SAC after each change, and reduces normalized root mean square error by up to 55.2% relative to DP and polling methods using the same number of resource blocks. The state-wise constraint formulation also keeps instantaneous resource consumption at or below the budget, unlike the expectation-constraint formulation, which temporarily exceeds the budget in resource-scarce settings.","pith_inferences":["The same MTR-SAC structure could be applied to other constrained scheduling problems with changing budgets, such as age-of-information or semantic-value scheduling, because the framework separates the reward from the constraint and stabilizes policy learning across budgets; this is an extension the paper does not test.","The reported 55.2% gain depends on physical-state dynamics that are smooth enough that the last-received mismatch and elapsed time remain informative; on faster or more erratic dynamics, the proxy state may lose information and the advantage would likely shrink.","The interaction between the IRM penalty and the Lagrange multiplier network is not analyzed separately; testing that interaction could identify when invariant policy learning and constraint satisfaction conflict.","A direct ablation comparing the proxy state with an oracle state containing the true current mismatch vector would isolate how much of the gain comes from the scheduling policy versus how much is lost to partial observability."],"forward_implications":["At the same resource-block budget, the MTR-SAC scheduler reduces NRMSE by up to 55.21% compared with DP and 68.42% compared with polling at M=15.","To achieve a target NRMSE of 0.02, the proposed method consumes up to 55.56% fewer resource blocks than DP and 61.54% fewer than polling, making fidelity gains translate directly into spectrum savings.","When the resource budget abruptly changes, MTR-SAC converges faster than standard SAC because the IRM penalty and multi-timescale replay retain common knowledge across budgets.","Under scarce resources (M=9 or 18), the state-wise constraint keeps instantaneous resource consumption at or below the budget, whereas the expectation-constraint formulation exceeds it.","As device count grows with fixed resource blocks, MTR-SAC degrades more slowly than DP and polling, indicating scalability to denser sensor deployments."],"supporting_citations":[{"why":"Supplies the multi-timescale replay buffer that stores old and new experiences with a power-law age distribution.","marker":"[36]"},{"why":"Provides the state-wise constraint and Lagrange multiplier formalism used to turn the CMDP into a dual problem.","marker":"[37]"},{"why":"Forms the dynamic programming baseline that schedules devices at fixed time intervals.","marker":"[38]"},{"why":"The soft actor-critic algorithm that the proposed method extends with MTR and IRM penalties.","marker":"[40]"},{"why":"Invariant risk minimization loss used to keep the policy stable across changing resource budgets.","marker":"[42]"},{"why":"Temperature and humidity traces from Intel Berkeley lab serve as physical-state data in simulation.","marker":"[45]"},{"why":"Indoor positioning data serve as location-state data in simulation.","marker":"[46]"}],"fun_headline_variants":["CRL cuts digital-twin error 55.2% at same resource budget","Continual RL for digital twin sync cuts error 55.2%","MTR-SAC trims twin sync error 55.2% under resource constraints","State-wise constrained RL keeps digital twin sync within budget","SAC variant reduces digital-twin mismatch 55.2% in simulations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that a device's time since its last successful upload and its most recently reported mismatch are enough information for the base station to make near-optimal scheduling decisions, even though the reward is defined using the full current mismatch of all devices.","fun_headline_variants_meta":{"raw":{"variants":["CRL cuts digital-twin error 55.2% at same resource budget","Continual RL for digital twin sync cuts error 55.2%","MTR-SAC trims twin sync error 55.2% under resource constraints","State-wise constrained RL keeps digital twin sync within budget","SAC variant reduces digital-twin mismatch 55.2% in simulations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001103,"raw_usage":{"total_tokens":4583,"prompt_tokens":910,"completion_tokens":3673,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":3575}},"tokens_in":526,"tokens_out":3673,"duration_ms":26574,"temperature":1.0,"reasoning_tokens":3575,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:31:22.025763+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained MTR-SAC policy in a simulator where the true mismatch of an unscheduled device evolves quickly between transmissions while its last received mismatch and elapsed time stay unchanged; if the scheduler keeps skipping that device and NRMSE climbs toward the polling baseline, the proxy state is not a sufficient statistic. A cleaner check is to give the agent the true current mismatch vector as state in the same simulator and compare NRMSE; a large gap would indicate the partial-observability assumption is the limiting factor.","supporting_citations":[{"cited_title":"Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,","cited_arxiv_id":null,"evidence_quote":"The soft actor-critic algorithm that the proposed method extends with MTR and IRM penalties."},{"cited_title":"The age of incorrect in- formation: an enabler of semantics-empowered communication,","cited_arxiv_id":null,"evidence_quote":"Forms the dynamic programming baseline that schedules devices at fixed time intervals."},{"cited_title":"Intel berkeley research lab sensor data,","cited_arxiv_id":null,"evidence_quote":"Temperature and humidity traces from Intel Berkeley lab serve as physical-state data in simulation."},{"cited_title":"Indoor received signal strength data generated from ray- tracing,","cited_arxiv_id":null,"evidence_quote":"Indoor positioning data serve as location-state data in simulation."}],"review_version":1}