{"id":"f3a2cf95-ee3a-452b-8b00-587b33d32b8e","arxiv_id":"2411.13121","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ReinFog is a modular framework for using deep reinforcement learning to schedule IoT tasks across edge, fog, and cloud, with a hybrid optimization algorithm for placing its learning components.","lead":"ReinFog is a software framework that lets developers plug deep reinforcement learning into edge, fog, and cloud systems to schedule Internet of Things tasks. The authors report cutting response time, energy use, and cost by roughly a third to a half compared with an older scheduling method, though the comparison is limited.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 45/39/37% reductions rest on an underspecified, self-extended OHNSGA baseline and no error bars; until the multi-objective extension and per-run variance are reported, the central performance claim is not yet supported.","rationale":"The reader's weakest_assumption correctly identifies the baseline fairness and missing variance as the key evidential weakness, and this pass confirms it is the load-bearing point. The architectural claims about centralized/distributed DRL integration are supported by the system description and the related-work table, but the quantitative headline depends entirely on a baseline extension that is not specified. A fair OHNSGA extension plus variance reporting would settle whether the 45/39/37% reductions are real. Because the architecture contribution remains plausible and the issue is evidential rather than foundational, the reader's CONDITIONAL verdict should stand unchanged.","tokens_in":29169,"tokens_out":6833,"duration_ms":74633,"concrete_test":"Release the code and configuration for the OHNSGA extension and rerun the Section 6.5.1 evaluation with an independently reimplemented, fully specified multi-objective OHNSGA using Eq. (23)'s weighted-sum scalarization with w1=w2=0.5, or NSGA-II with an explicit knee-point rule, tuned on identical training workloads and evaluated on identical 240p workloads, with 30 independent seeds and paired confidence intervals. If the best DRL technique does not show a statistically significant reduction over this fair baseline, the abstract's 45/39/37% numbers should be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is the baseline for the headline quantitative claim. In Section 6.1, OHNSGA is described only as a GA-based meta-heuristic algorithm natively supported by FogBus2 that the authors extended to support multi-objective optimization. The extension is never specified: the paper does not state OHNSGA's fitness function, how or whether its multi-objective objective aligns with the weighted cost in Eq. (23), how a single solution is selected from a Pareto front, or whether its hyperparameters were tuned on the same training workload as the DRL agents. The subsequent comparison in Section 6.5.1 reports reductions of up to 45/39/37% against this baseline, but if the extension used an ad hoc scalarization or an unfavorable weight selection, those percentages are inflated and are not a valid measure of ReinFog's advantage. This matters because the performance claim, not just the architecture, is what the abstract leads with. In addition, Section 6 reports only averages over 10 runs with no standard deviations or confidence intervals, so it is impossible to tell whether the reported gaps are stable or within run-to-run noise. No comparison against an existing DRL-based scheduler is provided, which would be the natural control for the 'first comprehensive DRL framework' claim. This is a specification gap, not an accusation of bad faith; the claim can be fixed by precise disclosure of the baseline and variance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ReinFog, a containerized and modular framework for IoT application scheduling in edge/fog and cloud environments, built on top of FogBus2. It supports centralized and distributed DRL techniques, both natively implemented and imported from external libraries (e.g., Ray), with customizable deployment of DRL Learners and Workers. It also presents MADCP, a memetic algorithm combining genetic algorithm, firefly algorithm, and particle swarm optimization for placing DRL components. Experiments compare six DRL techniques (IMPALA, A3C, PPO, DQN, R2D2, SAC) against a GA-based heuristic OHNSGA, reporting reductions of up to 45% in response time, 39% in energy consumption, and 37% in weighted cost, as well as low startup/RAM overhead and good scalability. The paper claims that ReinFog is the first framework comprehensively integrating centralized and distributed DRL techniques for IoT application scheduling, and that MADCP accelerates DRL convergence by up to 38%.","tokens_in":29432,"tokens_out":6017,"duration_ms":56396,"significance":"If its quantitative claims were substantiated, ReinFog would be a valuable contribution as a flexible testbed for DRL-based resource management research. The modular architecture—separating DRL Learners from Workers, providing both native and library-based integration, and adding a placement optimization component—addresses a real gap, since most existing frameworks are heuristic-based or support only a single RL technique. The implementation of six DRL techniques within one framework is a substantial engineering effort, and the scalability measurements (RAM and startup time) are concrete and worth reporting. However, the empirical evidence does not currently support the headline performance claims: the only baseline is an underspecified extension of the authors' own OHNSGA algorithm, no variance or statistical tests are reported, and no independent DRL-based scheduler is compared. No code or artifact is released, limiting reproducibility. With a properly specified baseline, variance reporting, and an independent comparison, the framework contribution could be significant; as it stands, the central performance claims are not yet established.","major_comments":[{"comment":"The OHNSGA baseline is not sufficiently specified: the paper states in §6.1 that the authors 'extended this technique to support multi-objective optimization' but never defines the extension's fitness function, the multi-objective handling method (e.g., weighted sum versus Pareto dominance), how a single solution is selected from a Pareto front, or how the normalization in Eq. (23) interacts with this baseline. Consequently, the headline reductions of up to 45/39/37% reported in §6.5.1 are not interpretable as a valid measure of ReinFog's advantage; they could reflect an unfavorable or ad hoc baseline construction rather than a genuine improvement.","section":"§6.1, §6.5.1"},{"comment":"The paper reports only averages over 10 repeated experiments with no standard deviations, confidence intervals, or statistical significance tests. In the convergence comparisons (Figs. 10 and 11) and the scalability analysis (Fig. 13), the absence of variance information makes it impossible to determine whether the observed gaps between techniques are stable or within run-to-run noise; the claim of 'significant improvements' is therefore not statistically supported.","section":"§6, §6.5.1"},{"comment":"No comparison is made against any existing DRL-based scheduling framework or technique, despite the paper's claim of being 'the first framework that comprehensively integrates both centralized and distributed DRL techniques.' The qualitative novelty argument in Table 1 is a self-assessed feature matrix, and the quantitative evaluation compares only against OHNSGA, a baseline from the authors' own prior FogBus2 work; without an independent DRL scheduler comparison, the claims of superiority over prior DRL-based approaches are not supported.","section":"§6.5, Table 1"},{"comment":"The weighted cost metric in Eq. (23) uses a normalization function Norm(x) 'relative to a predefined baseline,' but the baseline is never specified anywhere in the manuscript. Since weighted cost is used as a primary metric in the convergence, scalability, and overhead analyses, the reported numeric values (e.g., 0.14–0.25 in Fig. 13) cannot be reproduced or interpreted without knowing what that baseline is.","section":"§6.3, Eq. (23)"},{"comment":"The claim that MADCP 'accelerates the convergence rate by up to 38%' compared to random placement is not backed by a precise definition of convergence rate; neither the number of iterations to stabilization nor a threshold-based definition is given. Figure 15 shows curves, but the quantification is informal and no variance or statistical test accompanies the claim.","section":"§6.6.1"}],"minor_comments":[{"comment":"The concluding paragraph contains a grammatical error: 'It offers capabilities to supporting centralized and distributed DRL techniqueintegrate both native and library-based DRL techniques, whiles.' This should be rephrased for clarity.","section":"§7"},{"comment":"The CO2 emission values (e.g., 0.99g per hour for Australia) lack context; please clarify whether these figures are per application or for the entire environment, and state the underlying electricity consumption and emission-factor calculations.","section":"§6.4.3"},{"comment":"In the overhead comparison, DQN and OHNSGA are both around 8 ms, yet the text says 'OHNSGA exhibits the lowest overhead'; this should be qualified as 'lowest among all techniques' or 'tied with DQN' to avoid an inaccurate impression.","section":"§6.5.2"},{"comment":"Table 2 lists hyperparameters for DRL techniques but omits the OHNSGA parameters (population size and generations) that are discussed in the text; including them in the table would improve completeness.","section":"§6.2"},{"comment":"The figure captions do not indicate the number of runs or the error-bar representation; please add a note that all curves are averages over 10 runs and state where the variance information is (or is not) shown.","section":"Figures 10, 11, 13"}],"recommendation":"major_revision","confidential_remarks":"The paper's self-citation pattern deserves editorial attention: the baseline OHNSGA comes from the authors' own FogBus2 work, and FogBus2 is also the foundation of ReinFog. This does not, by itself, invalidate the claims, but it strengthens the requirement that the baseline extension be specified fully enough for independent re-implementation. Given the framework's claimed modularity, I also suggest requesting a statement on code availability in the revision, since no repository or artifact link is provided."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short take: this is a genuine engineering contribution — a modular DRL scheduling framework built on FogBus2 — but the empirical headline needs work before it can be trusted.\n\nWhat's actually new: ReinFog is the first framework I know of that packages both centralized and distributed DRL techniques for IoT scheduling in edge/fog/cloud, with a clean separation between DRL Learners and Workers and a dual integration path (native implementations plus external libraries like Ray). That modular design is genuinely useful; the detail on components and the deployment modes is concrete. MADCP, a hybrid of GA/FA/PSO for placing DRL components, is the least novel piece, but the complexity analysis is honest and the convergence improvement over random placement is plausible.\n\nThe experiments are a real effort: real cloud VMs, a MacBook, a Raspberry Pi, multiple DRL techniques, repeated 10 times. I believe the framework works and is lightweight. That said, the quantitative claims are weaker than the abstract suggests. The comparisons against OHNSGA are load-bearing, and the paper never specifies how OHNSGA was extended for multi-objective optimization — fitness function, scalarization, weight selection, Pareto handling. Without that, the 45/39/37% reductions are not anchored. No error bars or standard deviations are reported anywhere, so we cannot assess run-to-run stability. And there is no comparison against an existing DRL-based scheduler, which would be the natural control for the \"first comprehensive DRL framework\" claim.\n\nThe citation pattern is not a problem: extending FogBus2 is a legitimate move and self-citation to the base framework is expected. The bigger reproducibility gap is the absence of code or a data link — and for a framework paper, that matters.\n\nWho this is for: researchers doing DRL-based scheduling in edge/fog systems who need a starting architecture, and readers surveying frameworks. It is not a theory paper, and the performance claims should be treated as provisional.\n\nRecommendation: send to peer review. A serious referee can push the authors to disclose the baseline extension, add variance reporting, and release the code. The engineering is real and worth refereeing; the numbers just need to be made credible.","headline":"ReinFog is a genuine, well-architected systems contribution for DRL-based scheduling, but the headline 45/39/37% gains are not yet supported because the OHNSGA baseline extension is unspecified and no variance is reported.","tokens_in":29973,"tokens_out":2103,"would_cite":false,"duration_ms":22020,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ReinFog brings centralized and distributed deep reinforcement learning to IoT scheduling across edge, fog, and cloud, reporting 45% lower response time, 39% lower energy, and 37% lower weighted cost than a genetic-algorithm baseline.","keywords":["Internet of Things","Edge Computing","Fog Computing","Cloud Computing","Deep Reinforcement Learning","Distributed Software Systems","IoT Application Scheduling","Resource Management"],"falsifier":"Reproduce the evaluation on the same four IoT applications and testbed with OHNSGA replaced by a comparably tuned standard multi-objective optimizer such as NSGA-II with identical objectives, and report the spread across the paper's 10 runs; if the 45%, 39%, and 37% gaps shrink to within run-to-run variability, the central performance claim would not survive.","tokens_in":28951,"feed_emoji":"🧠","tokens_out":12332,"duration_ms":101744,"temperature":0.7,"pith_summary":"ReinFog is a containerized, modular framework for scheduling IoT applications across heterogeneous edge, fog, and cloud nodes, with deep reinforcement learning at the core of the scheduling decisions. The paper's central claim is that a single framework can host both centralized and distributed DRL techniques, as native implementations or as imported libraries, and that these DRL schedulers beat traditional meta-heuristic scheduling on several objectives at once. Measured against OHNSGA, a genetic-algorithm scheduler from the FogBus2 framework that ReinFog extends, the DRL techniques reduce response time by 45%, energy consumption by 39%, and weighted cost by 37%, while keeping per-decision overhead in the 8–28 ms range. The paper also introduces MADCP, a memetic algorithm that places DRL Learners and Workers across nodes, which it reports accelerates training convergence by up to 38%.","feed_headline":"DRL scheduling cuts IoT response time 45%, energy 39%","feed_subtitle":"ReinFog unifies deep RL methods for edge-cloud scheduling, beating the GA baseline on all metrics.","key_machinery":"The Worker/Learner split is the load-bearing architectural pattern: DRL Workers carry the Environment Engine (state processing, action conversion, reward generation) and emit scheduling decisions, while DRL Learners carry the DRL Core (exploration engine, network architecture hub, DRL technique repository, policy constructor) and update the global policy. The DRL Technique Repository is the integration point that matters, offering a base class for native technique implementations and an adapter interface for external libraries. MADCP is the proposed memetic placement algorithm, combining genetic-algorithm operations, firefly movement, and particle-swarm velocity updates to solve the NP-hard assignment of DRL components to nodes under CPU, memory, and deadline constraints, with overall complexity $O(G \\times P^2 \\times M)$. The scheduling problem itself models each IoT application as a directed acyclic graph of dependent tasks, and the reward function penalizes response time, energy consumption, and their weighted combination.","core_discovery":"ReinFog's core claim is that a modular, containerized framework can make deep reinforcement learning practical for IoT application scheduling in edge/fog/cloud environments without sacrificing the flexibility researchers need. The design separates the learning system into DRL Workers, which interact with the environment and produce scheduling decisions, and DRL Learners, which aggregate experience and optimize the global policy, an arrangement that spans the spectrum from a single centralized agent (DQN) to distributed actor-learner architectures (A3C, IMPALA). The same modular core lets users implement DRL techniques natively or import mature libraries, with R2D2 and SAC from the Ray library serving as the integration examples. On a testbed mixing cloud VMs, laptops, and a Raspberry Pi, the DRL schedulers converge to substantially better schedules than the GA-based OHNSGA baseline across response time, energy, and a weighted combined cost, with linear scaling as Workers grow from 1 to 30. The framework's placement engine, MADCP, treats the placement of DRL components as a constrained combinatorial optimization problem and shows up to 38% faster convergence than random placement.","pith_inferences":["A stronger test than OHNSGA alone would pit the framework's DRL schedulers against well-tuned NSGA-II variants and against other DRL schedulers; if the gap shrinks, the headline numbers would read as 'DRL versus one GA baseline' rather than 'DRL versus meta-heuristics in general.'","Because the paper does not ablate the framework from the algorithms, running the same DRL techniques standalone would separate the contribution of the reward design and component placement from the contribution of the algorithms themselves.","The Worker/Learner interfaces are general enough to host techniques the paper does not try, such as multi-agent reinforcement learning, offline RL from logged schedules, or privacy-preserving federated policy sharing — the last being exactly the security direction the paper lists as future work.","The three-region CO2 analysis implies an operational consequence worth testing in practice: DRL-based scheduling cuts the carbon footprint of IoT processing most where the electricity mix rewards the energy savings."],"forward_implications":["The same framework hosts centralized agents (DQN, PPO) and distributed actor-learner systems (A3C, IMPALA), so researchers can compare scheduling techniques without rebuilding environment interfaces for each one.","DRL scheduling is light enough for real deployments: decisions cost 8–28 ms, and growing from 1 to 30 DRL Workers adds only about 0.3 seconds of startup time and roughly 60 MB of RAM.","Across all three objectives, the DRL techniques beat the GA-based OHNSGA scheduler, with IMPALA the strongest in convergence speed and scalability.","Where the learning components sit matters: MADCP placement accelerates convergence by up to 38% and roughly halves the per-decision overhead of random placement.","The trained schedulers generalize to workloads they did not train on, as gains persist when evaluation video resolution drops from 480p to 240p."],"supporting_citations":[{"why":"Serves as the foundation framework ReinFog extends and supplies the OHNSGA heuristic baseline that all headline performance comparisons are measured against.","marker":"[28]"},{"why":"Supplies the importance-weighted actor-learner architecture (IMPALA) that ReinFog implements natively and that delivers the best convergence and scalability results.","marker":"[33]"},{"why":"Supplies the clipped-objective policy optimization technique (PPO) used as a centralized native technique in the experiments.","marker":"[32]"},{"why":"Supplies the deep Q-network technique that anchors the centralized deployment illustration and the slowest-converging native baseline.","marker":"[34]"},{"why":"Supplies the asynchronous advantage actor-critic technique (A3C) demonstrating the distributed native deployment path.","marker":"[40]"},{"why":"Demonstrates the library-based integration path via the Ray library with a recurrent distributed DQN variant.","marker":"[36]"},{"why":"Demonstrates library-based integration of an off-policy maximum-entropy technique (SAC) from the Ray library.","marker":"[42]"}],"fun_headline_variants":["ReinFog DRL cuts IoT latency 45%, energy 39%","Deep RL framework ReinFog: 45% faster IoT response, 39% energy","ReinFog: DRL scheduling reduces IoT energy 39%, response 45%","MADCP placement accelerates DRL convergence by 38%","ReinFog scales: 30 DRL workers add 0.3s start, 2MB RAM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported 45%, 39%, and 37% gains are measured against a single genetic-algorithm scheduler that the authors extended themselves, so the comparison assumes that extended baseline is a fair and representative opponent.","fun_headline_variants_meta":{"raw":{"variants":["ReinFog DRL cuts IoT latency 45%, energy 39%","Deep RL framework ReinFog: 45% faster IoT response, 39% energy","ReinFog: DRL scheduling reduces IoT energy 39%, response 45%","MADCP placement accelerates DRL convergence by 38%","ReinFog scales: 30 DRL workers add 0.3s start, 2MB RAM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000925,"raw_usage":{"total_tokens":4033,"prompt_tokens":1084,"completion_tokens":2949,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":700,"completion_tokens_details":{"reasoning_tokens":2836}},"tokens_in":700,"tokens_out":2949,"duration_ms":20900,"temperature":1.0,"reasoning_tokens":2836,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:47:52.982750+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reproduce the evaluation on the same four IoT applications and testbed with OHNSGA replaced by a comparably tuned standard multi-objective optimizer such as NSGA-II with identical objectives, and report the spread across the paper's 10 runs; if the 45%, 39%, and 37% gaps shrink to within run-to-run variability, the central performance claim would not survive.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Serves as the foundation framework ReinFog extends and supplies the OHNSGA heuristic baseline that all headline performance comparisons are measured against."},{"cited_title":"Espeholt, H","cited_arxiv_id":null,"evidence_quote":"Supplies the importance-weighted actor-learner architecture (IMPALA) that ReinFog implements natively and that delivers the best convergence and scalability results."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the asynchronous advantage actor-critic technique (A3C) demonstrating the distributed native deployment path."},{"cited_title":"Kapturowski, G","cited_arxiv_id":null,"evidence_quote":"Demonstrates the library-based integration path via the Ray library with a recurrent distributed DQN variant."},{"cited_title":"Haarnoja, A","cited_arxiv_id":null,"evidence_quote":"Demonstrates library-based integration of an off-policy maximum-entropy technique (SAC) from the Ray library."}],"review_version":1}