{"id":"80a31c8c-ef07-4ffb-8882-251e68bb97f7","arxiv_id":"2607.22805","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"OrchNAS uses a cloud NAS service to search global architectures and prune per-device subnets so federated edge models meet device energy, compute, and memory budgets.","lead":"OrchNAS is a server-side neural architecture search service that designs personalized, energy-aware models for federated edge devices. It combines global architecture search, per-device pruning, and a primal-dual optimizer to enforce device energy budgets, which matters for deploying AI on battery-limited phones, wearables, and drones.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Energy claims rest entirely on an unvalidated FLOPs proxy (Eq. 3); hardware measurement is required.","rationale":"The paper proposes a federated NAS system with the central empirical claim that it 'consistently improves accuracy and significantly reduces energy consumption compared to existing PFL with NAS-based approaches' (Section V-F). For that claim to hold, the quantity called 'energy' must correspond to physical energy consumption on edge devices. The manuscript defines energy as E_k(a_k)=α_k F(a_k) T_k (Eq. 3), with the same functional form reused in the global search score (Eq. 7), the pruning objective (Eq. 13), the constraints (Eq. 11), and the dual update (Eq. 23). All table entries reporting 'Energy (J)' are thereby derived from FLOPs through a constant device coefficient. No experiment measures power, voltage, or wall-clock time; no α_k is calibrated against a real device; no communication or memory-access energy is included. This is not merely an approximation: for many edge workloads, memory access and radio transmission dominate energy, and the relationship between FLOPs and energy is highly architecture- and hardware-dependent. If the proxy diverges from physical reality, then the claimed energy reductions, feasibility rates, and budget-violation behavior are artifacts of the proxy, and the framework's practical value is unestablished. This is the single most load-bearing concern because it directly invalidates the quantitative contribution, regardless of whether the algorithms are internally coherent. The theorem in Section IV-D is also weak—it assumes convexity, cites ADMM without derivation, and bounds violations of the same proxy—but it is secondary. The paper does have strengths: a clear problem formulation, detailed algorithms, a public code repository, and experiments across multiple datasets. Those do not compensate for the absence of hardware validation. The reader's verdict of CONDITIONAL is appropriate; our concern reinforces it. The proposed hardware-measurement test would settle whether the concern lands. If it does, the central energy claims should be downgraded to claims about FLOPs reduction, which is a much weaker and already known benefit.","tokens_in":14581,"tokens_out":4299,"duration_ms":51769,"concrete_test":"Re-run the final selected subnets from Table I (and from the baselines) on a real edge device (e.g., NVIDIA Jetson Nano or Raspberry Pi 4) with a power monitor, measuring per-round energy for the same local epochs. Compare measured relative energy reduction between OrchNAS and the best baseline against the reported reduction. Also compute Spearman rank correlation between predicted E=α·FLOPs·T and measured energy across the pruned family. If the rank correlation is <0.8 or the measured energy reduction is lower than 50% of the reported value, the central energy claims are unsupported and the paper should be revised to present FLOPs as a proxy, not energy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claims of accuracy improvement and significant energy reduction (Section V-B, Table I) are computed entirely from the energy model E_k(a_k) = α_k F(a_k) T_k (Eqs. 3, 6, 10). This model is used in the energy-aware score (Eq. 7), feasibility constraints (Eq. 11), pruning scores (Eqs. 15–16), dual updates (Eq. 23), and in all reported 'energy' numbers in Tables I–IV. Neither α_k nor the FLOPs-to-energy relationship is measured; energy per FLOP is treated as a constant for a device, ignoring memory-access costs, communication energy, idle power, and hardware-specific operator efficiency. For small edge models these terms are often the dominant fraction of energy. If the proxy is not faithful—e.g., a FLOPs-reducing operation like depthwise convolution may actually increase memory-bounded energy per inference—then the claimed 30% energy reduction, 92.6% feasibility rate, and 4.8% violation rate describe a synthetic quantity, not battery consumption. The paper's stated limitation (Section V-F) acknowledges missing latency/accuracy trade-offs but never addresses proxy validity. The theorem (Section IV-D) bounds violations of this model's energy, not physical energy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OrchNAS, a server-side NAS-as-a-service framework for personalized federated learning on heterogeneous edge devices. It defines an energy model E_k(a_k) = α_k F(a_k) T_k, uses evolutionary search with an energy-aware score to select a global architecture, derives per-service subnets by progressive greedy energy-aware pruning, and enforces energy budgets through a primal-dual update. Experiments on five datasets compare against SPIDER, PerFedRLNAS, diffusion-based NAS, Green-EDP, and MCFL, reporting higher accuracy and lower energy, with additional ablations, scalability, and sensitivity analyses. The public code link and breadth of experiments are positive features, but the energy claims rest entirely on an unvalidated surrogate model, and the theoretical guarantee is stated without a matching proof.","tokens_in":14837,"tokens_out":5735,"duration_ms":74168,"significance":"If the energy model were independently validated, the orchestrated NAS-service idea would be a useful practical contribution to personalized federated edge intelligence. The paper is broad: five datasets, several NAS/PFL baselines, component-wise ablations, scalability experiments, and publicly available code. The accuracy improvements over baselines (e.g., +1.1% to +2.2% with lower FLOPs in Table I) are independent of the energy-proxy issue and are the strongest part of the empirical contribution. However, the central energy-efficiency claim is currently an artifact of the paper's own energy definition: all reported Joules, energy reductions, feasibility rates, and violation rates are computed from E=αFT, with α never calibrated against hardware. The theorem in Section IV-D is also not proven for the actual nonconvex, pruning-based algorithm. These are load-bearing limitations that must be addressed before the paper's headline claims can be accepted.","major_comments":[{"comment":"The theorem is not derived for the setting actually used. The proof is two sentences citing [16] without identifying a supporting result; F_k for deep networks is nonconvex, the primal update is implemented as greedy pruning rather than a convex minimization, and the energy function is linear, not strongly convex. The claimed bound ε=O(ρ) does not follow from the proof's 'cumulative violation O(√T), divided by T' reasoning, which would give O(1/√T) and contains no ρ. Moreover the dual update Eq. (23) bounds only average violation; it cannot justify the text's repeated claims of 'strict' energy budget satisfaction. The theorem should be replaced with a statement whose assumptions match the algorithm and a complete proof, or removed with the primal-dual step presented as a heuristic with empirical support only.","section":"Section IV-D, Theorem 1 and Eq. (24)"},{"comment":"All reported energy values depend on the surrogate E_k(a_k)=α_k F(a_k) T_k, but α_k is never measured or specified and no hardware measurement validates the FLOPs-to-Joules mapping. For edge devices, memory access, communication, idle power, and hardware-specific operator efficiency can dominate; a FLOPs-reducing operation such as depthwise convolution may increase measured energy under memory-bound conditions. Thus the claimed 30% energy reduction, 92.6% feasibility rate, and 4.8% violation rate describe a synthetic quantity, not physical battery consumption. Section V-F explicitly notes missing latency trade-offs but never addresses proxy validity. The authors should either add real-device energy measurements (e.g., on representative edge hardware, reporting correlation/error relative to Eq. (3), and specifying how α_t_k is set) or rescope all energy claims as 'estimated FLOP-based cos","section":"Section III Eq. (3); Section V-B, Tables I-IV"},{"comment":"The energy-aware score, the pruning score, the primal objective, and the dual update all optimize exactly the same E=αFT that is later reported as 'energy consumption' in Tables I-IV. Consequently the observed energy reductions and constraint-satisfaction rates are consequences of the algorithm minimizing its own objective, not independent evidence of energy efficiency. The accuracy comparisons are independent and valuable, but the headline claim that OrchNAS 'significantly reduces energy consumption' is circular as currently presented. An external validation metric—hardware-measured energy, or at least a separately measured cost model not used in the training loop—is needed before the energy-efficiency claim can be supported.","section":"Section IV-B/C/D, Eqs. (7), (16), (22)-(23)"}],"minor_comments":[{"comment":"Algorithm 2 line 5 uses τ_t_k while Eq. (10) uses T_t_k. Please unify the notation for local training steps.","section":"Algorithm 2 vs Eq. (10)"},{"comment":"Typo: '(a_t_k)' appears where 'α_t_k' is intended.","section":"Eq. (10)"},{"comment":"The aggregation weight p_t_k is used in Eq. (8) but defined only later in Eq. (25). Define it before first use.","section":"Eqs. (8) and (25)"},{"comment":"The search space is described as 3×5×3×3×3=405. If depth is a dimension, the effective space over layers is larger unless each architecture is constrained to one global configuration; please clarify whether the enumerated 405 count is per network or per layer.","section":"Section V-A, search space"},{"comment":"The text says energy is 'estimated from the computational cost' but Table headers report units of J. Please either report hardware-measured energy or explicitly label the values as model-estimated 'FLOP-based cost' in the table and text.","section":"Section V-B, Table I"},{"comment":"The pruning loop calls Acc_k(a') for every candidate removal; the cost of these forward evaluations is not discussed. A complexity estimate or practical note would help reproducibility of the reported overheads.","section":"Algorithm 2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a broad experimental study and a clear system proposal, but the energy story is overstated: the surrogate E=αFT is never validated, and Theorem 1 is not actually proven. These are fixable with additional measurements or careful rescoping, so I do not recommend rejection. However, the current text should not be accepted as an energy-efficiency claim; the authors should be asked to either provide hardware validation or significantly weaken the claims. The accuracy and FLOPs comparisons could still form the basis of a solid contribution after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: OrchNAS fills a real gap. Existing personalized federated NAS methods mostly ignore energy, and this paper combines server-side NAS service, energy-aware evolutionary search, per-service greedy pruning, and primal-dual constrained personalization. I checked the cited work; the integrated orchestration does not appear there. That is a genuine contribution, and the paper is clearly written about a sensible system.\n\nWhat it does well: the problem is well motivated, the framework decomposition is reasonable, and the experiments span five datasets with relevant baselines. The accuracy gains are independent of the energy model and look consistent, so the core accuracy claim is probably fine. They also provide a code link, which is more than many papers do.\n\nWhere it is soft: First, Theorem 1 is not a proof. It assumes convexity for a problem that is explicitly nonconvex neural network training, cites Boyd's ADMM paper without stating a result, and the derivation gives cumulative violation O(sqrt(T)) while the theorem claims epsilon = O(rho). That mismatch should have caught an internal review. The paper also calls this 'strict energy budget enforcement,' but the theorem only bounds average violation over T rounds. That is an overstatement.\n\nSecond, and more load-bearing: the energy claims are entirely synthetic. Equation (3) defines training energy as alpha times FLOPs times T, and this same quantity appears in the search score, the pruning criterion, the feasibility constraints, and the dual update. Every 'energy' number in Tables I-IV is computed from that model. Alpha is never measured, and there is no hardware experiment. For small edge models, memory access and idle power often dominate FLOPs, so a FLOPs-based proxy can be badly misleading. The paper's own limitation paragraph mentions latency trade-offs but never addresses proxy validity. The stress-test note lands: the claimed 30% energy reduction and 92.6% feasibility rate describe a synthetic quantity, not battery consumption.\n\nMinor: no error bars anywhere, and the sensitivity analysis only varies lambda, not rho or beta. I would not call these fatal, but they weaken the empirical support.\n\nOverall: the framework is worth engaging with, and the novelty is real, but as stated the energy claims are not supported. The paper needs hardware measurement (even on a couple of devices) and a proper theorem statement and proof, or the energy framing should be downgraded to 'FLOPs reduction' throughout.\n\nIf this crossed my desk, I would send it to peer review with a clear request for major revision. The accuracy results and the orchestration idea justify referee time; the current energy evidence and theory do not justify acceptance. For a reading group, it would be a good case study on how proxy metrics can drive a whole experimental section.","headline":"A genuinely integrated energy-aware federated NAS framework, but the energy numbers are computed from an unvalidated FLOPs proxy and the main theorem is more asserted than proven.","tokens_in":882,"tokens_out":3365,"would_cite":true,"duration_ms":50664,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OrchNAS claims a server-side NAS service can design per-device subnets that meet energy, compute, and memory budgets while improving accuracy.","keywords":["federated learning","neural architecture search","edge intelligence","energy efficiency","personalization","primal-dual optimization","resource-constrained devices","NAS service"],"falsifier":"Run OrchNAS on two devices with identical FLOPs and local steps but different memory bandwidth or idle power, and compare actual battery drain with the model's predicted energy. If measured energy differs substantially while predicted energy is equal, the energy-aware scores and pruning decisions are optimizing a synthetic quantity, not physical energy.","tokens_in":14427,"feed_emoji":"⚡","tokens_out":6105,"duration_ms":66829,"temperature":0.7,"pith_summary":"OrchNAS is an attempt to make personalized federated learning practical on battery-powered edge devices by treating neural architecture search as a managed cloud service. The paper claims that a server-side orchestrator can learn a compact global architecture, then let each device prune that architecture down to a subnet that fits its energy, compute, and memory budget, and finally enforce the energy budget during training with a primal-dual update. The reported experiments on five datasets show higher accuracy than prior NAS-based PFL methods while cutting training energy and FLOPs. The value of the claim, if true, is that heterogeneous fleets of devices could each receive a suitably sized model without manual architecture design and without burning device battery on the search itself.","feed_headline":"Server-side NAS trims energy use in federated edge learning","feed_subtitle":"Per-device subnets are pruned and dual-updated to meet energy budgets, with accuracy gains on five datasets.","key_machinery":"The machinery is a three-level energy-aware design: (1) an evolutionary, energy-scored global architecture search on the server, (2) progressive greedy edge-wise pruning per service using score ΔAcc - βΔE until FLOPs, params, and energy constraints are feasible, and (3) a primal-dual Lagrangian personalisation step whose dual variable η_k is updated by projected gradient ascent on energy violation. The load-bearing identity is the energy model E = αFT, which turns architecture choices into a scalar energy cost, and the saddle-point formulation that converts the energy constraint into a penalty.","core_discovery":"The paper's central claim is that energy consumption in federated NAS can be made a first-class constraint rather than an afterthought. Energy per training round is modeled as E_k(a_k) = α_k F(a_k) T_k, so it is proportional to FLOPs and local steps. Architectures are scored as validation accuracy minus an energy penalty; each service greedily removes the operation with the lowest accuracy-loss-per-energy-saved score until its constraints hold; and a non-negative dual variable is raised whenever energy exceeds budget, making the energy constraint a saddle-point problem. Theorem 1 states that, under convexity and bounded energy, the average energy-budget violation over time is asymptotically","pith_inferences":["The energy model E = αFT treats energy as linear in FLOPs and local steps; memory access, communication, idle power, and hardware-specific efficiency are not modelled, so reported energy savings may be synthetic unless the proxy is validated on real hardware.","Theorem 1's proof is deferred to standard saddle-point optimisation analysis without stating the required assumptions beyond convexity and Lipschitz continuity; applying the O(ρ) bound to non-convex deep networks is a leap, and the empirical 4.8% violation rate is the more concrete evidence.","The same primal-dual skeleton could enforce other per-device constraints, such as latency or memory, by swapping the cost function; a direct extension would be to add a latency penalty and test whether the violation bound and accuracy-energy trade-off transfer.","A hardware-in-the-loop benchmark comparing predicted energy (αFT) against actual battery drain on a few device classes would clarify whether the accuracy-energy Pareto improvements persist physically."],"forward_implications":["Federated learning services can receive architectures tailored to their current battery, compute, and memory budgets without manual design or on-device search.","The primal-dual update gives a simple distributed mechanism for enforcing resource constraints: each device only needs its own energy estimate and a scalar dual variable.","Since architecture search runs centrally on the NAS service, the per-device cost of personalisation is limited to pruning and local fine-tuning, improving scalability.","A single coefficient λ controls the accuracy-energy trade-off, letting a fleet operator dial in the desired operating point.","Dynamic device states (budgets that change per round) are handled naturally, as pruning and dual updates re-run each round."],"fun_headline_variants":["Energy-aware NAS for federated edge","Federated NAS with strict energy budgets","Energy constraints drive federated NAS pruning","NAS adapts to edge energy limits","Personalized edge models via energy-aware NAS"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire energy-aware pipeline treats energy as a linear function of FLOPs and local training steps, ignoring memory access, communication, idle power, and hardware-specific efficiency; if this model is not faithful to real device energy, the energy claims describe a synthetic quantity rather than physical battery consumption.","fun_headline_variants_meta":{"raw":{"variants":["Energy-aware NAS for federated edge","Federated NAS with strict energy budgets","Energy constraints drive federated NAS pruning","NAS adapts to edge energy limits","Personalized edge models via energy-aware NAS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0003,"raw_usage":{"total_tokens":1533,"prompt_tokens":675,"completion_tokens":858,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":419,"completion_tokens_details":{"reasoning_tokens":795}},"tokens_in":419,"tokens_out":858,"duration_ms":8805,"temperature":1.0,"reasoning_tokens":795,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T04:41:52.655112+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run OrchNAS on two devices with identical FLOPs and local steps but different memory bandwidth or idle power, and compare actual battery drain with the model's predicted energy. If measured energy differs substantially while predicted energy is equal, the energy-aware scores and pruning decisions are optimizing a synthetic quantity, not physical energy.","supporting_citations":[],"review_version":1}