{"id":"f650d816-4a9d-47a5-ad17-a89f7a9b80db","arxiv_id":"2505.07041","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"On five physical edge devices, asynchronous federated learning reached 75% accuracy about ten times faster than synchronous learning, but high-end devices supplied far more updates and accumulated up to roughly five times more privacy loss, while slow devices lost more accuracy under local…","lead":"This paper measures how asynchronous federated learning shifts efficiency, fairness, and privacy on a testbed of five edge devices with different processing speeds. It reports that asynchronous training can converge about ten times faster but concentrates influence and privacy loss on high-end devices, a trade-off relevant to privacy-sensitive federated systems.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-client ε accounting not shown to track measured participation; Table 3's FedAvg row conflicts with documented dropouts, so the 5x privacy-disparity claim lacks support.","rationale":"The paper's efficiency and participation-skew results are credible empirical measurements: the 9-10x convergence speedup and the dominance of high-end devices in FedAsync are directly supported by Figures 4 and 5. The privacy-disparity claim, however, rests on per-client ε values from a custom Moments Accountant that is not released. The reader's weakest-assumption analysis identifies exactly this: the accountant must be driven by each device's measured update count. My reading agrees. The strongest evidence for the concern is internal: Table 3 reports identical FedAvg ε for all devices even though Section 4.2.1 documents low-end dropouts, and the FedAsync ε ratios do not clearly match the Fig. 5 participation percentages under the paper's own composition equations. These inconsistencies do not necessarily invalidate the qualitative trade-off, but they do mean the quantitative 'up to 5x higher privacy loss' claim is not currently supported. Because the issue is addressable by releasing logs/code and recomputing ε, the appropriate verdict remains CONDITIONAL—unchanged from the reader's verdict. I do not see a more load-bearing concern: the efficiency and fairness findings have direct experimental support, and the privacy accounting issue is the one that would change the headline quantitative claim if it cannot be resolved.","tokens_in":13010,"tokens_out":8741,"duration_ms":87318,"concrete_test":"Recompute Table 3 from the logged per-device update counts using Eqs. (7)-(8): for FedAvg, sum per-round moments over the actual number of completed rounds for each device, excluding dropout rounds; for FedAsync, sum over the actual number of updates sent per device, as reflected in the Fig. 5 participation percentages. If the recomputed FedAvg ε values are not identical across devices, or if the FedAsync ε ratios cannot be reproduced from the measured participation fractions, then the privacy-disparity claim does not follow from the measurements. Releasing the custom accountant code and per-client round/update logs would make this check possible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—that FedAsync produces up to 5x higher per-client privacy loss for high-end devices—depends entirely on the custom Moments Accountant of Section 3.2 being driven by each device's real update count. The paper's own data make this doubtful. (i) Section 4.2.1 reports that HW T1 and HW T2 dropped out several times during FedAvg training, yet Table 3 reports a single identical ε for 'All Devices' under FedAvg. If the accountant summed per-round moments only for rounds actually completed, devices with dropouts would have strictly fewer composed mechanisms and thus lower ε; equal ε implies the accountant was fed a nominal round count rather than measured participation. (ii) Under Eqs. (7)-(8), ε is a composition over a device's number of local updates. At σ=0.5, α=0.2, Fig. 5a gives participation ratios T5:T1 = 31.87:8.24 = 3.87, while Table 3 gives ε ratio 31.75:12.57 = 2.53. At α=0.6 the participation ratio is 9.41 but the ε ratio is only 5.17. These could be consistent with sublinear RDP composition, but the FedAvg contradiction shows the accountant is not demonstrably coupled to measured participation. Since the '5x privacy disparity' is the paper's headline privacy result, the quantitative privacy claims are unsupported unless the authors release per-client update-count logs and accountant code and show that Table 3 follows from them.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports an empirical comparison of synchronous FedAvg and staleness-aware FedAsync on a physical testbed of five edge devices (Raspberry Pi 3/4, NXP HummingBoard) for speech emotion recognition on CREMA-D. Local updates are perturbed with Gaussian noise under a DP-SGD-style mechanism, and per-client privacy loss is tracked with a custom Moments Accountant. The paper claims that FedAsync reaches 75% accuracy roughly 9-10x faster in wall-clock time than FedAvg, that high-end devices contribute 6-10x more updates and incur up to 5x higher per-client epsilon, and that low-end devices suffer greater accuracy degradation. The efficiency and participation measurements are plausible and largely internally consistent, but the privacy-loss accounting is not reconciled with the measured participation and dropout behavior, so the quantitative privacy-disparity claims are not supported as reported.","tokens_in":13227,"tokens_out":8219,"duration_ms":84082,"significance":"If the privacy accounting is corrected, the central message—that asynchronous FL improves wall-clock convergence while shifting both influence and privacy burden toward high-end devices—would be a useful empirical contribution for systems designers choosing aggregation and privacy mechanisms. The paper's strengths are the physical testbed spanning five hardware tiers, multi-seed averaging, direct wall-clock convergence measurements, and explicit per-device participation, staleness, and accuracy breakdowns. The fairness and efficiency findings are more solid than the privacy findings. However, the paper does not release code, per-client update-count logs, or the custom accountant implementation, and the privacy-loss numbers are the part of the paper that most needs independent verification.","major_comments":[{"comment":"The FedAvg row in Table 3 reports a single epsilon for 'All Devices' at each sigma, yet Section 4.2.1 and Table 2 document that HW T1 dropped out three times and HW T2 dropped out twice during FedAvg training. Under the per-client accountant of Eqs. (7)-(8), a client that misses rounds must have strictly fewer composed moments and hence strictly lower cumulative epsilon than a client that never drops out. Equal epsilon for all devices is therefore only consistent with an accountant fed a nominal fixed round count rather than measured per-client participation. Since the same accounting pipeline is used for FedAsync, the per-client epsilon values in Table 3 are not shown to follow from the update counts measured in Section 4.2.2. This directly undermines the headline claim of up to 5x higher privacy loss for high-end devices; the authors should release per-client update-count logs and the accountant code, and recompute Table 3 from the measured participation, or explicitly reconcile the FedAvg row with the documented dropouts.","section":"Section 4.2.1, Table 2; Section 4.2.3, Table 3; Section 3.2, Eqs. (7)-(8)"},{"comment":"The privacy accounting adds one moment mu_t per client round, but DP-SGD as written in Eqs. (4)-(6) applies clipping and Gaussian noise to every mini-batch. With E=1, B=128, and about 941 training samples per client, each round contains roughly eight noised mini-batch steps, and the Moments Accountant for DP-SGD must compose over those mini-batch steps rather than over whole rounds. If mu_t in Eq. (7) is intended to summarize an entire round, the paper must explain how the composition over mini-batches is folded into a single mechanism; as written, the reported epsilon values appear to undercount privacy loss by approximately the number of mini-batches per round. All absolute epsilon values in Table 3 are therefore questionable independently of the participation-mismatch issue.","section":"Section 3.2, Eqs. (4)-(8); Algorithm 1, lines 8-16"},{"comment":"The epsilon ratios in Table 3 do not track the participation ratios in Fig. 5 under the stated accountant. At alpha=0.2 and sigma=0.5, HW T5 has 31.87% participation versus 8.24% for HW T1 (ratio 3.87), while the reported epsilon ratio is 31.75/12.57 = 2.53; at alpha=0.6, the participation ratio is 42.34/4.50 = 9.41 but the epsilon ratio is 35.12/6.79 = 5.17. Sublinear RDP composition might explain the divergence, but the paper does not provide the per-update RDP parameters or the accountant trace needed to verify this. Because the headline privacy ratio is derived from these numbers, the authors should show the step-by-step accountant output for at least one alpha and one sigma value.","section":"Section 4.2.2, Fig. 5; Section 4.2.3, Table 3"}],"minor_comments":[{"comment":"The conclusion states that high-end clients experience 'up to 6x greater privacy loss', while the abstract and Section 4.2.4 state 'up to 5x'; Table 3 gives a maximum ratio of about 5.17, so the multiplier should be harmonized across the paper.","section":"Section 6 and Abstract"},{"comment":"The paper says FedAsync runs 'continuous updates' until convergence, but the total number of asynchronous updates and the per-client update counts are never stated; Table 3's epsilon values cannot be reproduced without these counts. A table or appendix listing update counts for each device and each alpha would resolve this.","section":"Section 4.1.4"},{"comment":"The inset in Figure 4 uses sparse tick labels such as 67, 467, 867, 1267 without a clear indication of the scale; the inset axes should be labeled explicitly and the tick format made readable.","section":"Figure 4"},{"comment":"Equation (7) does not precisely define D and D' for the local mechanism; the authors should state explicitly that they are neighboring local datasets differing in one sample, and clarify whether privacy is accounted at the sample level or the client level.","section":"Section 3.2, Eq. (7)"}],"recommendation":"major_revision","confidential_remarks":"The efficiency and fairness parts of the paper are promising and the testbed measurements appear solid. The privacy analysis is the load-bearing novel contribution, and as written it is internally inconsistent with the documented dropouts and lacks the accounting trace needed for verification. If the authors have per-client update logs and accountant code, this can likely be fixed in a revision; if not, the quantitative privacy claims should be scaled back to qualitative statements. It may also be worth asking for the Opacus/custom accountant code as a supplement, since all reported epsilon values depend on it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful empirical paper, not a breakthrough. The physical testbed (five hardware tiers, SER, real FedAsync/FedAvg) is a nice change from simulation, and the qualitative message—async FL accelerates convergence but skews participation and privacy burden toward fast hardware—is almost certainly right and worth saying.\n\nWhat's new: per-client privacy accounting under FedAsync using the Moments Accountant on real devices, with participation percentages, staleness, and per-device accuracy degradation. The convergence speedup (about 9-10x to 75% accuracy) and the participation skew (high-end devices contributing 30-42% vs 4-8% for low-end) are concrete measurements that support the fairness concern. Those parts are solid.\n\nThe soft spot is the privacy accounting. In Section 3.2, epsilon is composed over a device's number of local updates; with fixed sigma and q, per-client epsilon should roughly scale with update count. But Table 3's epsilon ratios don't match the participation percentages in Fig 5. At alpha=0.2, T5/T1 participation is 31.87/8.24 ≈ 3.9, while the epsilon ratio is 31.75/12.57 ≈ 2.5. At alpha=0.6, participation ratio is about 9.4 but epsilon ratio is 5.2. That gap is too large to excuse as accountant rounding. More telling, the FedAvg row reports a single epsilon for all devices even though Section 4.2.1 documents low-end dropouts; per-client accounting should lower their composed epsilon. That suggests the accountant wasn't actually driven by measured update counts. Without code or per-client logs, the headline '5x privacy disparity' is not supported by the data as presented.\n\nOther, smaller issues: the accuracy degradation baseline in Table 3 isn't defined, and no code or data are released, which is a miss for an empirical paper.\n\nWho this is for: people working on heterogeneous FL, fairness, or DP in FL will find the experimental setup useful and the trade-off story worth citing as motivation, but only after the authors either fix the accounting or release artifacts showing it matches participation. The paper deserves peer review, but I'd want the accounting discrepancy addressed.","headline":"Empirically interesting and credible on efficiency/fairness, but the privacy-disparity numbers don't cohere with the paper's own accounting, so treat the 5x claim with caution.","tokens_in":13824,"tokens_out":6266,"would_cite":false,"duration_ms":61078,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Asynchronous federated learning converges up to 10x faster, but the privacy burden falls unevenly on high-end devices, which incur up to 5x higher privacy loss.","keywords":["federated learning","asynchronous aggregation","device heterogeneity","local differential privacy","moments accountant","fairness","speech emotion recognition"],"falsifier":"Recomputing each Table 3 epsilon from the paper's measured participation percentages using Eqs. (7)-(8) with $\\sigma$ fixed would settle it: if a device's epsilon is not proportional to how often it actually updated, or if FedAvg's epsilon is not identical across devices, the claim that privacy loss tracks participation fails.","tokens_in":12733,"feed_emoji":"⚖️","tokens_out":7043,"duration_ms":62525,"temperature":0.7,"pith_summary":"This paper argues that the efficiency gain of asynchronous federated learning comes with a hidden, systematic fairness and privacy cost on heterogeneous hardware. Using a physical testbed of five devices running a speech-emotion-recognition task, it compares synchronous FedAvg with staleness-aware FedAsync under local differential privacy, tracking per-client cumulative privacy loss with the Moments Accountant. It finds that FedAsync reaches target accuracy about 9-10x faster in wall-clock time, but high-end clients contribute 6-10x more updates and accumulate up to roughly 5x higher privacy loss, while low-end clients lose more local accuracy under the same noise. The authors conclude that static uniform privacy mechanisms are inadequate and that aggregation, staleness handling, and privacy budgets should be tuned jointly based on client capacity and participation.","feed_headline":"10x faster async FL hides 5x privacy gap on fast devices","feed_subtitle":"On heterogeneous hardware, asynchronous aggregation boosts speed but makes high-end devices carry most of the privacy loss","key_machinery":"The load-bearing mechanism is the coupling of two standard components: staleness-aware asynchronous aggregation (FedAsync's decay factor $\\alpha_k = \\alpha/(1+\\tau_k)$, which down-weights old updates) and per-client Moments Accountant privacy accounting ($\\varepsilon$ computed from a fixed noise scale, sampling probability $q$, and the number of rounds the client actually contributes). The argument runs through this coupling: because faster devices send more updates with low staleness, they receive larger aggregation weights and accumulate proportionally larger $\\varepsilon$; slower devices send fewer, staler updates, so they get less influence and smaller $\\varepsilon$, yet suffer more noise-induced accuracy loss. The trade-off is summarized by varying $\\alpha$, which trades convergence speed against participation skew and privacy disparity.","core_discovery":"The central claim, on the paper's own terms, is that under realistic device heterogeneity asynchronous aggregation converts the straggler problem into an inequality problem: the same staleness-aware weighting that makes FedAsync fast also makes it unfair and privacy-uneven. Empirically, at alpha=0.6 high-end devices account for about 80% of updates while low-end devices fall below 5%, and the cumulative privacy loss at sigma=0.5 reaches roughly 35 for the fastest device versus below 10 for the slowest. Because each device's update count drives its Moments Accountant budget, the privacy burden tracks participation exactly, and uniform LDP noise does not equalize outcomes: low-end devices, whose few updates are stale and down-weighted, show up to about 16% local accuracy degradation while high-end devices degrade by roughly 6%.","pith_inferences":["The paper does not state this, but its epsilon equations imply that the privacy disparity is a participation fact rather than a hardware fact: any asynchronous mechanism that changes who sends updates changes who pays the privacy cost.","A direct extension of the authors' own suggestion would be to inverse-calibrate per-client noise, adding more noise to frequent high-end contributors and less to rare low-end ones, and test whether the epsilon spread narrows without pushing global accuracy below the FedAvg level.","Because the paper reports participation, staleness, and epsilon together, one could define a composite privacy-fairness index equal to epsilon divided by influence; under the reported numbers this index would be near-constant if the accountant tracks participation, which would make the disparity purely a weighting artifact.","Comparing with client-level differential privacy instead of LDP would isolate whether low-end accuracy loss comes from per-message noise or from stale-model training; the paper does not run this control."],"forward_implications":["If the paper is right, reported convergence speedups of asynchronous FL should be read together with the per-device participation skew; the 10x wall-clock gain is purchased by letting fast hardware dominate the update stream.","Uniform local differential privacy noise does not deliver uniform privacy or uniform utility; low-end clients bear disproportionate accuracy loss under the same noise scale.","The aggregation strength $\\alpha$ acts as a fairness-privacy dial: lowering $\\alpha$ narrows participation and privacy gaps at the cost of convergence speed.","Per-client privacy accounting that uses real update counts would show the same disparity in any asynchronous FL deployment where participation is hardware-bound.","Systems that need both speed and equity should weight updates by capacity or staleness while calibrating noise per client, rather than applying one-size-fits-all LDP."],"supporting_citations":[{"why":"Defines the synchronous FedAvg aggregation rule that forms the baseline for all comparisons.","marker":"[1]"},{"why":"Introduces staleness-aware asynchronous aggregation with the $\\alpha/(1+\\tau)$ decay that drives the measured participation skew.","marker":"[7]"},{"why":"Supplies DP-SGD and the Moments Accountant equations used to compute per-client privacy loss.","marker":"[9]"},{"why":"Defines local differential privacy, the noise model applied to every client update.","marker":"[8]"},{"why":"Provides the CREMA-D speech emotion dataset and its four-class partition used in the benchmark.","marker":"[24]"},{"why":"Supplies the FL framework used to orchestrate the physical testbed.","marker":"[22]"},{"why":"Supplies the DP library used to inject clipping and Gaussian noise during local training.","marker":"[23]"}],"fun_headline_variants":["Async FL speedup comes with unfair privacy cost","Faster FL, uneven privacy: high-end devices bear brunt","10x faster but 5x more privacy loss for fast devices","Asynchronous FL: speed gains, but fairness and privacy suffer","Study: async FL speeds up 10x, but privacy gap widens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline privacy-disparity numbers rest on the assumption that each device's reported epsilon was driven by that device's real update count, an assumption the paper's own FedAvg row (identical epsilon for all five devices despite documented low-end dropouts) leaves unverified.","fun_headline_variants_meta":{"raw":{"variants":["Async FL speedup comes with unfair privacy cost","Faster FL, uneven privacy: high-end devices bear brunt","10x faster but 5x more privacy loss for fast devices","Asynchronous FL: speed gains, but fairness and privacy suffer","Study: async FL speeds up 10x, but privacy gap widens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000211,"raw_usage":{"total_tokens":1420,"prompt_tokens":959,"completion_tokens":461,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":373}},"tokens_in":575,"tokens_out":461,"duration_ms":5366,"temperature":1.0,"reasoning_tokens":373,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:27:37.848205+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recomputing each Table 3 epsilon from the paper's measured participation percentages using Eqs. (7)-(8) with $\\sigma$ fixed would settle it: if a device's epsilon is not proportional to how often it actually updated, or if FedAvg's epsilon is not identical across devices, the claim that privacy loss tracks participation fails.","supporting_citations":[{"cited_title":"Local differential privacy for deep learning.IEEE Internet of Things Journal, 7(7):5827– 5842, 2019","cited_arxiv_id":null,"evidence_quote":"Defines local differential privacy, the noise model applied to every client update."}],"review_version":1}