{"id":"fd211e40-e27a-4f3c-92ad-f7a13df29332","arxiv_id":"2506.02709","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A cross-cloud study of 11 HPC proxy apps finds that clouds with fast networks can handle small to medium HPC workloads, but setup effort, cost, and performance vary widely.","lead":"The authors ran 11 high-performance computing benchmarks and proxy apps on Amazon, Microsoft, and Google clouds plus two on-premises clusters, measuring both raw performance and how much effort each environment took to set up and use. The result is one of the broadest cross-cloud HPC usability datasets to date, useful for scientists deciding whether to move workloads off traditional HPC centers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"AMG2023 process-topology inconsistency confounds the central Kubernetes-vs-VM evidence for cloud suitability.","rationale":"The reader identified hardware and network comparability as the weakest assumption and returned a CONDITIONAL verdict. My concern is more specific and more internal: the AMG2023 process-topology mismatch is an acknowledged experimental control flaw, not a hardware difference, and it affects the one weak-scaling application that the paper uses as evidence that cloud environments are well-suited. If the proposed re-run confirms a ~10% FOM inflation, the quantitative cloud-versus-Kubernetes comparisons in Figure 2 would weaken, but the paper's broader qualitative contributions—the dataset, containers, usability assessment, and cost observations—would remain valuable. The reader's CONDITIONAL verdict is therefore still appropriate; I do not think the concern is severe enough to reject the paper, but it should be stated explicitly as a condition on any claim that Kubernetes and VM environments show comparable performance. I agree only partially with the reader's weakest assumption because the most load-bearing issue is not cross-vendor hardware differences but an inconsistency within the paper's own experimental controls.","tokens_in":20602,"tokens_out":4000,"duration_ms":44582,"concrete_test":"Re-run AMG2023 at size 64 GPUs on Compute Engine and GKE (and, if budget permits, at sizes 32, 128, and 256) using an identical process topology such as -P 4 4 4, identical problem size, and five iterations per environment, then compare the FOM distributions. If the GKE versus Compute Engine gap shifts by roughly 10% or changes sign, the Kubernetes/VM comparability conclusion needs to be revised or explicitly limited to the topology that was actually used.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest support for the Section 4.1 claim that cloud environments are 'well-suited' for small-to-medium HPC apps explicitly names AMG2023 as a weak-scaling app with 'precedence to run in cloud.' Section 3.3 admits that VM and Kubernetes AMG2023 runs used different MPI process topologies: -P 4 4 4 for VM environments and -P 8 4 2 for Kubernetes environments, and that -P 8 4 2 produces roughly 10% higher FOM on GKE at size 64. Only the corrected GKE value is reported. This is not another hardware-comparability caveat; it is an internal control failure in exactly the application used to anchor the central suitability claim and the cost table. The qualitative usability findings may still hold, but the quantitative evidence that Kubernetes and VM cloud environments perform comparably is systematically confounded for AMG2023, weakening the paper's flagship demonstration of cloud readiness.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a cross-platform usability and performance evaluation of 11 HPC proxy applications and benchmarks across three public clouds (AWS, Azure, Google Cloud), two on-premises clusters, VM and Kubernetes environments, and CPU and GPU configurations. It reports qualitative usability scores, scaling results for applications such as AMG2023, Laghos, LAMMPS, Kripke, MiniFE, MT-GEMM, Quicksilver, OSU benchmarks, and STREAM, together with cost data and suggested practices. The central claim, stated in Section 4.1, is that cloud environments with OS-bypass or RDMA-based networks can be well-suited to small-to-medium HPC applications at tens to a few hundred CPU nodes, with AMG2023 and LAMMPS given as examples. The paper also claims to be the first study of this breadth, covering more than 10 applications across multiple environments at over 100 nodes and 100 GPUs, with a substantial data release of 25,541 runs.","tokens_in":20793,"tokens_out":5545,"duration_ms":52827,"significance":"If the conclusions hold, this would be a valuable reference for HPC practitioners evaluating cloud adoption: the breadth of applications and environments, the reproducible artifact release (Zenodo DOI), the containerized application set, and the qualitative usability framework are genuine contributions. The paper also provides practical details, such as an Azure InfiniBand daemonset and VM images, that are likely to save other teams considerable effort. However, the quantitative comparative performance claims are weakened by several internal control issues: inconsistent AMG2023 process topologies between VM and Kubernetes, OSU point-to-point benchmarks run concurrently on the same nodes for EKS and AKS, and mixed Azure ECC states that are not controlled for. These issues directly affect the evidence cited for the central suitability claim, so the manuscript needs substantial revision before the comparative results can be accepted as reliable.","major_comments":[{"comment":"In the AMG2023 size-64 GPU runs, VM environments used the process topology -P 4 4 4 while Kubernetes environments used -P 8 4 2, and the authors state that -P 8 4 2 gives about 10% higher FOM on GKE at size 64. Only the corrected GKE value is reported in Figure 2. Since AMG2023 is cited in Section 4.1 as evidence that cloud environments are well-suited and is the basis of the cost comparison in Table 4, the Kubernetes-versus-VM comparison for this application is confounded in favor of Kubernetes by roughly the reported 10%. Please rerun the affected VM size-64 runs with the same topology, report results for both topologies, or explicitly restrict the comparative claims to configurations with matched process topologies.","section":"§3.3 (AMG2023) and §4.1"},{"comment":"The point-to-point latency and bandwidth tests for EKS and AKS were run simultaneously on the same nodes, which the authors state likely negatively impacted performance. This makes the EKS and AKS OSU results non-comparable with the other environments, and these data are used to support networking-related conclusions, such as identifying which fabrics provide the lowest latency and highest bandwidth. Please rerun the benchmarks in isolation, or remove the simultaneously-run EKS/AKS results from the comparative analysis and clearly state that those environments are not compared on point-to-point metrics.","section":"§3.3 (OSU Benchmarks)"},{"comment":"Azure GPU environments had a mixture of ECC settings, with ECC off for 12.5-25% of instances and on for 50-100%, while all other cloud GPU environments had ECC on. The paper notes that ECC can reduce performance by up to 15%, so any Azure GPU performance advantage in applications such as MiniFE, MT-GEMM, or the GPU OSU measurements could partly reflect ECC state rather than environmental quality. Report the ECC status for every Azure GPU run and either control for it statistically or explicitly exclude the affected comparisons from performance conclusions.","section":"§3.3 (Mixbench and ECC)"},{"comment":"Cross-platform comparisons are made across nodes with substantially different resource counts and interconnects: Google CPU nodes have 56 cores versus 96 on AWS and Azure, the on-premises A cluster has 112 cores per node, and the on-premises B cluster uses POWER9 with a different fabric and only 4 GPUs per node versus 8 on cloud GPU nodes. The paper acknowledges these differences but still draws comparative conclusions, such as the claim that on-premises Laghos FOM is one order of magnitude larger than cloud. For strong-scaling runs, equal node counts do not imply equal per-core or per-GPU resources; please normalize by core or GPU count, or restrict comparative claims to configurations that are equivalent in resource count.","section":"§2.2, §2.4, and §3.3"}],"minor_comments":[{"comment":"The abstract says the study covers 'six environments,' while Table 3 and Section 4 discuss 11 cloud environments plus two on-premises clusters; the environment count should be made consistent throughout.","section":"Abstract and §1"},{"comment":"The contributions list mentions '12 cloud environments' while the Discussion says '11 cloud environments'; the difference is explained by the missing AWS ParallelCluster GPU assessment, but this should be stated in the same place to avoid an apparent inconsistency.","section":"Contributions and §4"},{"comment":"In the 'Accounts and Resources' paragraph, the sentence 'We ascribe a medium score to for acquiring GPUs on AWS' contains a typo ('to for') and should be corrected.","section":"§3.1"},{"comment":"The total costs in Table 4 are sums over iterations and cluster sizes, but the number of iterations and any excluded failed runs are not given; adding this information would make the cost totals reproducible from the stated methodology.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper's breadth and artifact release are strong assets, and the qualitative usability findings are likely to be useful. The blocking issue is that the quantitative evidence for the central cloud-suitability claim is confounded by the AMG2023 topology mismatch and by the OSU and ECC control problems. These are fixable within the manuscript's scope through reruns or explicit scope restrictions, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is the broadest honest cross-cloud HPC usability and performance dataset I've seen, and the practical value is real. But one of the load-bearing quantitative comparisons—AMG2023 on Kubernetes vs VMs—is internally confounded by a process-topology change, and the authors don't do enough to flag that. Still worth refereeing.\n\nThe genuinely new thing is the scale and the release of the data: 11 apps, 11 cloud environments plus two on-prem clusters, up to 256 nodes/GPUs, with 25,541 runs deposited. That's a real contribution. The usability assessment is subjective but clearly framed, and the account of the setup pain (custom Terraform, InfiniBand daemonset for AKS, quota fights) is useful practical guidance that most papers don't provide. The authors also deserve credit for openly listing several limitations: the Google nodes have 56 cores vs 96 elsewhere, the V100 is three generations old, and they admit they didn't consistently measure comparable Kubernetes-vs-VM performance.\n\nThe soft spot the stress-test calls out is real. In Section 3.3 they state that VM runs of AMG2023 used -P 4 4 4 while Kubernetes runs used -P 8 4 2, and that -P 8 4 2 yields ~10% higher FOM on GKE at size 64. Only the corrected GKE number is reported. That means the size-64 GPU comparison between Kubernetes and VMs for the one weak-scaling app they use to anchor the 'cloud is well-suited' claim is systematically biased in Kubernetes' favor. Clearly an internal control failure, even if transparently disclosed. Two other things: the OSU point-to-point tests on EKS and AKS were run simultaneously on the same nodes, which likely depressed their results, and the Laghos claim that it would be 'infeasible to run on any cloud' goes beyond what the data show—they only pushed it to 64 nodes and even on-premises it segfaulted at 128/256. The Azure ECC inconsistency also muddies GPU comparisons.\n\nNet: I'd take the qualitative findings and the dataset at face value, but I'd be careful with the cross-environment performance rankings, especially AMG2023 and OSU. This paper deserves a serious referee; the right ask is for the authors to either redo or clearly invalidate the AMG2023 Kubernetes-vs-VM comparison and soften the Laghos extrapolation. I'd bring it to a reading group and would cite the dataset if I were working on cloud HPC placement.","headline":"Broad, honest dataset on HPC in the cloud, but the AMG2023 topology inconsistency weakens the flagship Kubernetes-vs-VM comparison.","tokens_in":21313,"tokens_out":2696,"would_cite":true,"duration_ms":26655,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Cloud instances with OS-bypass or RDMA networks can run small- and medium-scale tightly coupled HPC jobs as well as on-premises clusters, a cross-cloud study of 11 applications argues.","keywords":["HPC in the cloud","Kubernetes for HPC","RDMA networking","proxy applications","usability evaluation","scalability benchmarks","cloud computing","high performance computing"],"falsifier":"Re-run the CPU suite with the same core count per node on every cloud and the on-premises cluster, using the same RDMA-class fabric, and check whether Laghos still fails to scale beyond 64 nodes while AMG2023 and LAMMPS remain competitive; if the Laghos gap disappears, the paper's blanket suitability claim overreaches, and if it persists, the claim is confirmed for all but the most tightly coupled applications.","tokens_in":20431,"feed_emoji":"☁️","tokens_out":8383,"duration_ms":78966,"temperature":0.7,"pith_summary":"This paper reports a large cross-cloud experiment asking whether commercial cloud environments can reasonably host traditional high-performance computing workloads. The authors set up the same 11 HPC proxy applications and benchmarks on three major clouds, in both virtual-machine and Kubernetes form, and compared them with on-premises clusters at up to 256 nodes and 256 GPUs. Their central conclusion is that clouds with OS-bypass or RDMA-style networks can be well-suited to small- and medium-scale tightly coupled HPC jobs, in the range of tens to a few hundred CPU nodes. The paper also makes a methodological point: portability across ephemeral, changing environments should be treated as a dimension of performance, not a separate concern. A sympathetic reader would care because the result offers concrete guidance for scientific teams deciding whether cloud can absorb HPC work without a dedicated center.","feed_headline":"Cloud can handle small and medium HPC jobs, study finds","feed_subtitle":"Testing 11 HPC apps on three clouds at up to 256 nodes, high-speed network fabrics close the gap with on-prem clusters.","key_machinery":"The load-bearing mechanism is the network fabric: OS-bypass and RDMA support that lets MPI traffic bypass the operating system and virtualization overhead that otherwise penalizes cloud HPC. The study's measurement instrument is a suite of 11 proxy applications and synthetic benchmarks, including AMG2023, Laghos, LAMMPS, Kripke, MiniFE, MT-GEMM, Mixbench, OSU benchmarks, Stream, and Quicksilver, run with identical containers across VM and Kubernetes environments. The same containers in both setups, with Flux-based orchestration on Kubernetes and Singularity on VMs, are what allow the paper to attribute performance differences to the environment rather than to application build.","core_discovery":"The paper's central claim is that the cloud is no longer categorically unsuitable for HPC: environments with OS-bypass or RDMA-based network fabrics can run small to medium HPC applications well, at scales of tens to a few hundred CPU nodes. It argues this on the basis of measured application figures of merit across VM and Kubernetes environments, using the same containers in both to isolate environment effects. The claim is qualified: applications that strong-scale, like LAMMPS, and weak-scale, like AMG2023, are the strongest cloud candidates, while the tightly coupled solver Laghos ran an order of magnitude slower on cloud and became infeasible beyond 64 nodes. The authors present the study as the first of its breadth and as a foundation for best practices rather than a definitive performance ranking.","pith_inferences":["If the OS-bypass network is indeed the main equalizer, then the actionable generalization for teams is a network-first checklist: choose an RDMA-capable instance, verify placement, and only then compare performance; otherwise the ranking reflects fabric choice rather than cloud usability.","The reported hookup-time data suggest a testable rule for mixed on-premises and cloud operation: include measured hookup time in the cost model, since it can dominate short runs and varies by cloud and scale.","A natural follow-up experiment, already implied by the paper's future-work list, would repeat the suite on current-generation GPUs and on the newly released VM images; this would separate hardware-generation effects from environment usability effects.","The observation that instance types can hide different underlying hardware indicates a reproducibility risk: studies that report results by instance label may be conflating node-to-node hardware variation with environment differences."],"forward_implications":["A scientific team without its own center can run tightly coupled MPI jobs at tens-to-a-few-hundred-node scale on cloud, provided it chooses RDMA or OS-bypass instance types and uses containers for portability.","Kubernetes becomes a credible execution layer for HPC, not just for cloud-native services, when the underlying network bypasses the kernel; the remaining performance delta with VMs is identified as a key open question.","Portability should be weighed as a first-class cost: time spent optimizing one platform may be better spent making the workload run reasonably well across several, because cloud environments change and vanish quickly.","Applications that weak-scale, such as AMG2023, can be economical on GPU cloud: despite more expensive instance types, shorter runtimes made the representative GPU runs cheaper than CPU runs in the reported cost table.","Older GPU generations such as the V100 remain a practical route to moderate-scale GPU capacity, with the caveat that software and driver support may be deprecated."],"supporting_citations":[{"why":"Documents the Elastic Fabric Adapter, the OS-bypass network used for CPU and GPU runs on one cloud; load-bearing for the claim that OS-bypass fabrics make cloud HPC viable.","marker":"[4]"},{"why":"Argues that RDMA and OS-bypass avoid Kubernetes network overhead; the paper positions its Kubernetes results against this claim.","marker":"[17]"},{"why":"Supplies the qualitative usability-effort scoring approach (low, medium, high) reused for the environment assessments.","marker":"[41]"},{"why":"Singularity container runtime used to run the same containers in VM environments as in Kubernetes; load-bearing for the comparison design.","marker":"[53]"},{"why":"Prior large-scale cloud evaluation whose node-provisioning failures this study reproduces; supplies the baseline that cloud scale trials can fail unexpectedly.","marker":"[55]"},{"why":"The converged-computing work that this study builds on to deploy MPI-based HPC in Kubernetes; the basis for all Kubernetes environments.","marker":"[70]"},{"why":"Explains the AllReduce latency spike observed in the environments that use a particular OS-bypass fabric; supplies the vendor fix.","marker":"[82]"},{"why":"Describes the Flux Operator used to orchestrate Flux across Kubernetes nodes in all cloud Kubernetes setups.","marker":"[86]"}],"fun_headline_variants":["RDMA networks make cloud viable for many HPC workloads","Study: Cloud fits small to medium HPC, but not all apps","Cloud HPC closes gap for small and medium jobs with right fabrics","HPC on cloud: 11 apps, 3 clouds, key role of network fabric"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparisons depend on the assumption that the hardware and network configurations chosen across environments are similar enough that performance differences can be blamed on the cloud environment rather than on different processors, core counts, or fabrics.","fun_headline_variants_meta":{"raw":{"variants":["RDMA networks make cloud viable for many HPC workloads","Study: Cloud fits small to medium HPC, but not all apps","Cloud HPC closes gap for small and medium jobs with right fabrics","HPC on cloud: 11 apps, 3 clouds, key role of network fabric"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000803,"raw_usage":{"total_tokens":3487,"prompt_tokens":864,"completion_tokens":2623,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":480,"completion_tokens_details":{"reasoning_tokens":2544}},"tokens_in":480,"tokens_out":2623,"duration_ms":16264,"temperature":1.0,"reasoning_tokens":2544,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:17:10.704381+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the CPU suite with the same core count per node on every cloud and the on-premises cluster, using the same RDMA-class fabric, and check whether Laghos still fails to scale beyond 64 nodes while AMG2023 and LAMMPS remain competitive; if the Laghos gap disappears, the paper's blanket suitability claim overreaches, and if it persists, the claim is confirmed for all but the most tightly coupled applications.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the qualitative usability-effort scoring approach (low, medium, high) reused for the environment assessments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Singularity container runtime used to run the same containers in VM environments as in Kubernetes; load-bearing for the comparison design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The converged-computing work that this study builds on to deploy MPI-based HPC in Kubernetes; the basis for all Kubernetes environments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Explains the AllReduce latency spike observed in the environments that use a particular OS-bypass fabric; supplies the vendor fix."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes the Flux Operator used to orchestrate Flux across Kubernetes nodes in all cloud Kubernetes setups."}],"review_version":1}