Enabling Seamless Transitions from Experimental to Production HPC for Interactive Workflows

Brian D. Etz; Christopher Zimmer; Daniel Pelfrey; David M. Rogers; Jack Lange; Jordan Webb; Kellen Leland; Ketan Maheshwari; Mallikarjun Shankar; Michael J. Brim

REVIEW 3 major objections 4 minor 1 cited by

Enabling Seamless Transitions from Experimental to Production HPC for Interactive Workflows

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A staged pathway moves interactive HPC from testbed to production, pairing streaming, token APIs, and QoS reservations with matched policy changes.

desk verdict Useful operational write-up of OLCF's staged testbed-to-production path for interactive workflows, but the conclusion claims demonstrated success while the paper supplies no production measurements and leans on a self-cited report. read the letter →

arxiv 2506.01744 v1 pith:UMNWAWXE submitted 2025-06-02 cs.DC

Brian D. Etz , David M. Rogers , Michael J. Brim , Ketan Maheshwari , Kellen Leland , Tyler J. Skluzacek , Jack Lange , Daniel Pelfrey

show 8 more authors

Jordan Webb Patrick Widener Ryan Adamson Christopher Zimmer Veronica G. Melesse Vergara Mallikarjun Shankar Sarp Oral Rafael Ferreira da Silva

This is my paper · ORCID

classification cs.DC

keywords interactiveHPCworkflowstime-sensitivecomputingdatastreamingmemory-to-memorytransfersecureservicemeshQoSschedulingtestbed-to-productiontransitioncross-facilityintegration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the gap between interactive HPC testbeds and production supercomputers can be closed by pairing technological innovations with deliberate policy changes. It reports a transition pathway at a US leadership computing facility that moves time-sensitive scientific workflows through progressively stricter environments, from an experimental testbed to a production exascale system. The authors claim that three components, a memory-to-memory streaming framework, a token-based API service mesh, and a SLURM quality-of-service reservation scheme, let interactive workflows keep their functionality while gaining production-grade security and reliability. A sympathetic reader would care because modern experiments generate data faster than batch-oriented HPC can respond, and this pathway offers a template for making near-real-time experimental steering routine rather than exceptional.

What carries the argument

The load-bearing mechanisms are three. First, a memory-to-memory data streaming framework that separates control and data planes, routing high-throughput transfers through dedicated data streaming nodes that act as application gateways or routers. Second, the Secure Scientific Service Mesh (S3M), a service-mesh API gateway that issues time-limited tokens and enforces fine-grained authorization, rate limiting, and auditing for external services. Third, an adaptive SLURM scheduling scheme that uses quality-of-service tiers and 'QoS reservations' to give time-critical jobs guaranteed, immediate starts by preempting lower-priority work without holding nodes idle. The argument rides on these three interoperating with policy changes in security, operations, and resource allocation at each transition stage.

What would settle it

Run the LCLStream-style streaming pipeline on the production exascale system and on the testbed with the same dataset and the same QoS reservation, and compare end-to-end latency and throughput; if production latency exceeds the experiment's timing window or throughput falls below the beamline data rate, the central transition claim fails.

Watch

Extended reading notes

Core claim

The central discovery claim is that a structured, staged transition path can successfully bridge development and production for interactive workflows without re-engineering them at each step. The paper presents the ACE-to-Frontier progression, with ACE as the flexible testbed, Odo and Seeker as production-like staging systems in different security enclaves, and Frontier as the final leadership-class destination. Each stage demands matching policy adaptations, such as token-based rather than SSH-only authentication, service-level agreements for uptime, and scheduling criteria that value time-sensitivity rather than raw utilization. The paper asserts that this combination preserves workflow functionality, enhances security, and is validated by user collaborations including a streaming pipeline for a free-electron laser experiment.

Load-bearing premise

The components were validated on a small, flexible testbed, and the central claim assumes they will deliver the same latency, throughput, and security when deployed on production systems at leadership scale, which the paper does not measure.

Editorial extensions

If this is right

The LCLStream pattern, compute jobs launched during beamtime with high-bandwidth data paths, can be reproduced without the fragile experimental network configuration that earlier attempts required, because the streaming framework replaces bespoke firewall setups.
External experimental facilities can gain programmatic, token-based access to HPC resources, so machine-to-machine workflow initiation becomes a supported pattern rather than a security exception.
Time-sensitive jobs can be guaranteed start times via QoS reservations, letting facilities support experiment steering while keeping batch throughput high.
The staged rubric gives workflow developers a checklist of authentication, API, and scheduling decisions for each environment, reducing rework when moving from testbed to production.
Slotting policy adaptations alongside technology, such as service-level agreements and incident-response protocols for interactive services, makes the transition reproducible across facilities.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The transition rubric is likely portable beyond the specific systems named in the paper: any facility with a flexible testbed and a production system could map its own environments onto the ACE-to-Frontier stages and adopt the same policy templates.
A testable extension would be to benchmark the same interactive workflow on the testbed, the staging systems, and the production system to quantify the latency, throughput, and security overhead each stage imposes, which the paper does not supply.
If the QoS-reservation scheme generalizes, it could reshape HPC allocation policy by giving 'time value' a formal role alongside core-hours in committee decisions, a change the paper gestures toward but does not fully specify.
The streaming framework's control/data plane separation may also apply to high-throughput edge-to-HPC pipelines outside experimental facilities, such as continuous data feeds from instruments or sensor networks, though the paper only discusses science-facility use.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is a genuinely useful operational paper, and the reader's take is close to mine. The new thing here is the staged transition pathway — ACE → Odo → Seeker → Frontier — and the policy-technology matrix in Table 1. The individual services (memory-to-memory streaming, S3M, QoS reservations) are described in prior ORNL reports and related literature, but packaging them into a concrete progression with explicit policy checkpoints is a real contribution. Facilities that are trying to figure out how to let interactive workflows out of the sandbox will find this roadmap practical and sensible.

The paper does several things well. The LCLStream case study is honest and instructive, particularly the admission in Section 2.2 that the experimental network configuration achieved good throughput but 'proved too complex for production environments.' That kind of candor is rare and gives the reader confidence in the authors' judgment. The discussion of policy friction — security models built around human SSH sessions, maintenance windows, allocation committees that don't value time-sensitivity — is well-observed and worth citing on its own.

The soft spot is exactly where the stress-test note lands: the central claim in Section 6 overstates what is demonstrated. The paper says the pathway 'can successfully bridge this gap,' but no production-scale measurements appear anywhere. Section 4.3's validation of the QoS scheme cites the authors' own technical report [11], and the streaming framework's sustained performance at Odo/Seeker/Frontier scale is simply not reported. Because the production case for streaming rests on the newer memory-to-memory framework — the LCLStream network approach was abandoned as too complex — this is a real external-validity gap, not a trivial omission. The paper never claims specific numbers, so it's not internally inconsistent, but the conclusion's tone implies a level of proof the body does not provide.

I don't think this is fatal. The paper could be revised to say 'we have designed and are validating a pathway' rather than 'we have demonstrated success,' or it could add even modest measurements from the production-like systems. As it stands, the paper is a well-structured experience report with a useful rubric, not a quantitative evaluation.

This deserves peer review — it's exactly the kind of operational knowledge that systems conferences and facility operators need — but it should be sent back for either added evidence or a reframed conclusion. I'd bring it to a reading group for the discussion of policy barriers alone, and I'd cite it as a reference point for transition pathways, though I'd be careful to describe it as a proposal rather than a validated result.

Referee Report

3 major / 4 minor

Summary. This paper presents a qualitative framework developed at ORNL/OLCF for transitioning interactive HPC workflows from experimental testbeds (ACE) to production systems (Odo, Seeker, Frontier). It proposes a staged pathway that combines three technological components—memory-to-memory data streaming over dedicated data streaming nodes, the S3M token/API gateway, and SLURM QoS-based preemptive scheduling—with policy adaptations covering security, operations, and resource allocation. The LCLStream project serves as the main case study, and Section 6 concludes that the pathway has been 'demonstrated' to successfully bridge the testbed-to-production gap. The manuscript contains no production-scale measurements, no comparative evaluation, and no reproducible data; the only concrete validation cited is a self-authored technical report [11].

Significance. The problem addressed is timely and important: interactive, time-sensitive scientific workflows do not fit traditional batch HPC operations, and a structured transition path is genuinely useful to the community. The paper's strengths are its clear three-pillar architecture (streaming, secure API access, QoS scheduling), the explicit policy considerations in Table 1, and a plausible staged progression across security enclaves. If the empirical claims were supported, this would be a valuable practice report. As it stands, however, the paper's evidence does not justify the word 'demonstrated'; the central claim either needs production measurements or a careful reframing as an experience/position paper.

major comments (3)

[Section 6] The concluding claim that 'we have demonstrated that a carefully structured pathway, combining technological innovations with policy adaptations, can successfully bridge this gap' is not supported by evidence in this manuscript. No production-scale measurements, latency or throughput figures, security evaluations, or reliability data are reported for Odo, Seeker, or Frontier. The evidence is limited to the ACE testbed and a validation cited to the authors' own technical report [11]. This is a load-bearing overstatement because the paper's central contribution is precisely this claim. The claim should either be backed by production measurements or explicitly softened to say that the pathway is proposed or demonstrated only at testbed level.
[Section 4.3] The QoS-based preemptive scheduling system is one of the three technological pillars, but its only validation is the sentence 'Our validation test of the SLURM workflow manager scheme demonstrated that this approach effectively balances interactive and batch workloads across diverse user communities [11].' The cited technical report is self-authored, is not summarized here, and does not allow the reader to audit preemption latency, throughput impact on preempted jobs, or reliability under production load. This gap is load-bearing because Section 6 explicitly lists adaptive resource scheduling among the requirements that the paper claims to have demonstrated.
[Section 2.2 and Section 4.1] Section 2.2 concedes that the LCLStream experimental network configuration 'proved too complex for production environments,' yet the production case for streaming rests on the memory-to-memory framework of Section 4.1 running on dedicated data streaming nodes. The paper reports no measurements showing that RabbitMQ/Redis-based streaming over DSNs achieves comparable throughput, acceptable latency, or the required security assurances at OLCF scale. Without such evidence, the external validity of the transition pathway is unestablished. This is an external-validity gap rather than an internal inconsistency, but it directly undermines the strong conclusion in Section 6.

minor comments (4)

[Abstract] The abstract text contains a source-formatting error: 'Theevolvinglandscapeofscientificcomputingrequires' appears without spaces; please fix the LaTeX source.
[Throughout] The word 'demonstrated' is used with varying strength (e.g., Section 2.2, Section 4.3, Section 6); please reserve it for claims with direct evidence and use 'implemented,' 'proposed,' or 'observed' elsewhere.
[Section 4.2] The contrast between S3M and 'traditional API gateways' would be clearer with a short feature comparison or a concrete example; the current statement 'not possible in traditional API gateways' is asserted without supporting examples.
[References] Reference [11] is used as the sole validation of the QoS scheduling claim; please mark it explicitly as a self-citation and provide the relevant measurements or a public dataset within this paper or in an appendix.

Circularity Check

1 steps flagged · score 4.0 of 10

No derivational circularity; one load-bearing self-citation supports the QoS scheduler validation behind the paper's 'demonstrated' conclusion.

self citation load bearing [Section 4.3, Time-Sensitive Workload Prioritization System; echoed in Section 6 conclusion]
"Our validation test of the SLURM workflow manager scheme demonstrated that this approach effectively balances interactive and batch workloads across diverse user communities [11]."

The paper's only validation evidence for the QoS reservation scheduler is reference [11], an Oak Ridge National Laboratory technical report authored by the same OLCF group (Sarp Oral et al., ACE FY24 efforts). No measurements, benchmarks, or independent evaluation appear in this paper. The Section 6 conclusion ('we have demonstrated that a carefully structured pathway ... can successfully bridge this gap') therefore leans for one of its three core pillars on a self-authored report rather than on evidence presented here. The demonstrated-ness of the scheduler component reduces to a self-citation that is not independently verified in the manuscript.

full rationale

This is an experience/architecture paper, not a derivation: there are no equations, fitted parameters, or predictive claims whose outputs could equal their inputs by construction. Patterns such as self-definitional reasoning, fitted-input-called-prediction, ansatz-smuggling, and renaming-known-results are absent. The transition pathway in Section 5 is a descriptive rubric, and the streaming and S3M components are presented as designs with references to ORNL reports [5, 11]; those self-citations document the facility's own prior work and are not, by themselves, circular. The one load-bearing self-citation is Section 4.3's statement that a 'validation test' of the SLURM QoS scheme 'demonstrated' its effectiveness, supported only by [11]. Because the conclusion asserts that the overall pathway has been 'demonstrated' to bridge the testbed-to-production gap, this self-referential validation of a central scheduler component raises the circularity score. It does not, however, make the entire framework equivalent to its inputs: the architecture, policy discussion, and staged transition rubric contain independent descriptive content, so the score is moderate rather than high.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper's central claim rests on the availability and scalability of OLCF infrastructure (ACE, Odo, Seeker, Frontier) and on the assumption that the described software services meet production requirements. No free parameters or invented entities appear, because the paper is an operations and systems report rather than a model with fitted values.

assumptions (3)

domain assumption Interactive HPC workloads require near real-time resource access and policy changes.
Section 1, paragraph 2: time-sensitive applications require preemptive scheduling and advance reservations; this motivates the whole paper but is taken as given.
domain assumption Testbed (ACE) behavior transfers to production at scale.
Section 5: each step must be tailored, but the framework assumes the technology stack (streaming, S3M, QoS) works in production; no production measurements are provided.
domain assumption Token-based authentication, science DMZ networks, and a service mesh are sufficient for production security without performance loss.
Section 3.1 and Section 4.2 assert these mechanisms preserve security and performance, but no benchmarks are supplied.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Enabling Seamless Transitions from Experimental to Production HPC for Interactive Workflows." pith.science (2026). https://pith.science/paper/UMNWAWXE

@misc{pith2026250601744,
  author       = {Pith},
  title        = {Pith review of: Enabling Seamless Transitions from Experimental to Production HPC for Interactive Workflows},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UMNWAWXE}},
  note         = {Machine review of arXiv:2506.01744}
}

read the original abstract

The evolving landscape of scientific computing requires seamless transitions from experimental to production HPC environments for interactive workflows. This paper presents a structured transition pathway developed at OLCF that bridges the gap between development testbeds and production systems. We address both technological and policy challenges, introducing frameworks for data streaming architectures, secure service interfaces, and adaptive resource scheduling for time-sensitive workloads and improved HPC interactivity. Our approach transforms traditional batch-oriented HPC into a more dynamic ecosystem capable of supporting modern scientific workflows that require near real-time data analysis, experimental steering, and cross-facility integration.

Figures

Figures reproduced from arXiv: 2506.01744 by the authors.

**Figure 1.** OLCF’s integrated approach to interactive computing. External users connect via secure authentication methods, accessing a suite of specialized services for status monitoring, computation, workflow management, and data streaming that collectively enable seamless interactivity across the computing environments. 2 End-to-End Interactive Workflow Modern scientific computing demands both specialized infrastructure and e… view at source ↗

**Figure 2.** OLCF transition path illustrating progression from the developmental testbed to production HPC to leadership computing, while spanning security enclaves. provides a structured approach for researchers and system administrators to identify and address the specific requirements at each stage of the transition pathway, ensuring that interactive scientific workflows can successfully bridge the gap between experimental f… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

A Study on Messaging Trade-offs in Data Streaming for Scientific Workflows
cs.DC 2025-09 conditional novelty 4.0 of 10

Batching publisher confirms, batching acknowledgements, raising prefetch, and using a few parallel queues recover most throughput lost to reliable-messaging settings in RabbitMQ for Deleria and LCLS-style streaming.

Reference graph

Works this paper leans on

15 extracted references · 3 canonical work pages · cited by 1 Pith paper

[11]

Sarp Oral et al.: Olcf’s advanced computing ecosystem (ace): Fy24 efforts for the doe integrated research infrastructure (iri) program. Tech. rep., Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States) (11 2024). https://doi.org/10.2172/2477506

doi:10.2172/2477506 2024
[1]

Bechtel Amara et al.: Accelerating discoveries at DIII-D with the Integrated Research Infrastructure

T. Bechtel Amara et al.: Accelerating discoveries at DIII-D with the Integrated Research Infrastructure. Frontiers in Physics12(2025). 12 B. Etz et al. https://doi.org/10.3389/fphy.2024.1524041

work page arXiv 2025
[2]

In: 2021 IEEE International Conference on Big Data (Big Data)

Katerina B Antypas et al.: Enabling discovery data science through cross-facility workflows. In: 2021 IEEE International Conference on Big Data (Big Data). pp. 3671–3680 (2021). https://doi.org/10.1109/BigData52589.2021.9671421

arXiv 2021
[3]

In: Proceedings of the Inter- national Conference for High Performance Computing, Networking, Storage and Analysis

Scott Atchley et al.: Frontier: Exploring exascale. In: Proceedings of the Inter- national Conference for High Performance Computing, Networking, Storage and Analysis. SC ’23 (2023). https://doi.org/10.1145/3581784.3607089

arXiv 2023
[4]

Future Generation Computer Systems161, 11–25 (2024)

Thomas Beck et al.: Integrating quantum computing resources into scien- tific hpc ecosystems. Future Generation Computer Systems161, 11–25 (2024). https://doi.org/https://doi.org/10.1016/j.future.2024.06.058

doi:10.1016/j.future.2024.06.058 2024
[5]

Brim et al.: A High-level Design for Bidirectional Data Streaming to High-Performance Computing Systems from External Science Facilities

Michael J. Brim et al.: A High-level Design for Bidirectional Data Streaming to High-Performance Computing Systems from External Science Facilities. Tech. rep., Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States) (03 2024). https://doi.org/10.2172/2338264

doi:10.2172/2338264 2024
[6]

Cruz et al.: Firecrest: a restful api to hpc systems

Felipe A. Cruz et al.: Firecrest: a restful api to hpc systems. In: 2020 IEEE/ACM International Workshop on Interoperability of Super- computing and Cloud Technologies (SuperCompCloud). pp. 21–26 (2020). https://doi.org/10.1109/SuperCompCloud51944.2020.00009

arXiv 2020
[7]

In: 2nd Annual Workshop on Large-Scale Experiment-in-the-Loop-Computing

Enders, B., et al.: Cross-facility science with the Superfacility Project at LBNL. In: 2nd Annual Workshop on Large-Scale Experiment-in-the-Loop-Computing. pp. 1– 7 (2020). https://doi.org/10.1109/XLOOP51963.2020.00006

arXiv 2020

Show all 15 references

[8]

In: High Performance Computing

Toshio Endo et al.: Challenges in Computing Resource Sharing Towards Next-Gen Interactive Accelerated HPC. In: High Performance Computing. ISC High Performance 2024 International Workshops. pp. 231–242 (2025). https://doi.org/10.1007/978-3-031-73716-9_16

2025 doi
[9]

arXiv preprint arXiv:2208.11745 (2022)

Jha, S., Pascuzzi, V.R., Turilli, M.: AI-coupled HPC workflows. arXiv preprint arXiv:2208.11745 (2022)

2022 arXiv
[10]

Miller et al.: Integrated Research Infrastructure Architecture Blueprint Activity (Final Report 2023) (7 2023)

William L. Miller et al.: Integrated Research Infrastructure Architecture Blueprint Activity (Final Report 2023) (7 2023). https://doi.org/10.2172/1984466

2023 doi
[12]

Albert Reuther et al.: Interactive and urgent hpc: Challenges and opportunities (2024), https://arxiv.org/abs/2401.14550

2024 arXiv
[13]

IEEE Computer57(8) (2024)

Rafael Ferreira da Silva et al.: Frontiers in Scientific Workflows: Pervasive Integration with HPC. IEEE Computer57(8) (2024). https://doi.org/10.1109/MC.2024.3401542

2024
[14]

EPJ Web of Conf

Thayer, Jana et al.: Massive Scale Data Analytics at LCLS-II. EPJ Web of Conf. 295(2024). https://doi.org/10.1051/epjconf/202429513002

2024
[15]

Cong Wang et al.: End-to-End Deep Learning Pipeline for Real-Time Bragg Peak Segmentation: From Training to Large-Scale Deployment. Front. High Perform. Comput.3(2025). https://doi.org/10.3389/fhpcp.2025.1536471

2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [11]

Sarp Oral et al.: Olcf’s advanced computing ecosystem (ace): Fy24 efforts for the doe integrated research infrastructure (iri) program. Tech. rep., Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States) (11 2024). https://doi.org/10.2172/2477506

doi:10.2172/2477506 2024

[2] [1]

Bechtel Amara et al.: Accelerating discoveries at DIII-D with the Integrated Research Infrastructure

T. Bechtel Amara et al.: Accelerating discoveries at DIII-D with the Integrated Research Infrastructure. Frontiers in Physics12(2025). 12 B. Etz et al. https://doi.org/10.3389/fphy.2024.1524041

work page arXiv 2025

[3] [2]

In: 2021 IEEE International Conference on Big Data (Big Data)

Katerina B Antypas et al.: Enabling discovery data science through cross-facility workflows. In: 2021 IEEE International Conference on Big Data (Big Data). pp. 3671–3680 (2021). https://doi.org/10.1109/BigData52589.2021.9671421

arXiv 2021

[4] [3]

In: Proceedings of the Inter- national Conference for High Performance Computing, Networking, Storage and Analysis

Scott Atchley et al.: Frontier: Exploring exascale. In: Proceedings of the Inter- national Conference for High Performance Computing, Networking, Storage and Analysis. SC ’23 (2023). https://doi.org/10.1145/3581784.3607089

arXiv 2023

[5] [4]

Future Generation Computer Systems161, 11–25 (2024)

Thomas Beck et al.: Integrating quantum computing resources into scien- tific hpc ecosystems. Future Generation Computer Systems161, 11–25 (2024). https://doi.org/https://doi.org/10.1016/j.future.2024.06.058

doi:10.1016/j.future.2024.06.058 2024

[6] [5]

Brim et al.: A High-level Design for Bidirectional Data Streaming to High-Performance Computing Systems from External Science Facilities

Michael J. Brim et al.: A High-level Design for Bidirectional Data Streaming to High-Performance Computing Systems from External Science Facilities. Tech. rep., Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States) (03 2024). https://doi.org/10.2172/2338264

doi:10.2172/2338264 2024

[7] [6]

Cruz et al.: Firecrest: a restful api to hpc systems

Felipe A. Cruz et al.: Firecrest: a restful api to hpc systems. In: 2020 IEEE/ACM International Workshop on Interoperability of Super- computing and Cloud Technologies (SuperCompCloud). pp. 21–26 (2020). https://doi.org/10.1109/SuperCompCloud51944.2020.00009

arXiv 2020

[8] [7]

In: 2nd Annual Workshop on Large-Scale Experiment-in-the-Loop-Computing

Enders, B., et al.: Cross-facility science with the Superfacility Project at LBNL. In: 2nd Annual Workshop on Large-Scale Experiment-in-the-Loop-Computing. pp. 1– 7 (2020). https://doi.org/10.1109/XLOOP51963.2020.00006

arXiv 2020