{"id":"54a381e0-3b13-48c0-936c-a5505d33fef0","arxiv_id":"2509.07199","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Batching publisher confirms, batching acknowledgements, raising prefetch, and using a few parallel queues recover most throughput lost to reliable-messaging settings in RabbitMQ for Deleria and LCLS-style streaming.","lead":"This paper measures how different RabbitMQ settings affect streaming speed for two scientific workflow patterns while keeping messages reliable. It gives HPC facility users practical guidance on choosing batch sizes, acknowledgements, and queue counts.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reliability tax likely conflates synchronous waiting with actual reliability; async confirms already give reliability at near-zero cost, so the 81% / 4.6x headline may be an artifact.","rationale":"The reader's weakest assumption targets workload transferability (synthetic payload sizes, 1 Gbps link), but the more load-bearing issue is internal: the paper's central 'reliability tax' may be an artifact of how the simulator uses publisher confirms. RabbitMQ confirms are asynchronous at the protocol level; a correct reliable client can publish without blocking and still process confirmations, so the measured 81% tax likely reflects a synchronous-wait implementation choice rather than an inherent cost of reliability. This does not invalidate all the parameter trade-offs, but it requires a new experiment and a reframing of the headline claim. The verdict should remain conditional until that experiment is run, because the paper's main practical guidance could otherwise mislead users into over-aggressive batching or accepting unnecessary throughput loss.","tokens_in":15246,"tokens_out":7510,"duration_ms":92997,"concrete_test":"Using the public StreamSim code, add a 'reliable-async' producer mode for the Dstream workload: enable publisher confirms, publish continuously (no per-message wait), register a confirmation listener that tracks delivery tags and retries nacks, and at the end of the run block until all outstanding confirmations arrive before reporting throughput. Run with 64 consumers and compare throughput to the 'none' and 'sync per-message' rows in Figure 3. If reliable-async throughput lands near the no-confirm/async curve (as expected), the claimed 81% reliability tax is an artifact of synchronous waiting; repeat with persistent messages to test the Figure 5 durability penalty.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The main quantitative claim—that reliable delivery imposes a large throughput tax—is not established because the paper compares 'no confirms' against a deliberately blocking use of publisher confirms. RabbitMQ confirms are delivered asynchronously; a producer that publishes continuously, tracks delivery tags via the confirmation listener, and only waits for outstanding confirms at shutdown/checkpoint obtains the same delivery guarantee as sync per-message confirms without putting the per-message RTT on the critical path. Figure 3 shows 'Async (per msg)' at essentially the no-confirm throughput, confirming that non-blocking confirm processing is cheap. The authors dismiss async as 'least reliable' (Section 4.2.1) because their simulator apparently never waits for confirmations; but that is an incomplete client pattern, not a property of the protocol. Consequently, the 'up to 81% reduction' for synchronous per-message confirms (Section 5.2) and the 4.6x batching recovery (Figure 4a) measure the cost of a synchronous wait, not the cost of reliability. The same conflation affects the durability experiment (Fig 5): guaranteed persistence ('sync confirm and persistence') includes synchronous waits; an async-confirm producer with persistent messages could give the same guarantee at much higher throughput. Thus the paper's practical guidance may lead users to accept unnecessary throughput loss or to batch aggressively when no such trade-off is needed.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies RabbitMQ messaging configuration trade-offs for two synthetic scientific streaming workloads, Deleria ('Dstream', 16 KiB messages) and LCLS ('Lstream', 1 MiB messages), on OLCF's DS2HPC testbed. Using a custom Go simulator and a three-node RabbitMQ 4.0.5 cluster, the authors vary one tunable at a time—publisher confirms, durability, consumer acknowledgements, prefetch count, and multi-queue parallelism—and report normalized aggregate consumer throughput. The central claim is that reliability mechanisms impose a large, configurable throughput tax (e.g., up to 81% reduction for synchronous per-message confirms on Dstream), which can be largely recovered via batching confirms/acks, higher prefetch, and a small number of parallel queues (up to 6.1x improvement).","tokens_in":15568,"tokens_out":3345,"duration_ms":38987,"significance":"The paper makes a pragmatic contribution by providing a reproducible simulation harness (publicly available code and configuration files) and a first-order map of RabbitMQ parameters for IRI-style scientific streaming. If the reliability-tax conclusion were sound, the guidance would help DS2HPC users avoid expensive real-instrument trial and error. However, the central interpretation is undermined by a conflation of synchronous waiting with reliability: RabbitMQ's asynchronous publisher confirms are dismissed as 'least reliable' even though a correctly implemented async-confirm client obtains the same end-to-end delivery guarantee without putting per-message RTT on the critical path. The measured 81% / 4.6x figures therefore quantify a blocking-wait cost, not a reliability cost. The Lstream results are additionally dominated by a 1 Gbps link bottleneck that the paper itself acknowledges. With corrected experiments and more cautious claims, the study could still be useful, but the headline message as written is not established.","major_comments":[{"comment":"The core 'reliability tax' claim conflates synchronous confirmation with reliability. RabbitMQ publisher confirms are asynchronous by design: a producer can publish continuously, track delivery tags, and wait only at shutdown/checkpoint, obtaining the same loss detection as synchronous per-message confirms. The paper labels async confirms 'least reliable' (§4.2.1) and observes in Figure 3 that they give essentially no-confirm throughput, which confirms that non-blocking confirm processing is cheap. The up-to-81% reduction for 'Sync (per msg)' and the 4.6x batching recovery in Figure 4 thus measure the cost of a per-message synchronous wait, not the cost of reliable delivery. The authors should either rerun the reliability comparison with a properly implemented async-confirm producer that actually drains confirmations, or reframe the claim as 'synchronous per-message waiting imposes a tax","section":"§4.2.1, §5.2"},{"comment":"The Lstream results are dominated by the 1 Gbps Ethernet link between Andes and the DSNs. The paper states in §5.2 that throughput saturates at 4 consumers 'due to network bandwidth limitations,' and §6 acknowledges that the DSNs' 100 Gbps interfaces were not yet usable. Under link saturation, the claimed differences—37% sync-confirm reduction, ~50% batch improvement, 5.3x multi-queue gain—are not clean measurements of RabbitMQ parameter effects; they partly reflect contention for a saturated link. The conclusions for Lstream should be explicitly scoped as 1 Gbps-limited, or the experiments should be re-run on a higher-bandwidth path before making workflow-specific recommendations.","section":"§5.2, §5.6, §6"},{"comment":"The synthetic workload model fixes Dstream to 16 KiB messages (2 KiB/event, 8 events) and Lstream to 1 MiB messages, despite Table 1 stating that Deleria's payload size and events-per-message are variable. This simplification is disclosed, but it is load-bearing for the quantitative transferability claim: the batching and prefetch improvements for Dstream (2.5x–4.6x) depend on many small messages being amortizable. No variance or confidence information is reported—each data point is the mean of three runs with no error bars or raw values—so the reader cannot judge whether the reported differences (e.g., the 4.6x figure at 64 consumers) are statistically meaningful. At minimum, the paper should show per-run spread or error bars and state the coefficient of variation.","section":"§5.1, Table 1"},{"comment":"The durability experiment repeats the same sync-wait conflation. The 'guaranteed persistence' condition ('With sync confirm and persistence') combines persistence with synchronous per-message confirms; the up-to-67% throughput reduction is again attributable to blocking waits, not to persistence itself. The paper even notes that with persistence alone, messages can still be lost if the broker crashes before flush, so the only meaningful durability guarantee tested is the sync-confirm variant. An async-confirm producer with persistent messages, waiting at the end of the run, would provide the same guarantee at substantially higher throughput. The durability section should be revised to separate disk-flush cost from confirmation-wait cost.","section":"§5.3"}],"minor_comments":[{"comment":"All figures report normalized means of three runs with no error bars or variance. Please add at least min/max error bars or a supplementary table with raw message-per-second values, especially for the headline 4.6x and 6.1x claims.","section":"Figures 3–8"},{"comment":"The term 'least reliable' for asynchronous confirms is inaccurate; async confirms provide the same broker-delivery guarantee if the client processes confirmations. Consider calling it 'non-blocking confirm processing' or 'confirms without synchronization.'","section":"§4.2.1"},{"comment":"The multi-queue experiment changes the ordering (producers started first, queues pre-filled) relative to all other experiments. This methodological difference should be stated more prominently, as it affects comparability with the other parameter sweeps.","section":"§5.6"},{"comment":"The x-axis label and caption should clarify that the baseline for Figure 4 is sync per-message confirms, not the no-confirm default used in Figure 3. Currently the reader must infer this from the text.","section":"§5.2, Figure 4"},{"comment":"Minor typographical and formatting issues: 'simulations' is used where 'simulation experiments' would be clearer; Table 2 lists 'synchronous confirms for per message and batch of messages' but the distinction between sync-batch and async-batch is not discussed in §4.2.1.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the WORKS workshop scope and has a reproducible artifact, but the central reliability-tax claim needs correction. The async-confirm issue is not a matter of taste—it directly invalidates the headline numbers as stated. The fixes (rerunning with a proper async-confirm client, adding error bars, and re-scoping Lstream to 1 Gbps) are feasible within the manuscript's scope, hence major revision rather than reject. I would also ask the editor to ensure the authors distinguish 'no confirms' from 'async confirms with draining' in all figures and text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the RabbitMQ streaming study from OLCF. Punchline: the new measurements are real and the multi-queue/prefetch findings are worth knowing. But the headline reliability-throughput trade-off is overstated, because the authors compare 'no confirms' against a deliberately blocking use of publisher confirms and then dismiss async confirms as 'least reliable.'\n\nRabbitMQ confirms are asynchronous. A producer can publish continuously, track delivery tags, and wait for acks only at checkpoint or shutdown—getting the same guarantee as sync per-message confirms without putting the RTT on the critical path. The paper's simulator appears to never wait for async acks, so it treats an incomplete client pattern as a protocol property. The 'up to 81% reduction' for reliable delivery and the 4.6x batching recovery are costs of synchronous waiting, not of reliability. Same issue in the durability experiment: 'sync confirm and persistence' bundles a synchronous wait with durability. The practical guidance could push users to batch aggressively when a proper async-confirm client would give reliability at near-zero cost.\n\nWhat's genuinely good: the simulator is public, configurations are explicit, and the parameter isolation is clean. The consumer-ack and prefetch experiments quantify known RabbitMQ behavior in a concrete way. The multi-queue result—peak throughput with a small number of queues, not one per consumer—is a real operational insight for DS2HPC. Section 6 is honest about the 1 Gbps testbed link and the fixed Deleria message size.\n\nMain other weaknesses: no error bars or raw throughput values (three-run averages only), and LCLS results are dominated by the 1 Gbps link, so the Lstream scaling conclusions tell you little about the 100 Gbps path they plan. Still, the qualitative rankings for ack, prefetch, and queue count are likely robust.\n\nWho is this for: DS2HPC users and anyone tuning RabbitMQ for edge-to-HPC streaming. It deserves a serious referee and likely acceptance after revision—the confirm experiments need to be redone with a correct async-confirm client, and the paper should stop calling async 'least reliable.' I'd send it to review with that flag.","headline":"Useful, reproducible RabbitMQ tuning study whose central reliability-tax numbers conflate synchronous waiting with reliability; the ack/prefetch/multi-queue findings are solid.","tokens_in":16016,"tokens_out":2813,"would_cite":false,"duration_ms":31745,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reliability mechanisms in RabbitMQ can cost up to 81% of streaming throughput, and batching recovers most of the loss.","keywords":["data streaming","RabbitMQ","scientific workflows","messaging reliability","publisher confirms","consumer acknowledgements","prefetch","queue parallelism"],"falsifier":"Run the same parameter sweep against live Deleria traffic with variable event batching and against LCLS on a 100 Gbps link; if the 81% sync-confirm penalty, 4.6x batching gain, or 6.1x multi-queue gain do not approximately reproduce, the paper's central characterization fails.","tokens_in":15196,"feed_emoji":"📡","tokens_out":6658,"duration_ms":68394,"temperature":0.7,"pith_summary":"Scientists increasingly want to stream data directly from experimental instruments into HPC memory, but reliable delivery usually seems to conflict with speed. This paper shows, through RabbitMQ simulations of two representative workflows (Deleria's small batched messages and LCLS's 1 MiB files), that the conflict is mostly configurable: synchronous per-message publisher confirms cost up to 81% of throughput, while batching confirms, batching consumer acknowledgements, raising prefetch, and using a small number of parallel queues recover 4.6x to 6.1x. If the simulations hold, workflow users can provision reliable streaming without expensive trial-and-error on real instruments. The paper's contribution is a mapping from tunable RabbitMQ parameters to the reliability-throughput trade-off for these two workloads.","feed_headline":"Batching reclaims RabbitMQ's reliability tax on science streams","feed_subtitle":"A RabbitMQ config sweep shows sync per-message confirms cost up to 81% throughput; batched acks and parallel queues win it back.","key_machinery":"The central mechanism is the round-trip cost of each reliability handshake between producer/consumer and broker. Synchronous confirms, per-message acknowledgements, and persistence-with-confirms each force a wait on the critical path; batching confirms or acks amortizes that wait over many messages; raising prefetch keeps messages in flight during the wait; and parallel queues give the broker additional CPU cores, since RabbitMQ's hot path for one queue uses one core. This round-trip-amortization model does the explanatory work across all six experiments.","core_discovery":"Using a purpose-built simulator that drives RabbitMQ with synthetic Deleria and LCLS workloads, the paper establishes that every reliability mechanism it tests taxes throughput through added round-trips, and that the tax can be amortized. Per-message synchronous publisher confirms reduce Dstream throughput by up to 81% and Lstream by up to 37%; batching 64 confirms at a time improves Dstream 4.6x. Batch consumer acknowledgements (64 messages) give up to 2.7x over per-message acks and near-auto-ack throughput for Deleria. Prefetch alone, with per-message acks, improves throughput 2.5x (Dstream) and 2x (Lstream). And 2-4 parallel queues outperform both one queue and many queues: 6.1x (Dstream)","pith_inferences":["If the round-trip-amortization model is the true cause, the same qualitative ranking should hold for other RabbitMQ workloads, but the optimal prefetch and batch sizes should scale with message size and network latency; that is testable outside this paper.","A likely extension the authors did not run is a mixed-message-size workload matching Deleria's real variable event batching; variable sizes may shift the optimal batch size and the measured gain factors.","The 1 Gbps bottleneck means the LCLS results mostly characterize network saturation rather than RabbitMQ's internal costs; rerunning on 100 Gbps would likely change the saturation point and reveal whether multi-queue gains persist for large messages.","Because DS2HPC also provisions Redis, an equivalent parameter sweep on Redis would show whether these reliability-versus-throughput trade-offs are framework-specific or general."],"forward_implications":["Combine batched publisher confirms with batched consumer acknowledgements and a prefetch count around 64 for Deleria-style traffic to get near-unreliable-mode throughput with full per-message reliability guarantees.","For LCLS-style 1 MiB messages over 1 Gbps, network bandwidth saturates at four consumers, so further consumer scaling or queue parallelism will not help until the network is upgraded.","Durability alone costs little on classic queues; the expensive case is guaranteed persistence, which combines persistence with synchronous confirms and can cut throughput by up to 67%.","The best queue count is a small fraction of consumer count, not equality; 2-4 queues outperform 64 queues, because RabbitMQ queue management overhead dominates at high queue counts.","Configuration selection for these workflows is non-obvious and depends on consumer count, message size, and server resources; simulation is a practical substitute for costly instrument time."],"supporting_citations":[{"why":"The amqp091-go RabbitMQ client library is the basis of the simulator that produced all measured throughput numbers.","marker":"[12]"},{"why":"RabbitMQ documentation defines the semantics of confirms, durability, acknowledgements, prefetch, and queue types that the study varies.","marker":"[9]"},{"why":"The Deleria/GRETA data-pipeline paper supplies the event-batching and 32 Gbps streaming characteristics used to build the Dstream synthetic workload.","marker":"[13]"},{"why":"The LCLS-II paper supplies the large-detector data-rate and real-time-analysis context that motivates the Lstream workload.","marker":"[29]"},{"why":"The LCLS deep-learning pipeline reference provides the live detector-streaming use case the Lstream workload approximates.","marker":"[32]"},{"why":"The DS2HPC design report describes the bidirectional streaming architecture on which the simulations are run.","marker":"[7]"}],"fun_headline_variants":["Batch acks and queues beat RabbitMQ's 81% reliability tax","RabbitMQ reliability tax: 81% throughput loss, batching recovers 4.6x","Sync confirms cost science streams up to 81%; batch to win it back","For science streams, batch RabbitMQ acks to dodge 81% throughput hit","How to beat RabbitMQ's reliability tax: batch acks, parallel queues"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The measured rankings and gain factors transfer to real experiments only if the synthetic workloads are faithful: Deleria's fixed 16 KiB messages and steady 1 MiB LCLS messages on the tested 1 Gbps link.","fun_headline_variants_meta":{"raw":{"variants":["Batch acks and queues beat RabbitMQ's 81% reliability tax","RabbitMQ reliability tax: 81% throughput loss, batching recovers 4.6x","Sync confirms cost science streams up to 81%; batch to win it back","For science streams, batch RabbitMQ acks to dodge 81% throughput hit","How to beat RabbitMQ's reliability tax: batch acks, parallel queues"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000307,"raw_usage":{"total_tokens":1624,"prompt_tokens":801,"completion_tokens":823,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":715}},"tokens_in":545,"tokens_out":823,"duration_ms":8122,"temperature":1.0,"reasoning_tokens":715,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T22:38:36.934672+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same parameter sweep against live Deleria traffic with variable event batching and against LCLS on a 100 Gbps link; if the 81% sync-confirm penalty, 4.6x batching gain, or 6.1x multi-queue gain do not approximately reproduce, the paper's central characterization fails.","supporting_citations":[{"cited_title":"2025.Go RabbitMQ Client Library","cited_arxiv_id":null,"evidence_quote":"The amqp091-go RabbitMQ client library is the basis of the simulator that produced all measured throughput numbers."},{"cited_title":"2025.RabbitMQ 4.1 Documentation","cited_arxiv_id":null,"evidence_quote":"RabbitMQ documentation defines the semantics of confirms, durability, acknowledgements, prefetch, and queue types that the study varies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The LCLS deep-learning pipeline reference provides the live detector-streaming use case the Lstream workload approximates."},{"cited_title":"Brim, Anjus George, Amir Shehata, Corwin Lester, David Rogers, Patrick Widener, Ross Miller, Gustav Jansen, Rafael Ferreira Da Silva, and Sarp Oral","cited_arxiv_id":null,"evidence_quote":"The DS2HPC design report describes the bidirectional streaming architecture on which the simulations are run."}],"review_version":1}