{"id":"9f95768a-fd15-447b-a790-b51f370f4973","arxiv_id":"2505.09375","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Four energy measurement methods were compared on Kubernetes with Nextflow; shell-script and plugin approaches are effective, but concurrent tasks require heuristics for per-task attribution.","lead":"This paper compares four ways to read Intel energy counters while scientific workflows run on Kubernetes-managed clusters. It shows that shell scripts and a Nextflow plugin are practical, and that per-task energy is only easy to assign when tasks do not overlap.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The empirical comparison in Section 5.3 reconstructs all methods from shared RAPL logs and start/stop timestamps, so coverage percentages reflect interval definitions rather than validated end-to-end behavior.","rationale":"The paper is transparent and does useful engineering work: it clearly identifies the practical obstacles to RAPL measurement in Kubernetes/Nextflow, offers four concrete designs, compares them against explicit criteria, and honestly flags missing physical power-meter validation in Sections 2.3 and 7.1. I do not treat the absence of power meters as the central problem, because the paper's claim is about RAPL-measurable energy and the authors acknowledge that boundary. The more load-bearing issue is internal: Section 5.3's simultaneous, post hoc interval extraction means the quantitative comparison does not actually test the methods' end-to-end behavior. The shell-script's 100% coverage is defined by the selected interval, and the plugin/task deficits are consequences of later start boundaries in the same trace. This does not make the paper worthless: the interval reasoning is useful, the LoC comparison supports ease of implementation, and the qualitative lessons about fault handling, portability, and overhead are plausible. But the empirical support for the central effectiveness claim needs separated end-to-end runs, or at least per-run logs and error bars, before the specific percentages are treated as definitive. The reader's verdict was already CONDITIONAL, and the missing code/data artifacts were noted; my concern reinforces the same condition rather than moving the verdict. Agreement with the reader is partial: they focused on RAPL absolute accuracy against physical meters, whereas I focus on the construction of the shell-script baseline and the lack of end-to-end method validation, which is a related but distinct validity threat.","tokens_in":20589,"tokens_out":7848,"duration_ms":87632,"concrete_test":"Independently run each method end-to-end on the same three workflows, with at least five executions per method, and keep the RAPL log files each method actually produces. Compare the totals and start/stop timestamps with the reconstructed values in Tables 4 and 5/6 and Figure 7, and report per-run values with standard deviations. Verify that the shell-script consistently starts before workflow initialization and that the plugin's first hook starts at the reconstructed time. If the actual totals differ from the reconstructed percentages by more than the run-to-run variance, the coverage claims are artifacts of interval definitions rather than validated measurements.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.3 does not run the shell-script, plugin, and task-based methods as separate end-to-end experiments. It states that because all methods read the same RAPL counters, \"it is not necessary to run individual experiments for each method separately\"; instead it records one continuous RAPL trace and \"determine[s] the points in time where each of the methods starts and stops the measurement\" to \"extract the exact amount of captured energy.\" Tables 4 and 5/6 and Figure 7 are therefore built from post hoc interval extraction, not from actually executing each method's coordination logic. The headline finding that the shell-script captures 100% of RAPL-measurable workflow energy while plugin and task-based methods miss 0.19% to 7.67% is then a property of the chosen start/stop boundaries, not an empirical demonstration that the shell wrapper reliably starts before workflow initialization, polls nodes correctly, reacts to faults, and logs all involved nodes. Real failure modes of the implemented methods, such as kubectl polling latency, daemon file-polling delay, plugin hook timing, missed overflow handling, and log-file flushing, are not exercised by this design. The paper's central claim that shell-script and plugin approaches are \"effective and easy to implement\" thus rests on implementation plausibility and LoC counts rather than on measured end-to-end validation. This is load-bearing because the empirical tables are the main quantitative support for the claim; if actual end-to-end runs show different start/stop times, the ranking, percentages, and the conclusion that all three methods are viable for long workflows may change.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses the practical problem of measuring the energy consumption of scientific workflows executed on Kubernetes clusters using Intel RAPL hardware counters. It presents four measurement strategies: managing measurement as part of the workflow (task-based), wrapping workflow execution in a shell script, integrating a Nextflow plugin, and using Prometheus-based monitoring. The methods are compared against eight design criteria (e.g., fault handling, portability, overhead) and implemented for three nf-core workflows (RNASeq, Quantms, Rangeland) on a commodity cluster. The empirical evaluation, based on five repeated runs per workflow, reports that the shell-script method captures the full RAPL-measurable workflow energy, while plugin and task-based methods miss between 0.19% and 7.67% depending on workflow runtime; Prometheus values differ by -3.94% to +36.31% from the shell-script baseline. The authors conclude that the shell-script and Nextflow plugin approaches are both effective and easy to implement, with Prometheus also recommended when its additional software is acceptable.","tokens_in":20808,"tokens_out":4810,"duration_ms":45856,"significance":"If the findings hold, the paper provides cluster users with a practical, low-cost way to obtain near-complete RAPL-based energy data for workflow execution, and it offers a useful eight-criterion checklist for evaluating energy-measurement approaches. The paper is notably transparent about its main limitations: Section 2.3 explicitly states that RAPL accuracy could not be validated against physical power meters on the cluster hardware, and Section 7.1 lists this as future work. The repeated experimental runs (five per workflow), the LoC-based complexity comparison, and the clear exposition of pitfalls such as counter overflow, privilege requirements, and container limitations are strengths. The principal weakness is that the empirical comparison is performed by extracting intervals from a single RAPL trace rather than by executing each method as an independent end-to-end system, so the reported percentages primarily characterize interval-selection logic rather than the implemented methods' runtime behavior; this limits the direct support for the paper's headline claims.","major_comments":[{"comment":"The empirical comparison does not exercise the methods as implemented. The paper states that 'it is not necessary to run individual experiments for each method separately' and instead records one continuous RAPL trace while determining the start and stop times of each method, then extracts the captured energy from the same log. As a result, Tables 4-6 and Figure 7 report the energy implied by the chosen start/stop boundaries, not the energy that the actual shell-script polling loop, daemon file-polling mechanism, plugin hooks, or task-based coordination would capture in real execution. Failure modes that matter for the central claim—kubectl polling latency, daemon polling delay, plugin hook timing, log-file flushing, overflow handling, and fault reactions—are never exercised. The claim that the shell-script and plugin are 'effective' therefore rests on implementation plausibility and LoC counts rather than on measured end-to-end validation. The authors should either run each method as a separate end-to-end experiment or explicitly reframe the results as 'energy captured under the interval definitions used by each method.'","section":"Section 5.3 and Tables 4-6, Figure 7"},{"comment":"The accuracy of RAPL on cluster hardware is assumed, not established. The authors acknowledge they cannot confirm RAPL accuracy for scientific workflows on compute clusters because they lack physical power meters, and all comparisons use the shell-script value as the reference. If RAPL under- or over-reports on the Intel Xeon Silver 4314 CPUs (for example, because the Package and DRAM domains exclude storage and network energy), then the '100% coverage' attributed to the shell-script and the relative percentages for other methods are measured against a possibly biased reference. This limitation is stated transparently, but the abstract and conclusion do not carry the corresponding qualification. The paper should consistently state that the methods capture RAPL-measurable energy, not total node energy, and should clarify whether the coverage percentages are intended as RAPL-relative or physical-energy-relative.","section":"Section 2.3 and Section 7.1"},{"comment":"The Prometheus results exhibit a large, unexplained discrepancy: for the short workflows Quantms and Rangeland, Prometheus reports 30-36% more energy than the shell-script baseline, while for the longer RNASeq workflow it reports about 4% less, and changing the scraping interval from 30s to 10s increases the RNASeq discrepancy instead of reducing it. The authors hypothesize possible causes but state that 'finding the exact nature of these differences remains future work.' Because the conclusion recommends Prometheus as one of the most beneficial methods when its software is available, an unresolved systematic discrepancy in the magnitude of the measured energy directly weakens that recommendation. The paper should either identify the cause (e.g., verify whether Prometheus includes additional metrics or different time windows) or temper the recommendation to note that Prometheus's absolute values require external validation.","section":"Section 5.4, Tables 5-6, and Section 8"},{"comment":"Only averages over five runs are reported, with no per-run values, standard deviations, or statistical tests. The RNASeq differences between methods (0.19% for the plugin versus 0.36% for the task-based method) are small and may be within run-to-run variation; without dispersion measures, the claim that the plugin and task-based methods miss 'about 0.19%' and 'about 0.36%' is not robust. The authors should report per-run data or at least standard deviations and confidence intervals for the key comparisons, especially because the paper's quantitative conclusions about method equivalence rest on these averages.","section":"Section 5.3 and Tables 4-6"}],"minor_comments":[{"comment":"The text says the method 'does not fulfill four of our seven criteria,' but Table 1 lists eight criteria including Multi-tenancy; the count should be eight.","section":"Section 4.2.1"},{"comment":"Section 3.4 states that the register is 'limited to 38 bits,' while Section 2.3 says the registers are limited to 32 or 36 bits depending on CPU model; these numbers should be reconciled.","section":"Section 3.4 versus Section 2.3"},{"comment":"The example Prometheus query uses 'node_hwmon_power_average_watt,' which is a power value, and multiplies by 30 to obtain energy; the text should clarify whether the underlying metric is power or energy and how the multiplication by the scraping interval works.","section":"Section 4.2.4"},{"comment":"Figures 9 and 10 in the appendix are not cited in the main text; they should either be referenced in Section 5 or removed.","section":"Appendix, Figures 9 and 10"},{"comment":"Table 3 lists RNASeq as having 9 physical tasks, but the modified pipeline described in Section 5.2 names seven tasks (fastp, star index, fastqsplit, star align, samtools, samtools merge, cufflinks); the task count should be clarified.","section":"Section 5.2 and Table 3"},{"comment":"The abstract mentions 'one additional method using IPMI,' but the fourth method is Prometheus reading energy values through RAPL/IPMI/ACPI; the wording should be adjusted to avoid implying a separate IPMI-only method.","section":"Abstract and Section 4.2.4"}],"recommendation":"major_revision","confidential_remarks":"This is a well-written lessons-learned/experience paper whose practical value is clear, but its empirical core is currently more limited than the abstract suggests because the method comparison is based on interval extraction from a shared RAPL log rather than end-to-end execution of each method. The authors are transparent about the RAPL-accuracy limitation, which is commendable, but the Prometheus discrepancy and the absence of variance reporting are unresolved issues that the revision should address. The paper may be a better fit for a venue that values practical experience reports and reproducibility artifacts over novel algorithmic contributions; if the journal is highly selective about empirical validation, the bar may be higher than what is currently provided."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — quick read on arXiv:2505.09375. The useful news: this is an honest, practical comparison of four ways to wire RAPL energy measurement into Nextflow workflows on Kubernetes, and it will save a newcomer time. The eight-criteria table is a good checklist, and the authors are unusually upfront about what they could not validate: they state in Section 2.3 that they have no physical power meters, and they flag the unresolved Prometheus discrepancy rather than hiding it. That honesty is real.\n\nThe bigger news is a methodological caveat that the paper itself underplays. Section 5.3 says they did not run the shell-script, plugin, and task-based methods as separate end-to-end experiments. They ran each workflow once, collected one RAPL trace, then computed what each method would have captured by applying start/stop timestamps post hoc. The coverage percentages in Tables 4 and Figure 7 are therefore properties of interval boundaries, not demonstrations that the implementations work under real conditions. The shell wrapper's polling latency, the daemon's file-watch delay, plugin hook timing, overflow handling, log flushing — none of those are exercised. The claim that all three are viable for long workflows may well be true, but the empirical tables are not actually testing the implementations.\n\nOther soft spots, in rough order: no per-run values or error bars; no code or data artifacts (LoC counts without the code are hard to check); the shell-script is treated as the 100% reference even though RAPL accuracy on cluster hardware is explicitly unvalidated; and the Prometheus numbers are 30–36% off for short workflows and still about 4% off for the long one, with the authors unable to explain it. That last one matters because the conclusion recommends Prometheus when the software is available, which sits awkwardly with the abstract naming shell and plugin as the effective choices.\n\nNone of this breaks the central qualitative finding — the shell-script and plugin are lightweight, plausible, and should work for long-running workflows — but the quantitative framing deserves a major revision, not a pass. My recommendation: send it to peer review, and insist that the comparison be run end-to-end (or that the reconstruction be justified and the intervals validated), and that code and data accompany the final version.","headline":"Honest and useful engineering comparison, but the empirical tables are built from post hoc interval extraction rather than end-to-end runs, so the coverage percentages are provisional.","tokens_in":21374,"tokens_out":2936,"would_cite":true,"duration_ms":30008,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Cluster users can capture full RAPL workflow energy with a shell script or Nextflow plugin.","keywords":["RAPL energy measurement","scientific workflows","Nextflow","Kubernetes","energy consumption","cluster computing","workflow energy monitoring","IPMI"],"falsifier":"Run the same three workflows on the same two-node cluster while measuring node-level power with a calibrated external physical power meter; if the shell-script-wrapped RAPL total differs systematically from the meter (beyond the known 21-37% of non-CPU/DRAM components such as disks, network, fans, and power supply), then the shell-script baseline is not the true energy of the workflow and the coverage percentages measured against it would shift accordingly. A second check would be to run an idle node for one hour and compare RAPL Package+DRAM deltas to the meter, isolating the fixed offset.","tokens_in":20345,"feed_emoji":"⚡","tokens_out":3376,"duration_ms":26183,"temperature":0.7,"pith_summary":"This paper addresses a practical gap: on managed Kubernetes clusters, reading Intel RAPL energy counters for a running scientific workflow is complicated by limited privileges, multi-node scheduling, and container constraints. The authors implement and compare four strategies—measurement managed inside the workflow, a wrapping shell script, a Nextflow plugin, and Prometheus-based monitoring—against eight design criteria. Their experiments with three real workflows show that the shell-script method captures the full RAPL-measurable workflow energy, while the plugin and task-based methods miss between 0.19% and 7.67% depending on workflow runtime. They conclude that the shell script and the Nextflow plugin are both effective and easy to implement, and that task-level measurement is exact only when one task runs at a time.","feed_headline":"A shell script or plugin captures full RAPL energy on clusters","feed_subtitle":"On Kubernetes, plugin-based measurement misses under 8% for short workflows; only a wrapper script captures it all.","key_machinery":"The central mechanism is the coordination of measurement start and stop around workflow execution: a privileged monitoring pod on each node continuously reads the RAPL model-specific registers (MSRs) and logs timestamped energy values, and the four methods differ in what triggers the measurement. The shell script triggers measurement before launching the workflow and stops it when the workflow ends, which is why it is the only method with full coverage; the Nextflow plugin hooks into workflow start and task boundaries; task-based management embeds start/stop signals into the workflow itself; Prometheus polls continuously via hwmon/IPMI. The load-bearing identity is that RAPL MSR delta over a time interval equals workflow energy, provided no other workload runs on the node.","core_discovery":"The paper's central claim is that on a commodity Kubernetes cluster, RAPL-based energy measurement can be achieved without cluster-administrator intervention by wrapping the Nextflow workflow in a shell script or by using a Nextflow plugin, and that both approaches capture essentially all of the RAPL-measurable energy of the workflow. In head-to-head measurements on three workflows (RNASeq, Quantms, Rangeland), the shell-script method is the only one that starts measurement before workflow initialization and therefore captures 100% of the RAPL-visible energy; the plugin and task-based methods miss between 0.19% and 7.67% of that energy, with the gap driven mainly by the few seconds of startup delay before measurement begins, amortized over total runtime. Prometheus, by contrast, reports deviations from the shell-script baseline that are not explained by polling interval alone, leaving its energy-reporting mechanism as an open question.","pith_inferences":["The 0.19% to 7.67% coverage gaps imply a practical rule of thumb: for workflows running longer than about 30 minutes, the startup-delay penalty of plugin-based measurement is negligible, making the plugin the default choice, whereas the shell script matters mainly for short workflows or when 100% coverage is a formal requirement.","Because RAPL Package and DRAM domains cover roughly 63-79% of a server's total energy, these methods measure workflow CPU-and-memory energy only; any workflow energy reported by these tools should be interpreted as 'CPU plus DRAM energy', not total node energy, and energy-optimization claims should be scoped accordingly.","The paper's measurement strategy could be reused to benchmark other monitoring stacks: any energy reporter whose output disagrees with the RAPL shell-script baseline on idle nodes is likely adding non-RAPL components or applying its own estimation, exactly as suspected for Prometheus.","A natural next experiment would be to run the shell-script method against a physical wall power meter on the cluster's two-node testbed to check the assumed accuracy of RAPL itself for containerized workflow workloads; a systematic under- or over-reporting there would change the meaning of all coverage percentages."],"forward_implications":["Workflow users on managed clusters can obtain near-complete RAPL energy data with a single wrapper script, with no cluster-administrator involvement beyond deploying privileged monitoring pods.","The roughly 0.2% to 7.7% gap between the full-coverage shell-script method and plugin/task-based methods is a fixed startup delay that shrinks for longer workflows, so the plugin becomes effectively equivalent for long-running production workflows.","Per-task energy measurement is reliable only for sequential, non-overlapping tasks longer than one second; concurrent or sub-second tasks require heuristics such as CPU utilization, so per-task data should be labeled as approximate in those cases.","Prometheus-based energy readings can disagree with RAPL by more than 30% on short workflows and even on idle periods, so it should not be assumed that Prometheus returns the same underlying RAPL values.","A measurement method's coverage can be quantified as the fraction of the shell-script baseline RAPL energy captured, giving future methods a simple comparable metric."],"supporting_citations":[{"why":"Khan et al., 'RAPL in Action', supplies the background on RAPL domains, MSR access, update frequency, and overflow behavior that justifies the need for continuous monitoring.","marker":"[19]"},{"why":"David et al., the original RAPL paper, establishes the RAPL interface as the energy counter that all methods read.","marker":"[7]"},{"why":"Hackenberg et al., 'An Energy Efficiency Feature Survey of the Intel Haswell Processor', provides the accuracy validation for RAPL that the paper relies on for its claim that RAPL readings are reliable.","marker":"[15]"},{"why":"Di Tommaso et al., the Nextflow paper, grounds the choice of Nextflow as the workflow engine and the plugin integration mechanism.","marker":"[8]"},{"why":"Carrión, 'Kubernetes Scheduling', supplies the background on Kubernetes orchestration that defines the cluster environment where the methods run.","marker":"[5]"},{"why":"Jay et al., 'An experimental comparison of software-based power meters', supplies the external power-meter accuracy standard against which RAPL is generally validated and which the paper's future validation would use.","marker":"[16]"}],"fun_headline_variants":["Shell script captures 100% RAPL energy on clusters","RAPL on Kubernetes: shell script beats plugin for full energy","Complete RAPL energy capture via shell wrapper on clusters","Plugin misses up to 8% of RAPL energy; shell script gets it all","For accurate cluster energy, use a shell script with Nextflow"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that RAPL MSR readings correctly represent the energy that a workflow task actually consumes on a cluster node, and it treats the shell-script method as the true 100% reference even though no physical power meter was available to verify RAPL itself on the cluster hardware.","fun_headline_variants_meta":{"raw":{"variants":["Shell script captures 100% RAPL energy on clusters","RAPL on Kubernetes: shell script beats plugin for full energy","Complete RAPL energy capture via shell wrapper on clusters","Plugin misses up to 8% of RAPL energy; shell script gets it all","For accurate cluster energy, use a shell script with Nextflow"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000241,"raw_usage":{"total_tokens":1576,"prompt_tokens":1054,"completion_tokens":522,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":670,"completion_tokens_details":{"reasoning_tokens":430}},"tokens_in":670,"tokens_out":522,"duration_ms":4571,"temperature":1.0,"reasoning_tokens":430,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:32:33.381513+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same three workflows on the same two-node cluster while measuring node-level power with a calibrated external physical power meter; if the shell-script-wrapped RAPL total differs systematically from the meter (beyond the known 21-37% of non-CPU/DRAM components such as disks, network, fans, and power supply), then the shell-script baseline is not the true energy of the workflow and the coverage percentages measured against it would shift accordingly. A second check would be to run an idle node for one hour and compare RAPL Package+DRAM deltas to the meter, isolating the fixed offset.","supporting_citations":[{"cited_title":"Nurminen, and Zhonghong Ou","cited_arxiv_id":null,"evidence_quote":"Khan et al., 'RAPL in Action', supplies the background on RAPL domains, MSR access, update frequency, and overflow behavior that justifies the need for continuous monitoring."},{"cited_title":"Hanebutte, Rahul Khanna, and Christian Le","cited_arxiv_id":null,"evidence_quote":"David et al., the original RAPL paper, establishes the RAPL interface as the energy counter that all methods read."},{"cited_title":"An Energy Efficiency Feature Survey of the Intel Haswell Processor","cited_arxiv_id":null,"evidence_quote":"Hackenberg et al., 'An Energy Efficiency Feature Survey of the Intel Haswell Processor', provides the accuracy validation for RAPL that the paper relies on for its claim that RAPL readings are reliable."},{"cited_title":"Nextflow enables reproducible computational workflows","cited_arxiv_id":null,"evidence_quote":"Di Tommaso et al., the Nextflow paper, grounds the choice of Nextflow as the workflow engine and the plugin integration mechanism."},{"cited_title":"Kubernetes Scheduling: Taxonomy, Ongoing Issues and Challenges","cited_arxiv_id":null,"evidence_quote":"Carrión, 'Kubernetes Scheduling', supplies the background on Kubernetes orchestration that defines the cluster environment where the methods run."},{"cited_title":"An experimental comparison of software-based power meters: focus on CPU and GPU","cited_arxiv_id":null,"evidence_quote":"Jay et al., 'An experimental comparison of software-based power meters', supplies the external power-meter accuracy standard against which RAPL is generally validated and which the paper's future validation would use."}],"review_version":1}