{"id":"42a41e96-1e37-4e9d-b039-3ce5c5f50cb8","arxiv_id":"2506.08727","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Benchmark-driven regression models can estimate prompt-level LLM inference energy and carbon emissions with roughly 15% average error on a curated external dataset.","lead":"The paper proposes R-ICE, a framework that estimates the carbon and energy cost of an LLM inference request from model size and public benchmark latency data, avoiding the need to instrument the deployment environment. It reports about 15% average error on an external energy leaderboard, suggesting that lightweight estimation of LLM emissions is within reach.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validation rescaling in Eq. (2) discards input-prompt information, so the reported 15.3% energy MAPE does not actually substantiate prompt-level estimation.","rationale":"The reader's weakest_assumption targets the fixed utilization (0.26), PUE (1.1), and minimum-hardware assumptions as multipliers that can scale estimates away from true values. That is a valid generalization concern, but it is acknowledged in the paper and user-configurable. The concern raised here is more fundamental: the validation protocol in Section V uses Eq. (2) to rescale ground-truth leaderboard values by output tokens alone, which removes the input-prompt dimension from the comparison. Since the paper's claimed granularity is prompt-level, the headline MAPE numbers do not currently demonstrate that capability. This is an internal consistency issue between the stated contribution and the validation, not merely an external-transfer issue. A concrete fix is to use the leaderboard's prefill and per-token latencies to reconstruct a proper prompt-level ground truth, or to restrict validation to points with matching native input/output lengths. If the error persists under the corrected protocol, the CONDITIONAL verdict should remain but with the additional condition of fixing this scaling. If the error degrades substantially, the central claim should be downgraded to 'unverified' pending re-validation. Given the paper's honest limitations and workshop context, I do not think the verdict category itself changes; rather, the specific condition 'expand the validation protocol' should include correcting Eq. (2). Therefore I leave the reader's CONDITIONAL verdict unchanged, with agreement that the validation needs expansion, though for a different and more immediate reason than the constants concern.","tokens_in":7455,"tokens_out":10983,"duration_ms":128791,"concrete_test":"Re-run the Section V validation without the Eq. (2) rescaling. For each of the 46 points, use the leaderboard's reported prefill latency and per-token decode latency (or reconstruct them from the native input/output lengths) to compute a ground-truth E2E latency and energy for the 192-input/250-output prompt. If the leaderboard lacks such decomposition, restrict evaluation to points whose native input and output lengths closely match the evaluation prompt and recompute MAPE on that subset. If the latency or energy MAPE changes substantially (e.g., energy MAPE exceeds 30%), the prompt-level accuracy claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—accurate prompt-level energy/carbon estimation—rests on the external validation in Section V. For the 46 leaderboard points, the ground-truth E2E latency is rescaled as E2E/256 * Output tokens and ground-truth energy as Output tokens / tok/kWh (Eq. 2). These formulas treat total latency and total energy as proportional to output-token count alone, ignoring the prompt-encoding/prefill phase that the framework itself models with Regression Model 2 and Eq. (1). The prediction side uses prompt-encoding latency for 192 input tokens plus per-token output latency for 250 output tokens, while the ground-truth side is merely a linear rescaling of a leaderboard value measured at some unspecified input length (the factor 256 suggests a default output-token length). Input-token dependence—the essence of 'prompt-level' granularity—is thus removed from the ground truth. If the leaderboard's native workload used a different input length (e.g., 512 tokens), the E2E ground truth is never corrected for that difference. The energy ground truth similarly omits prefill energy. Consequently, the reported 15.3% energy MAPE may reflect the accuracy of a single scalar power-conversion factor rather than the framework's ability to estimate emissions for arbitrary prompts. This is a load-bearing flaw because the paper's contribution over existing tools is precisely prompt-level granularity.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes R-ICE, a regression-based framework for estimating prompt-level inference energy and carbon emissions of LLMs without intrusive monitoring. It trains three regression models on HELM efficiency data: minimum GPU count versus parameter count, prompt-encoding latency versus input tokens and parameters, and per-output-token latency versus parameter count; these are combined with TDP, utilization, PUE, and regional carbon intensity. The framework is validated on 46 filtered points from the LLM-Perf leaderboard, with reported average MAPEs of 22.9% for E2E latency and 15.3% for energy, from which the paper concludes that prompt-level energy/carbon estimation is accurate.","tokens_in":7760,"tokens_out":9331,"duration_ms":112567,"significance":"The core idea—using public benchmark efficiency data to build a non-intrusive, low-input estimator for inference emissions—is timely and potentially useful for pre-inference LLM routing and sustainability accounting. The paper makes its assumptions explicit, attempts external validation on an unseen leaderboard, and candidly lists limitations such as small training sets and singleton-batch assumptions. However, the current validation does not actually test prompt-level granularity, so the headline accuracy numbers are not yet evidence for the paper's central claim. If the validation is reworked, the approach could become a useful, practical baseline in this space.","major_comments":[{"comment":"The validation ground truth is constructed to be a function of output-token count only: Scaled(E2E Lat.) rescales the leaderboard E2E by output tokens/256, and Scaled(Energy) is output tokens divided by tok/kWh. Neither expression contains any input-token or prefill information, even though Eq. (1) and Regression Model 2 in the framework explicitly model prompt-encoding latency as a function of input tokens. Because all 46 data points are evaluated at a single prompt length (192 input tokens, 250 output tokens), the reported 22.9%/15.3% MAPE can only speak to output-token scaling and model-specific constant factors; it does not substantiate the Section V statement that estimates are 'accurate at a prompt level.' The authors should compare against leaderboard measurements with distinct, reported input and output lengths, or otherwise preserve the prefill contribution in the ground truth, and should also compare against a trivial output-token-only baseline to show that the prompt-aware model adds value.","section":"Section V, Eq. (2)"},{"comment":"The validation is performed at one fixed operating point: 192 input tokens and 250 output tokens for every one of the 46 models. This yields a single MAPE per model at one prompt, not a distribution over prompts, so it cannot establish that the method tracks variation in input prompt length, which is precisely the paper's stated granularity contribution. At minimum, the authors should report errors over a range of input and output token counts and provide confidence intervals for the aggregate MAPE.","section":"Section V"},{"comment":"Regression Model 3 (per-output-token latency) is fit to 10 data points and Regression Model 1 (minimum devices) to fewer than 15 data points, with reported R² values computed in-sample and no held-out split for these components. Because the per-token latency enters Eq. (1) multiplicatively and the device count enters the carbon formula as a factor, overfitting in either component propagates directly into every energy estimate. The Limitations section acknowledges the small datasets, but Section V does not report how sensitive the validation MAPE is to these two regressions; leave-one-out or bootstrap assessment is needed before the accuracy claim can be regarded as robust.","section":"Section IV-B"},{"comment":"The energy and carbon estimates inherit three unvalidated multiplicative constants: fixed hardware utilization 0.26, PUE 1.1, and the minimum-A100 deployment assumption with batch size 1. These constants are not calibrated or varied in Section V, and the leaderboard's hardware and workload conditions are not reported. Since any error in these constants scales the final estimate linearly, the 15.3% energy MAPE should be stated as conditional on these assumptions; without a sensitivity analysis (for example, utilization in [0.1, 0.9] or batch size > 1), the headline accuracy number is not informative about real-world deployments.","section":"Section IV-A1 and Fig. 1"}],"minor_comments":[{"comment":"The symbol p is used both for the input token count and for 'prompt encoding time(p)'; this should be written as a function, e.g., T(p,o) = f(p) + (o-1)β, to avoid ambiguity.","section":"Eq. (1)"},{"comment":"The XGBoost MAPE is reported as 0.60, which is an order of magnitude worse than Gradient Boosting despite having a higher R²; this is likely a typo for 0.060 and should be corrected.","section":"Table III"},{"comment":"The units are inconsistent: Scaled(Energy) = Output tokens / (tok/kWh) yields kWh, while Table V labels Energy in J. The conversion factor should be stated explicitly.","section":"Eq. (2) and Table V"},{"comment":"The origin of the denominator 256 is not explained; the authors should state the leaderboard's native output-token length and input-token length so that the scaling in Eq. (2) can be reproduced and assessed.","section":"Section V, Eq. (2)"},{"comment":"The acronym is given as RICE in the Section IV title but R-ICE elsewhere; use one consistent form.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the core idea is useful — use public benchmark data to estimate inference emissions without touching the deployment environment — and the paper is pleasantly honest about its limits. But the validation in Section V does not actually test the claim of prompt-level granularity, because Eq. 2 strips out input-token information from the ground truth. That is a real gap, not a quibble.\n\nWhat is new here: R-ICE fits three cheap regressions to HELM efficiency data — device count vs. parameters, prompt-encoding latency, and per-output-token latency — and then combines them with utilization, PUE, and carbon intensity. This is not conceptually deep, but it is a sensible non-intrusive estimation route, and checking it against an external leaderboard (LLM-Perf) is the right instinct. The Limitations section is candid about small datasets and singleton batching.\n\nThe soft spots are proportional. Most importantly, the validation rescales the leaderboard's E2E latency as (E2E/256) * output tokens, and energy as output tokens / tok/kWh. That treats latency and energy as linear in output tokens only, ignoring the entire prefill/prompt-encoding phase that the framework itself models. The evaluation fixes input at 192 tokens and output at 250, but the ground truth is never corrected for the leaderboard's actual input length. So the reported 22.9% latency and 15.3% energy MAPE mainly test the tok/kWh conversion and a linear output-token scaling, not the model's ability to estimate emissions for varying prompts. The per-output-token latency model is fit on 10 points; the device-count model on less than 15. The fixed utilization (0.26) and PUE (1.1) multipliers mean real deployments with different specs will see linearly scaled errors, and no confidence intervals are given. Those are all acknowledged in Limitations, except the Eq. 2 issue, which is not.\n\nWho gets value from this: practitioners in sustainable AI who want a quick, no-access-needed estimator, and workshop audiences interested in benchmark-driven carbon accounting. The paper deserves a serious referee because the direction is promising and the flaws are fixable. My recommendation: accept conditionally with the validation redone — vary input and output lengths, report per-token or per-prompt errors, justify the constants or show sensitivity. As it stands, the headline accuracy numbers are not evidence for prompt-level granularity.","headline":"The benchmark-based estimation idea is practical and honestly presented, but the external validation rescales away the input-token granularity that is the whole point, so the headline accuracy numbers overstate what is demonstrated.","tokens_in":8285,"tokens_out":4692,"would_cite":false,"duration_ms":51008,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Benchmark-trained regressions estimate prompt-level LLM inference energy within about 15 percent average error, enabling non-intrusive carbon accounting.","keywords":["LLM inference","carbon emissions estimation","energy estimation","benchmark-based modeling","prompt-level granularity","regression","sustainable AI","non-intrusive estimation"],"falsifier":"Measure actual energy with a power meter on a production model served at batch size 8 with known utilization, run the same prompts through R-ICE, and compare: if energy error substantially exceeds 15 percent, the fixed utilization and singleton-batch assumptions are the weak point.","tokens_in":7259,"feed_emoji":"⚡","tokens_out":8304,"duration_ms":88516,"temperature":0.7,"pith_summary":"The paper argues that public LLM efficiency benchmarks, which already report latency and energy for many models, can be repurposed as training data for lightweight regression models that estimate the carbon footprint of a single inference prompt. It presents R-ICE, a framework that takes only a model's parameter count and the input and output token counts, then predicts end-to-end latency and energy use before the prompt is sent. On an unseen external leaderboard, energy estimates land within about 15 percent of reported values on average and latency within about 23 percent. The authors claim this makes prompt-level carbon accounting practical without intrusive monitoring or access to proprietary model internals.","feed_headline":"Benchmark data alone estimates AI energy to ~15 percent","feed_subtitle":"An estimator built from public LLM benchmarks gives prompt-level carbon numbers before inference, no server access needed.","key_machinery":"The mechanism is a latency decomposition: end-to-end runtime equals prompt-encoding latency plus the product of per-output-token latency and the number of generated tokens. Three regression models feed this decomposition — parameter count to minimum GPU count, parameter count and input tokens to prompt-encoding latency, and parameter count to per-output-token latency — and the resulting latency is multiplied by hardware count, thermal design power, utilization, data-center power usage effectiveness, and regional carbon intensity to get grams of CO2-equivalent. The framework deliberately replaces proprietary architecture details with two public numeric features, parameter count and token counts, which is what makes closed models addressable.","core_discovery":"The central claim is that prompt-level inference energy and carbon emissions can be estimated accurately from benchmark data alone, using regression models that map model size and token counts to latency, then convert latency to energy through a standard operational-carbon formula. The framework trains one linear model for minimum GPU deployment, a random-forest model for prompt-encoding latency, and a polynomial model for per-output-token latency, with total latency as the sum of encoding time and per-token time times the number of output tokens. Validation on 46 filtered data points from an independent efficiency leaderboard gives an average mean absolute percentage error of 22.9 percent for end-to-end latency and 15.3 percent for energy consumption. The authors read this as evidence that benchmark-based estimation is accurate enough for pre-inference use cases such as dynamic LLM routing and carbon accounting.","pith_inferences":["If the 15 percent energy error holds across broader model families and hardware, benchmark-based estimation could complement or replace fine-grained monitoring for carbon reporting, since the same method scales to any LLM covered by a public benchmark.","The fixed utilization and PUE assumptions make estimates proportional to the true value when deployments differ only by these factors; a straightforward extension would calibrate utilization as a function of batch size, making the framework valid for production batching.","The latency decomposition may transfer to other autoregressive sequence models, including multimodal models whose text decoding dominates runtime, a testable extension.","A public repository of benchmark-derived carbon estimates would let auditors compare vendor claims against independent numbers, which the paper does not itself develop."],"forward_implications":["Organizations can obtain prompt-level carbon estimates for closed or restrictive-license LLMs without vendor architecture disclosures.","Carbon-aware routing becomes feasible at design time, letting a router compare estimated emissions across models before sending a prompt.","A green ranking of LLMs can be derived from benchmark-based estimates and plugged into orchestration engines.","The approach works pre-inference, unlike monitoring tools that require access to the deployment environment after inference.","With only a few input features, the framework can be adopted at enterprise scale with low integration effort."],"supporting_citations":[{"why":"Supplies the latency decomposition (Eq. 1) and the training dataset of prompt-encoding and per-output-token latencies.","marker":"[12]"},{"why":"Provides the unseen external leaderboard data points used to validate the energy and latency estimates.","marker":"[15]"},{"why":"Source of the default 0.26 hardware utilization figure for the BLOOM model.","marker":"[14]"},{"why":"An analytical estimation tool the paper contrasts with, reliant on architecture details the framework avoids.","marker":"[8]"},{"why":"The efficiency benchmark framework whose data feeds the regression models.","marker":"[11]"}],"fun_headline_variants":["Benchmark data alone estimates AI energy to ~15%","No server access: benchmarks predict LLM carbon footprint","Public benchmarks replace energy monitors for LLM inference","From model size to carbon: benchmark-based emissions estimates","A non-intrusive path to LLM energy estimates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The estimates assume every model runs on a minimum number of a single GPU model at 26 percent utilization with a power usage effectiveness of 1.1 and a batch size of one, so any deployment with different utilization, batching, or hardware scales the estimates away from truth.","fun_headline_variants_meta":{"raw":{"variants":["Benchmark data alone estimates AI energy to ~15%","No server access: benchmarks predict LLM carbon footprint","Public benchmarks replace energy monitors for LLM inference","From model size to carbon: benchmark-based emissions estimates","A non-intrusive path to LLM energy estimates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000281,"raw_usage":{"total_tokens":1656,"prompt_tokens":926,"completion_tokens":730,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":653}},"tokens_in":542,"tokens_out":730,"duration_ms":8423,"temperature":1.0,"reasoning_tokens":653,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:02:53.972217+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure actual energy with a power meter on a production model served at batch size 8 with known utilization, run the same prompts through R-ICE, and compare: if energy error substantially exceeds 15 percent, the fixed utilization and singleton-batch assumptions are the weak point.","supporting_citations":[{"cited_title":"Cheaply estimating inference efficiency metrics for autoregressive transformer models,","cited_arxiv_id":null,"evidence_quote":"Supplies the latency decomposition (Eq. 1) and the training dataset of prompt-encoding and per-output-token latencies."},{"cited_title":"Llm-perf leaderboard,","cited_arxiv_id":null,"evidence_quote":"Provides the unseen external leaderboard data points used to validate the energy and latency estimates."},{"cited_title":"Estimating the carbon footprint of bloom, a 176b parameter language model,","cited_arxiv_id":null,"evidence_quote":"Source of the default 0.26 hardware utilization figure for the BLOOM model."},{"cited_title":"LLMCarbon: Modeling the end-to-end carbon footprint of large language models,","cited_arxiv_id":null,"evidence_quote":"An analytical estimation tool the paper contrasts with, reliant on architecture details the framework avoids."}],"review_version":1}