{"id":"745b24b5-accb-4050-b8ca-31a4ecf4783e","arxiv_id":"2509.22092","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Validation experiments on 50 models show CodeCarbon and the ML Impact Calculator underestimate or overestimate AI inference energy by up to 40 percent compared with external wall-meter measurements.","lead":"This paper measured the real electricity use of 50 AI models with a plug-in energy meter and compared it with two popular estimation tools. The tools follow general energy trends but miss by up to 40 percent, so reported AI energy and carbon numbers should be treated as rough estimates.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The uncalibrated plug-meter/OCR ground truth cannot support the specific error ranges: per-run energy (~0.01 kWh) is at the display resolution of typical plug-in meters.","rationale":"The reader's weakest assumption—that the external meter plus camera OCR pipeline yields true energy consumption—is exactly the load-bearing concern. The paper's headline is a quantitative claim about estimation error percentages, and those percentages are only as reliable as the ground-truth measurement. The paper does not report the LogiLink meter's resolution or calibration, and it does not report the OCR classifier's accuracy. For short, low-energy runs, the displayed energy can be a single 0.01 kWh step, so quantization alone can dominate the measured value. This is not a disagreement with consensus or a stylistic issue; it is a direct threat to the central numbers. The direction of the errors may still be correct, but the specific figures (-40% to +40%, 20-30%) are not established. The paper has real strengths: a reproducible setup, released code and logs, three-run averaging, and a sensible experimental design across 50 models. Those strengths support the qualitative conclusions and the usefulness of the validation framework, but they do not fix the missing calibration. A CONDITIONAL verdict is appropriate: the quantitative claims should be accepted only after the authors either calibrate the measurement pipeline or report its resolution and accuracy and show that the quantization does not materially change the error ranges. The reader already reached this verdict, so no change is recommended.","tokens_in":12157,"tokens_out":2363,"duration_ms":24312,"concrete_test":"In the released repository, add a calibration experiment: run a known resistive load (e.g., a 100 W incandescent bulb for 2 hours) through the same LogiLink meter and camera-OCR pipeline, and compare against a calibrated reference meter. Also record the meter's display resolution and the OCR per-digit error rate. If the resulting calibration offset or quantization error exceeds 5% of the per-run energies observed in the paper (~0.01 kWh for 2-minute vision runs), recompute Figures 1 and 4 and the Section 4 error ranges with corrected values.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claims (takeaways 3 and 4 in Section 4, and Figure 1) depend entirely on the external measurement pipeline described in Sections 2.3 and 2.4. The LogiLink meter's resolution, accuracy, and calibration are not reported, nor is the accuracy of the custom random-forest OCR classifier. For a 2-minute vision run at roughly 300 W, total energy is 36000 Ws = 0.01 kWh, which is the display step of many inexpensive plug-in kWh meters. If the meter increments in 0.01 kWh steps, measurement quantization alone is about ±50% per run (and worse for smaller, lower-power models), so the reported -40% to +40% static errors and 20-30% dynamic underestimates are not robust. Averaging over three runs does not remove this systematic display quantization, and OCR errors could add or cancel further. The qualitative direction of underestimation may survive, but the precise error percentages—the paper's headline finding—are not grounded without a calibration study.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript validates two widely used AI energy-estimation tools, the ML Impact Calculator (static TDP-based estimates) and CodeCarbon (dynamic hardware profiling), against external wall-plug measurements. The authors build a low-cost ground-truth pipeline consisting of a LogiLink plug-in energy meter, a Logitech camera, and a custom random-forest OCR classifier, and apply it to 30 vision classifiers and 20 language models under different batch sizes, temperatures, and CPU/GPU settings. They report that static estimates under- or overestimate by roughly -40% to +40%, while dynamic estimates consistently underestimate by 20-30%, and they derive six takeaways about how energy demand and estimation error scale with model size, hyperparameters, and processor choice.","tokens_in":12485,"tokens_out":5063,"duration_ms":49557,"significance":"If the quantitative error ranges are reliable, this is a useful and timely result for sustainable-AI practice: it would demonstrate that CodeCarbon-based carbon footprint reports for local inference are systematically understated in most cases. The paper has concrete strengths: it is reproducible in principle (code and logs are provided), the experiment spans 50 models across two domains, the comparison is not circular because no constants are fitted from tool outputs, and the general qualitative pattern (dynamic estimates track ground truth but with a downward bias) is plausible and consistent with known sources of overhead such as power supplies and cooling. However, the headline quantitative percentages rest entirely on the accuracy of the external meter/OCR pipeline, and that pipeline is not adequately characterized in the manuscript.","major_comments":[{"comment":"This is a load-bearing issue: Figure 1 and Section 4, Takeaways 3 and 4, depend entirely on the external measurements from Sections 2.3-2.4.","section":"§2.3, §2.4, §4 (Takeaways 3 and 4), Figure 1"}],"minor_comments":[{"comment":"Minor presentation/clarity issue.","section":"§2.4"},{"comment":"Minor presentation/clarity issue.","section":"§2.4, Figure 4"},{"comment":"Minor presentation/clarity issue.","section":"§3, Figures 4 and 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a resource-aware ML or sustainable-AI venue, and the open code/data are a genuine asset. My main concern is the unquantified uncertainty in the ground-truth meter/OCR pipeline, which is a load-bearing component of the specific error percentages. If the authors can supply a calibration study and OCR validation, or downgrade the claims to qualitative trends, the paper would be defensible. I see no circularity or fitting-to-tool-output problem. The single-environment limitation is acknowledged in the Discussion and is not by itself fatal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a real contribution — the first broad external validation of CodeCarbon and the ML Impact Calculator — but the central quantitative claims are not yet grounded. The meter/OCR ground truth is under-specified, and the specific error ranges in the abstract and takeaways do not survive scrutiny. I would still send it to review, expecting major revision or a softening of the numbers.\n\nWhat's new and good: 50 models (30 vision, 20 LLMs), hundreds of inference runs, and a direct comparison of dynamic estimation (CodeCarbon), static estimation (ML Impact Calculator), and wall-meter ground truth. Code and logs are public. That alone fills a gap: many papers report CodeCarbon numbers, but essentially none validate them externally at this scope. The qualitative patterns are plausible and useful: dynamic estimation consistently underestimates wall power; static estimation has a sweet spot for mid-sized models and over/underestimates small and large ones; CPU inference errors dwarf GPU errors; batch size and temperature materially change both energy and error. The Figure 2 framework is straightforward and reproducible.\n\nThe soft spot is exactly what the stress-test note flags. The paper never states the LogiLink meter's resolution, calibration, or accuracy, and never gives the OCR classifier's accuracy. A 2-minute vision run at roughly 300 W consumes about 0.01 kWh, which is the display step of many inexpensive plug-in meters. If that is this meter's resolution, per-run quantization alone is on the order of ±50%. Averaging three runs does not remove systematic display quantization. The abstract's \"errors up to 40%\" and takeaways 3 and 4 (the −40/+40% static range and the 20–30% dynamic underestimates) depend entirely on this pipeline. So the headline numbers are not currently supported. The qualitative direction probably survives — dynamic estimators really do miss overhead like cooling and PSU losses — but the paper's own limitations section does not acknowledge this metrological gap.\n\nMinor issues: CodeCarbon 3.0.1 is used but the citation is to v2.4.1, and the static TDP values are taken from the tools' defaults, which is fine for validation but should be stated more explicitly.\n\nWho gets value: people doing carbon accounting for on-premise AI inference, and the maintainers of CodeCarbon and the ML Impact Calculator. It deserves a serious referee. The revision needs a calibration appendix: report the meter's resolution, test it against a known load, give OCR accuracy, or explicitly downgrade the quantitative claims to qualitative ones. As it stands, the dataset is worth knowing about, but the headline numbers are not citable.","headline":"A genuinely useful first-of-its-kind validation dataset for CodeCarbon and static energy estimators, but the headline error percentages rest on an uncalibrated plug-meter/OCR pipeline whose resolution is never reported.","tokens_in":12852,"tokens_out":2770,"would_cite":true,"duration_ms":27042,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CodeCarbon and the ML Emissions Calculator misreport AI energy use, with dynamic estimates consistently low by 20–30% and static estimates off by up to 40%.","keywords":["energy consumption estimation","CodeCarbon","ML Emissions Calculator","sustainable AI","carbon footprint accounting","inference energy measurement","ground-truth validation","resource-aware machine learning"],"falsifier":"Run the same 50-model battery on the same workstation while logging wall power with a calibrated power analyzer sampling at least once per second, and compare the analyzer's integrated kWh against the plug-in meter display readings; if the display readings differ from the analyzer by less than the claimed 20–40% error bands, the percentages lose their ground, and if CodeCarbon's dynamic estimates match the analyzer, the underestimation claim fails.","tokens_in":1539,"feed_emoji":"⚡","tokens_out":2119,"duration_ms":67657,"temperature":0.7,"pith_summary":"The paper sets out to answer a simple question: when CodeCarbon and the ML Emissions Calculator report the energy used by an AI model, how close are those numbers to what the hardware actually draws at the wall? By running 30 vision classifiers and 20 large language models on a single workstation, reading a plug-in energy meter with a camera and OCR as ground truth, and comparing those readings with the two estimation tools across hundreds of runs, the paper finds that both tools follow the general shape of energy demand but miss by large margins. Static estimates under- or overestimate consumption by up to 40 percent, while dynamic estimates consistently undercount by 20–30 percent. If correct, published carbon-footprint figures for local AI inference are mostly understated, and the error grows with model size, runtime, and CPU-only deployment.","feed_headline":"AI energy tools miss real power use by up to 40 percent","feed_subtitle":"Wall-meter tests show CodeCarbon undercounts inference energy by 20–30%; static estimates swing both ways.","key_machinery":"The load-bearing machinery is a three-way comparison loop run on the same inference jobs. Ground truth is obtained externally: a plug-in energy meter between wall and workstation, photographed by a camera, with the digits decoded by a custom random-forest OCR classifier. Static estimation uses the ML Impact Calculator formula $\\text{Energy} = \\text{Power} \\times \\text{Time}$ with fixed TDP values (300 W GPU, 125 W CPU); dynamic estimation uses CodeCarbon 3.0.1 to sample CPU/GPU power at intervals and sum $\\text{Power}_t \\cdot \\Delta t$. The paper's numbers all ride on the difference between those two estimates and the meter readings.","core_discovery":"On the paper's own terms, the discovery is that the energy numbers produced by the two most widely used estimation tools are not reliable as quantitative accounts even though they are reliable as qualitative trends. CodeCarbon's dynamic profiling captures CPU/GPU power but omits the power supply, cooling, and peripherals, so it consistently reports less than the meter sees; the ML Impact Calculator's static TDP assumption overshoots small models, undershoots large ones, and lands near the truth only for mid-sized models. The measured maximum draw, 534 W for a 24-billion-parameter model, is nearly double the static estimate's 300 W assumption. The paper frames this as a validation framework: any practitioner can reproduce the three-way comparison with an off-the-shelf meter, a camera, and OCR.","pith_inferences":["Because the dynamic underestimation is consistent across nearly all 50 models, a simple multiplicative correction of order 1.25–1.4 might bring CodeCarbon's local-inference numbers much closer to measured draw without changing its architecture.","The same meter-and-camera validation could be extended to training runs and to multi-GPU or data-center settings, where outlet-level power distribution units would replace the plug-in meter; the 20–30% and 40% figures would likely shift, but the direction of dynamic underestimation may persist.","A direct test of the paper's mechanism: if cooling and power-supply overhead are the cause, then a workstation with a more efficient PSU and better cooling should show smaller CodeCarbon underestimation; that is a prediction the paper does not test."],"forward_implications":["Carbon-footprint reports that rely on CodeCarbon for local inference are likely too low by roughly a fifth to a third, not by a rounding error.","Static estimates like the ML Impact Calculator are not uniformly wrong: they are most accurate for mid-sized models and systematically off for small and large ones, so users should match the tool to model size or apply correction factors.","Estimation error scales with model size and with how long a model runs, making longer deployments the ones most in need of measurement.","CPU-only inference is where both tools fail worst; the paper's largest CPU dynamic error was hundreds of times the GPU error for the same model.","Hyperparameters that affect utilization, such as batch size and sampling temperature, change both the true energy draw and the size of estimator error, so energy reports should state those settings."],"supporting_citations":[{"why":"The dynamic estimator whose per-run reports are compared to meter readings; the paper's 20–30% underestimation claim is about this tool.","marker":"[10]"},{"why":"Source of the static TDP assumptions (300 W GPU) and the Power × Time formula used for static estimates.","marker":"[38]"},{"why":"Provides the images classified by the 30 vision models during the 2-minute inference runs.","marker":"[12]"},{"why":"Provides the prompts answered by the 20 language models during 15-minute runs.","marker":"[39]"},{"why":"Introduced the dynamic software-profiling approach (experiment-impact-tracker) that CodeCarbon refines, defining what dynamic estimation means here.","marker":"[24]"},{"why":"Cited evidence that cooling can exceed 10% of AI power draw, the kind of neglected component that explains part of the gap.","marker":"[64]"}],"fun_headline_variants":["AI energy estimates off by up to 40%, wall meter shows","Wall meters reveal AI energy tools miss by up to 40%","AI power tools undercount real usage by 40% at worst","Energy tools for AI often wrong by 40%: validation study"],"cache_read_input_tokens":15104,"weakest_assumption_plain":"The comparison assumes the plug-in energy meter plus camera/OCR pipeline yields true energy consumption, yet the paper does not state the meter's resolution or calibration or the OCR classifier's accuracy, and for short high-power runs the per-run energy total can sit near the meter's display step.","fun_headline_variants_meta":{"raw":{"variants":["AI energy estimates off by up to 40%, wall meter shows","Wall meters reveal AI energy tools miss by up to 40%","AI power tools undercount real usage by 40% at worst","Energy tools for AI often wrong by 40%: validation study"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000399,"raw_usage":{"total_tokens":2065,"prompt_tokens":905,"completion_tokens":1160,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":1085}},"tokens_in":521,"tokens_out":1160,"duration_ms":7985,"temperature":1.0,"reasoning_tokens":1085,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:44:15.869178+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 50-model battery on the same workstation while logging wall power with a calibrated power analyzer sampling at least once per second, and compare the analyzer's integrated kWh against the plug-in meter display readings; if the display readings differ from the analyzer by less than the claimed 20–40% error bands, the percentages lose their ground, and if CodeCarbon's dynamic estimates match the analyzer, the underestimation claim fails.","supporting_citations":[{"cited_title":"Quantifying the Carbon Emis- sions of Machine Learning","cited_arxiv_id":null,"evidence_quote":"Source of the static TDP assumptions (300 W GPU) and the Power × Time formula used for static estimates."},{"cited_title":"2023.URL:https : / / huggingface","cited_arxiv_id":null,"evidence_quote":"Provides the prompts answered by the 20 language models during 15-minute runs."}],"review_version":2}