{"id":"5f3ab77e-8505-4690-b591-4b63d953ebbd","arxiv_id":"2412.00329","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Larger language models do not reliably deliver higher accuracy on software tasks, and quantized large models can outperform full-precision medium models on energy and accuracy together.","lead":"This study measures how much electricity 54 versions of open-source coding assistants use and how often they solve four coding tasks on two types of GPUs. It finds that bigger models often burn extra energy without being more accurate, and that compressed large models can beat full-size medium models on both energy and accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Energy comparisons depend on an unstated idle-power subtraction; if absent, the reported Wh values and Pareto fronts are systematically biased.","rationale":"The reader's weakest assumption correctly identifies the most load-bearing concern: the energy numbers that drive every comparison and Pareto frontier depend on whether idle power was subtracted, and the paper leaves this implicit. This is not an accusation of misconduct; it is a reproducibility gap in the measurement methodology. The concern is substantive because the A100 idle draw is large relative to the reported task energies, so failure to subtract would systematically inflate the energy of longer-running, larger models and could alter the paper's central energy-accuracy conclusions. The proposed check -- inspecting the replication package and recomputing a few representative entries with and without idle subtraction -- would settle the issue. I did not find a more significant concern. The accuracy estimates lack confidence intervals, but the core claim of diminishing returns and non-monotonic accuracy-vs-size is supported by the Spearman correlation analysis (which applies Bonferroni correction) and by multiple large energy differences, so the idle-power ambiguity is the primary weakness. The paper has real strengths: a public replication package, three repeated runs per configuration, a broad model family sample, and an honest threats-to-validity section. Given the ambiguity, the conditional verdict is appropriate; if the replication scripts show idle subtraction is correctly performed, the main conclusions are likely to hold.","tokens_in":19990,"tokens_out":6691,"duration_ms":65850,"concrete_test":"Inspect the replication package's measurement scripts and determine whether the reported energy is computed as sum((power_sample - idle_power) * dt) or simply sum(power_sample * dt). If idle subtraction is absent, recompute the 'Total energy (Wh)' column of Table II for at least three representative models spanning the size range (e.g., gemma:2b-q4, phi3:3.8b-q8, granite-code:20b-fp16) after subtracting the measured idle power, and re-plot the Figure 4 Pareto fronts for all four tasks. If the Pareto-front membership changes, or if the qualitative conclusion about large versus small model energy changes, the central claim requires qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV-B reports idle power values for both GPUs (9.92W for RTX 3070, 46.82W for A100) but never explicitly states that these baselines were subtracted from the Wh values used in Figures 2-4 and Table II. The text says only that energy is 'determined as the product of time and average power' and separately that idle power was measured. If the reported Wh values include the idle draw over the entire job duration, then every energy value contains an additive component proportional to runtime. Because larger and full-precision models run longer, this component inflates their energy disproportionately, potentially creating or exaggerating the pattern that larger models have a much greater energy footprint and shifting which models lie on the Pareto frontier. The A100 idle draw of 46.82W is not negligible: a 30-minute run contributes about 23.4Wh of idle energy, which is the same order of magnitude as many reported task energies. The paper's central claim that quantized large models can dominate full-precision medium models on both energy and accuracy depends on precise relative energy values. Even if the authors did subtract idle power, the methodology section does not say so, and the replication package must be checked to confirm. This is a concrete, testable reproducibility gap rather than a fundamental flaw, but it is load-bearing because the energy axis is the basis of every comparison and Pareto-front analysis.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript reports an experimental measurement study of inference energy use and task accuracy for 18 open-LLM families, in full-precision and GGUF-quantized variants, on four HumanEvalPack Python tasks (code generation, bug fixing, docstring generation, and test generation). Experiments were run on two infrastructures, an NVIDIA A100 and an RTX 3070, with GPU power sampled at 10 Hz via pyNVML and CPU energy via RAPL; each configuration was run three times and averaged. The central descriptive finding is that accuracy and energy do not always trade off: larger models often consume substantially more energy without commensurate accuracy gains, quantized versions of large models can dominate full-precision medium models on both axes, and the best model varies by task.","tokens_in":20232,"tokens_out":5375,"duration_ms":52769,"significance":"If the measurements are sound, this is a useful empirical contribution to green-AI and model-selection research for software engineering. Its strengths include the breadth of models (18 families, 54 variants), fixed hyperparameters, test-execution-based evaluation for three of the four tasks, a replication package, and falsifiable Pareto-front and correlation claims. The paper also makes a practical point that task-specific model selection matters. However, because the central claims rest on the energy axis and on the docstring-accuracy axis, the unclear idle-power accounting, missing uncertainty reporting, and unvalidated LLM-as-judge step are material. The findings are plausible in direction but need measurement-reporting revisions before they can be accepted.","major_comments":[{"comment":"The methodology reports idle power values (9.92 W for the RTX 3070, 46.82 W for the A100, and 1.92 W for the CPU) but never explicitly states whether these baselines were subtracted from the Wh values used in Figures 2-4 and Table II. Since total energy is computed as time times average power, including idle draw adds a component proportional to runtime. Because longer-running full-precision and larger models would receive a larger additive bias, this can change relative energy rankings and Pareto-front membership; for a 30-minute A100 run the idle contribution alone is about 23.4 Wh, which is the same order as many reported task energies. Please state the exact computation, include the subtraction in the replication code, and, if idle power was not subtracted, re-run the analysis. The omission should also be acknowledged in Section VI.","section":"Section IV-B"},{"comment":"No variance or uncertainty information accompanies the three-run averages. Many comparisons in Section V-B are close, such as the roughly 2% energy differences between some q4 and q8 variants or the 4.27% versus 3.05% accuracy improvements cited for phi3 and starcoder2, and without confidence intervals it is not possible to determine whether the reported orderings and Pareto-front memberships are stable. Please report per-run spread, standard deviations, or a sensitivity analysis for at least the Pareto-front comparisons. This is load-bearing because the central 'dominates' claims depend on relative energy and accuracy values.","section":"Section IV-B / Section V-B"},{"comment":"Docstring accuracy is evaluated by having GPT-4o-mini generate code from the generated docstring and then applying pass@1, but the manuscript does not validate this judge against ground truth, a manual sample, or an alternative judge. Because docstring generation is one of the four tasks used in the task-level conclusions, a systematic judge bias could affect which models lie on the Pareto front in Figure 4(c). Please add a validation subset, an agreement measure, or a sensitivity comparison with another judge, and report the results.","section":"Section IV-D"},{"comment":"The caption states that data points exceeding the y-axis limit are not displayed. A Pareto-front plot is only correct if all non-dominated points are shown; hiding points outside the axis range can omit points that belong on the frontier. Please either include all points using an inset or broken axis, or explicitly report that all omitted points are dominated by displayed points and identify them.","section":"Figure 4"}],"minor_comments":[{"comment":"Table II is difficult to parse as rendered: the header groups for Accuracy, C1, C2, and Efficiency do not align unambiguously with the numeric fields. Please reformat the table or split it into separate tables so each task column is clearly identified.","section":"Table II"},{"comment":"The text states that CPU energy accounts for approximately 16% of total energy on the RTX 3070, but it does not specify whether the CPU idle baseline of 1.92 W was subtracted. This should be clarified in the methodology alongside the GPU idle-power treatment.","section":"Section V-A"},{"comment":"The Pareto-front figures do not label their axes. Adding explicit axis labels for energy (Wh) and accuracy (%) would make the plots self-contained.","section":"Figure 4"},{"comment":"The claim that the 4-bit variant had the lowest energy usage in 212 out of 216 analyzed cases needs a clear definition of the denominator; please state that this corresponds to 54 model variants across four tasks and list the exceptions.","section":"Section V-C"},{"comment":"The threats-to-validity section is candid about prompt templates, incomplete Ollama responses, code extraction, and dataset coverage, but it does not discuss the idle-power accounting or the uncertainty of the energy measurements; these should be added given their importance to the central claims.","section":"Section VI"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a software-engineering or green-AI venue and the authors are appropriately building on their prior measurement work; I saw no circularity issue. The main risk is that the energy numbers, which drive every comparison, are not yet fully documented. The replication package should be checked during revision to confirm whether idle power was subtracted in the scripts."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look: this is the broadest comparison I know of that puts energy and accuracy side by side for LLMs in coding tasks. Eighteen model families, three precision formats, four tasks, two GPUs—that measurement matrix is a legitimate new contribution. The headline finding, that spending more energy doesn't reliably buy more accuracy and that quantized large models can dominate full-precision medium ones on both axes, is supported by the displayed Pareto fronts and Table II. The authors also did the right things: fixed hyperparameters, fixed seed, isolated GPU, three runs, public replication package. The tokens/J efficiency metric in Section V-C is a useful normalization.\n\nNow the soft spots. The load-bearing issue is in Section IV-B. They report idle powers for both GPUs (9.92W and 46.82W) but never explicitly state that these baselines were subtracted from the active Wh values. If idle power was not subtracted, every energy number carries an additive component proportional to runtime. Larger and full-precision models run longer, so that component would inflate their energy disproportionately and could create or exaggerate the paper's central pattern. On the A100, 46.82W over 30 minutes is about 23Wh, the same order as many reported task totals. The text says idle power was measured 'to measure the energy usage of the hardware and OS when idle,' which suggests subtraction was intended, but the methodology never says so. The replication package should settle this, but it was not inspected here. That's a concrete reproducibility gap, not a fundamental flaw.\n\nMinor concerns: three-run averages come without variance; docstring accuracy depends on GPT-4o-mini as judge without validation; the model sample is a convenience sample capped at 20B parameters, using only Python and HumanEvalPack. The authors acknowledge the last two in Threats to Validity, which is honest. The central descriptive claim likely survives the minor issues; the idle-power subtraction is the one thing that could move the Pareto fronts.\n\nBottom line: this deserves a serious referee. A revision that states the idle-power subtraction explicitly, reports per-run data or error bars, and validates the judge would make it solid. I'd cite it as a benchmark in green-software work. Send it to review.","headline":"Useful measurement matrix on LLM energy/accuracy trade-offs for coding tasks; main caveat is an unstated idle-power subtraction that needs checking before the numbers are trusted.","tokens_in":20740,"tokens_out":2457,"would_cite":true,"duration_ms":21391,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper measures inference energy and accuracy for 18 language-model families across four software development tasks and two GPUs.","keywords":["energy efficiency","LLM inference","model quantization","GGUF","HumanEvalPack","Pareto front","software development tasks","pass@1"],"falsifier":"Re-run the benchmark on the same GPUs using a wall-plug power meter or a sampling rate above 10 Hz, and verify whether subtracting the reported idle-power baselines changes the per-model Wh values; if the Pareto fronts shift materially, the conclusion that bigger models do not buy accuracy would need qualification.","tokens_in":19813,"feed_emoji":"⚡","tokens_out":3721,"duration_ms":32376,"temperature":0.7,"pith_summary":"This paper asks whether spending more energy on a locally deployed language model buys better results on everyday software development tasks. Across 18 model families, four tasks (code generation, bug fixing, docstring generation, test generation), full-precision and quantized variants, and two GPUs, it measures both inference energy and pass@1 accuracy. The central finding is that the energy-accuracy trade-off is not strict: larger models usually draw far more power while performing about the same as, or worse than, smaller models, and quantized versions of large models often beat full-precision medium models on both energy and accuracy. The paper concludes that no single model is best for all tasks, so model selection should be task-specific and informed by measured trade-offs. If right, this gives developers a practical argument for choosing smaller or quantized local models instead of assuming bigger is better.","feed_headline":"Bigger coding LLMs don't reliably buy accuracy","feed_subtitle":"A 4-task, 18-model benchmark on two GPUs finds quantized small models can dominate full-precision giants.","key_machinery":"The quantitative backbone is a controlled measurement setup: GPU power sampled at 10 Hz with pyNVML, Intel RAPL for CPU energy, fixed hyperparameters (temperature 0.1, top-p 0.95), and identical prompts across models on the Python subset of HumanEvalPack. Accuracy is pass@1 against the benchmark's test assertions, docstrings judged by GPT-4o-mini, and the energy-accuracy tension is visualized with Pareto fronts. The architecture-efficiency link is quantified with Spearman correlations (Bonferroni-corrected) between model characteristics and metrics like tokens per joule.","core_discovery":"The paper claims that in software-development inference, higher energy expenditure does not reliably translate into higher accuracy, and in several cases a quantized small model dominates a full-precision larger one on both objectives simultaneously. It shows this with Pareto fronts per task: the frontier models are often not the largest, and general-purpose models like gemma:2b appear near the ideal point for docstring and test generation. It also reports that tokens per joule is nearly constant for a given model across tasks, so a model's efficiency can be estimated from its architecture plus expected output length.","pith_inferences":["The finding implies that energy-aware model selection could be offered as a service by model registries, presenting Pareto fronts per task so users can pick a model from measured data rather than from reputation or size.","The strong correlation between architecture size and tokens per joule suggests that energy estimates could be derived from model-card metadata alone, before running any benchmark, which would make energy-aware selection far cheaper to apply at scale.","A testable extension is to run the same 18 models on a CPU-only machine, since the paper's measurements show CPU energy is non-negligible and quantized large models might behave differently without a GPU.","The paper's null result on accuracy versus parameter count is specific to HumanEvalPack; extending to more complex, multi-file programming tasks could change the Pareto fronts and should be checked before generalizing the advice."],"forward_implications":["Model selection for local LLM deployment should be driven by the target task and by measured energy-accuracy trade-offs, not by parameter count alone.","Quantized large models are often the best compromise when a single model must serve several tasks, since they can beat full-precision medium models on both energy and accuracy.","The tokens-per-joule metric allows estimating a model's inference energy for a new task from its architecture and expected output length, before running a full benchmark.","Code-specific models dominate only in code generation; fine-tuning them for docstring and bug-fixing tasks is a promising direction for improving both accuracy and efficiency.","Energy benchmarking of inference should include CPU energy, which accounted for roughly 16% of total energy on the tested laptop GPU setup."],"supporting_citations":[{"why":"Supplies the four-task benchmark (HumanEvalPack) and the prompt format used for all accuracy and energy measurements.","marker":"[20]"},{"why":"Defines the pass@k metric and the HumanEval benchmark lineage that the correctness evaluation relies on.","marker":"[21]"},{"why":"Supports the choice of GGUF as the energy-efficient quantization format for the quantized model variants.","marker":"[16]"},{"why":"Motivates the 4- and 8-bit quantization bit widths by showing that lower bit widths significantly degrade performance.","marker":"[14]"},{"why":"Provides the prior method for measuring LLaMA inference energy that this study extends to software development tasks.","marker":"[31]"},{"why":"Frames inference energy as a significant long-term cost for deployed LLMs, motivating the study's focus on inference rather than training.","marker":"[7]"}],"fun_headline_variants":["Quantized coding LLMs can outdo full-precision giants","Coding LLM accuracy isn't tied to energy spend","Small quantized beats big full-precision for coding LLMs","High-energy coding LLMs don't reliably improve accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All comparisons rest on the measured watt-hours, which depend on the 10 Hz GPU power sampling and on how the reported idle powers (9.92 W for the RTX 3070, 46.82 W for the A100) were treated relative to the active measurements.","fun_headline_variants_meta":{"raw":{"variants":["Quantized coding LLMs can outdo full-precision giants","Coding LLM accuracy isn't tied to energy spend","Small quantized beats big full-precision for coding LLMs","High-energy coding LLMs don't reliably improve accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00021,"raw_usage":{"total_tokens":1360,"prompt_tokens":844,"completion_tokens":516,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":460,"completion_tokens_details":{"reasoning_tokens":448}},"tokens_in":460,"tokens_out":516,"duration_ms":5128,"temperature":1.0,"reasoning_tokens":448,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:29:29.275903+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the benchmark on the same GPUs using a wall-plug power meter or a sampling rate above 10 Hz, and verify whether subtracting the reported idle-power baselines changes the per-model Wh values; if the Pareto fronts shift materially, the conclusion that bigger models do not buy accuracy would need qualification.","supporting_citations":[{"cited_title":"Benchmarking emerging deep learning quantization methods for energy efficiency,","cited_arxiv_id":null,"evidence_quote":"Supports the choice of GGUF as the energy-efficient quantization format for the quantized model variants."},{"cited_title":"A comprehensive evaluation of quantization strategies for large language models,","cited_arxiv_id":null,"evidence_quote":"Motivates the 4- and 8-bit quantization bit widths by showing that lower bit widths significantly degrade performance."},{"cited_title":"From words to watts: Benchmarking the energy costs of large language model inference,","cited_arxiv_id":null,"evidence_quote":"Provides the prior method for measuring LLaMA inference energy that this study extends to software development tasks."}],"review_version":1}