{"id":"47c6f774-208c-4fb8-a327-2230e4323d1e","arxiv_id":"2505.02346","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On seven single-threaded Computer Language Benchmarks Game programs with no third-party libraries, Codon, PyPy, and Numba cut average execution time and energy by roughly 86 to 95% compared with CPython, while Nuitka reduced memory most consistently.","lead":"Seven benchmark programs written in Python were compiled with eight different compilers and run on two machines to measure time, energy, memory, and cache misses against the standard interpreter. Codon, PyPy, and Numba delivered the largest speedups and energy savings, while the other compilers had smaller, benchmark-dependent effects.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's 'over 90%' claim is contradicted by Section 6 (PyPy/Numba are ~86%), and the Section 6 'overall' averages are not reproducible from Table 2, indicating an unspecified per-benchmark averaging over seven compiler-friendly CLBG kernels.","rationale":"The reader's verdict is CONDITIONAL, and this stress test supports that conditionality rather than overturning it. The reader correctly noted in the strongest_claim that the abstract overstates PyPy/Numba improvements, and in the rationale that aggregating heterogeneous benchmarks can inflate significance. However, the reader's stated weakest_assumption is benchmark representativeness, whereas the sharpest fixable problem is that the paper's own 'overall' numbers are not reproducible from its Table 2 and the aggregation method is unspecified. This is a concrete internal-validity flaw, not merely an external-generalization caveat, and it can be checked immediately from the replication package. If the reanalysis shows the headline changes under a defensible weighting, the paper's main quantitative contribution needs revision; if not, the current CONDITIONAL verdict stands with the representativeness caveat. Either way, no change to the verdict is needed beyond what the reader already recommended.","tokens_in":18204,"tokens_out":13366,"duration_ms":161441,"concrete_test":"From the replication package raw data, recompute the overall time and energy improvements for PyPy, Numba, and Codon under three definitions: (1) unweighted mean of the seven per-benchmark improvement percentages, which Section 6 appears to use; (2) pooled aggregate improvement from all runs, which Table 2 means imply; and (3) a workload-weighted average weighting each benchmark by CPython execution time. Compare all three against the abstract's 'over 90%' claim and the Section 6 values. If PyPy and Numba fall below 90% under every defensible definition, the abstract is wrong; if the Section 6 numbers only match definition (1), the paper must state that and report confidence intervals, because the current text invites readers to interpret the improvements as applying to pooled or typical workloads.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim is the abstract's statement that Codon, PyPy, and Numba 'achiev[e] over 90% speed and energy improvements.' Section 6 reports 94.18%/94.66% for Codon but only 86.67%/86.89% for PyPy and 85.86%/86.61% for Numba, so the 'over 90%' headline is false for two of the three compilers. More importantly, the Section 6 averages cannot be reproduced from the paper's own Table 2. Using the server means in Table 2, the pooled improvement for PyPy is (24.98-2.36)/24.98 = 90.5% in time and (16.41-1.54)/16.41 = 90.6% in energy, not 86.67%/86.89%. The reported numbers therefore appear to be unweighted means of the seven per-benchmark improvement percentages, while the significance analysis in Section 5 pools all runs across benchmarks, treating benchmark as a hidden factor, for Kruskal-Wallis and Cliff's Delta. This mismatch means the 'large effect size' is computed on a different quantity than the headline percentages. Because the seven CLBG kernels are all single-threaded, library-free, numeric workloads that were modified for Numba/Codon typing, equal-weight averaging over them can overstate the improvement a realistic mixed Python workload would receive. The authors acknowledge the benchmark-selection limitation in Sections 7.2-7.3, but the aggregation inconsistency that produces the headline number is not acknowledged.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a controlled empirical comparison of eight Python compilers (PyPy, Numba, Nuitka, Mypyc, Codon, Cython, Pyston-lite, and the experimental Python 3.13 JIT) against CPython on seven single-threaded, library-free benchmarks drawn from the Computer Language Benchmarks Game. The experiments were run on two testbeds (a server and an Intel NUC) with fixed CPU frequency and single-core execution, measuring energy consumption, execution time, memory usage, and LLC miss rate. The paper claims that compilation significantly improves execution time, energy, and memory usage, with Codon, PyPy, and Numba achieving very large speedups and energy savings, and that Nuitka consistently improves memory usage. It also reports that LLC miss behavior is inconsistent across benchmarks and compilers. A replication package is provided.","tokens_in":18493,"tokens_out":7710,"duration_ms":79513,"significance":"This is a useful and generally well-controlled empirical study. Its strengths include fixed CPU frequency, single-core execution with Hyper-Threading disabled, two distinct testbeds, randomized run order, warm-up and cool-down phases, 1575 total runs, a mix of JIT and AOT compilers, and a replication package. If the results are corrected and confirmed, the paper provides solid evidence that Codon, PyPy, and Numba deliver large and consistent execution-time and energy improvements for single-threaded numerical Python kernels, that Nuitka tends to reduce memory usage, and that LLC miss behavior is compiler- and benchmark-dependent. The study is not circular: it compares measured behavior against an external CPython baseline and does not fit models to its own conclusions. However, the headline quantitative claims and several internal statistics need reconciliation before the findings can be considered fully reliable.","major_comments":[{"comment":"The abstract and Section 8 state that Codon, PyPy, and Numba achieve 'over 90% speed and energy improvements,' but Sections 6.1 and 6.2 report server averages of 86.89%/86.67% for PyPy and 86.61%/85.86% for Numba. Moreover, these Section 6 percentages cannot be reproduced from Table 2: using the server means in Table 2, the pooled improvement for PyPy is (24.98 - 2.36)/24.98 = 90.5% in time and (16.41 - 1.54)/16.41 = 90.6% in energy, not 86.67%/86.89%. The reported numbers appear to be unweighted means of per-benchmark improvement percentages, whereas the significance analysis in Section 5 pools all runs across benchmarks. The paper should define the aggregation rule, make the headline consistent with it, and restrict the 'over 90%' claim to Codon or qualify it.","section":"Abstract, Section 6.1, Section 6.2, Table 2"},{"comment":"The memory results are internally contradictory. Section 5.2.2 states that Cliff's Delta shows 'a small positive impact for both Nuitka and Codon' on the server, while Section 6.3 states that Nuitka's memory improvement is 'confirmed by Cliff's Delta test indicating a large effect size on the server' and that Codon 'also shows a large effect size on the server.' These cannot both be correct. Because the abstract's memory claim ('Nuitka optimizes memory usage consistently') rests on this analysis, the discrepancy must be resolved and the effect sizes reported consistently.","section":"Sections 5.2.2 and 6.3"},{"comment":"The statistical analysis pools all runs across the seven benchmarks before applying Kruskal-Wallis and Cliff's Delta. Since the benchmarks differ by orders of magnitude in energy and execution time (Table 2), the reported Cliff's Delta of 1.0 reflects between-benchmark separation as much as within-benchmark compiler effects. A blocked or per-benchmark analysis (e.g., per-benchmark Cliff's Delta, or normalization per benchmark before pooling) is needed to support the 'large effect size' conclusion and to align with the per-benchmark percentages used in Section 6.","section":"Sections 5.1, 5.2.1, 6.1, 6.2"},{"comment":"The headline conclusions are stated more broadly than the experimental scope. Section 4.1 explains that the CLBG code was modified for Numba and Codon compatibility (class types, printing, @njit decorators), and Sections 7.2 and 7.3 acknowledge that code characteristics and the exclusion of third-party libraries and parallelism may favor these compilers. The abstract and Section 8 should scope the claims to single-threaded, library-free, compute-intensive Python kernels rather than presenting the improvements as general Python compilation benefits.","section":"Section 4.1, Abstract, Section 8"}],"minor_comments":[{"comment":"The heading uses 'LCC miss percentage' instead of 'LLC miss percentage'.","section":"Section 6.4"},{"comment":"The caption mentions two shades for NUC versus server, but the figure does not define which shade is which; a legend or footnote should be added.","section":"Figure 2b"},{"comment":"The phrase 'increase energy consumption of 47.96%, -68.65%, -54.98%' mixes positive and negative signs without defining the sign convention; rephrase as 'change by +47.96%, -68.65%, -54.98%' or similar.","section":"Section 6.1"},{"comment":"Compiler names are inconsistent (e.g., 'MyPyC' in Table 1 versus 'Mypyc' in the text and figures); standardize the naming.","section":"Table 1 and throughout"},{"comment":"The conclusion's phrase 'over 90% improvement on the majority of benchmarks' should be accompanied by the appropriate per-benchmark figure or table reference so readers can verify it against Figure 3.","section":"Section 8"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of EASE and includes a replication package. The issues identified are internal inconsistencies and reporting/aggregation problems rather than fundamental flaws in the experimental design; I believe they are fixable in a revision. I would not reject on the current evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one. It's a controlled empirical comparison of eight Python compilers against CPython on seven CLBG benchmarks, measuring time, energy, memory, and LLC misses on two testbeds, with CPU frequency fixed and single-core execution. There's a replication package. That alone puts it above most papers in this space.\n\nWhat's genuinely new: the breadth of compilers (PyPy, Numba, Nuitka, Cython, Mypyc, Codon, Pyston-lite, Python 3.13 JIT) and the combination of energy and cache behavior under controlled conditions. Prior work compared subsets or missed energy, so this is a useful dataset. The two-testbed consistency for time and energy is reassuring. The Nuitka memory result is a nice, counterintuitive finding.\n\nThe central result — Codon, PyPy, and Numba deliver large, consistent efficiency gains on these workloads — is credible and supported by the per-benchmark figures. I don't doubt that.\n\nBut there are soft spots. The abstract claims all three 'achiev[e] over 90% speed and energy improvements.' Section 6 says Codon is 94.18%/94.66%, but PyPy is 86.67%/86.89% and Numba is 85.86%/86.61%. So the abstract is simply wrong for two of the three compilers. On top of that, the Section 6 averages are not reproducible from Table 2: pooling the server means gives PyPy about 90.5% improvement, so the reported 86.67% must be an unweighted average of per-benchmark percentages. That's a legitimate choice, but the paper never says so, and the significance analysis appears to pool all runs across benchmarks for each compiler, treating benchmark as a hidden factor. Cliff's Delta of 1.0 for the big three is an artifact of pooling non-overlapping scales; it doesn't add much beyond the descriptive stats.\n\nThe benchmark selection is also narrow — seven compute-heavy, single-threaded kernels from CLBG, modified for Numba/Codon typing. The authors acknowledge this in the threats, and it limits generalization to typical Python code. The abstract and conclusion should qualify accordingly.\n\nOverall: a useful, honest empirical contribution that needs a corrected abstract, a stated averaging rule, and a clearer statistical description. Who's it for? Practitioners working on numerical, single-threaded Python, and green-software researchers looking for a controlled energy dataset.\n\nI'd send it to peer review — it deserves serious referee attention — but with the expectation of moderate revisions. The core finding will survive; the presentation needs fixing.","headline":"Solid, controlled comparison of eight Python compilers, but the abstract overstates PyPy/Numba improvements and the averaging is unclear; the core finding of large gains from Codon, PyPy, and Numba is credible.","tokens_in":19038,"tokens_out":3344,"would_cite":true,"duration_ms":35775,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Compiling Python with Codon, PyPy, or Numba cuts single-threaded execution time and energy use by roughly 85–94 percent compared with the standard CPython interpreter, while Nuitka reliably reduces memory use.","keywords":["Python compilation","energy efficiency","execution time","just-in-time compilation","ahead-of-time compilation","Last-Level Cache misses","benchmarking","CPython"],"falsifier":"Run the same fixed-frequency, single-core protocol on a set of unmodified Python programs that use dictionaries, strings, file I/O, and popular numerical libraries, comparing CPython against Codon, PyPy, and Numba; if the average execution-time or energy improvements over CPython fall well below the 85–94% range, or reverse on several programs, the headline gains do not generalize beyond the benchmark set.","tokens_in":17980,"feed_emoji":"⚡","tokens_out":14516,"duration_ms":157397,"temperature":0.7,"pith_summary":"This paper asks whether compiling Python code delivers the large efficiency gains that compiler projects advertise, and it answers with a controlled measurement on two testbeds. Eight compilers—PyPy, Numba, Nuitka, Mypyc, Codon, Cython, Pyston-lite, and the experimental Python 3.13 JIT—are each run on seven single-threaded, library-free benchmarks drawn from a standard language-benchmark suite, with CPU frequency fixed, Hyper-Threading disabled, all work pinned to one core, and 1,575 total executions compared against CPython. The central finding is that compilation can substantially cut both execution time and energy use: Codon improved server execution time by 94.18% and energy by 94.66%, PyPy by 86.67% and 86.89%, and Numba by 85.86% and 86.61%, while Nuitka consistently lowered memory usage on both testbeds. The study matters because it gives practitioners a measured, machine-controlled basis for choosing a Python compiler, and it shows that the choice of compiler, not just the choice of language, determines how much energy a Python workload consumes.","feed_headline":"Codon, PyPy and Numba make Python up to 94% faster","feed_subtitle":"Across seven compute benchmarks, they cut energy use almost as much, beating CPython by 85–94 percent.","key_machinery":"The quantitative claims rest on simple ratios: percentage improvement is $(\\bar{x}_{\\mathrm{CPython}} - \\bar{x}_{\\mathrm{compiler}})/\\bar{x}_{\\mathrm{CPython}}$ and speedup is $\\bar{x}_{\\mathrm{CPython}}/\\bar{x}_{\\mathrm{compiler}}$, applied to execution time, energy, memory, and LLC miss percentages. Feeding those ratios is a full-factorial experiment: 9 execution modes (8 compilers plus CPython) × 7 benchmarks × 10–15 repetitions per treatment on each of two testbeds, with CPU frequency fixed, Hyper-Threading disabled, all commands pinned to one core, a two-minute warm-up and cooldown, and energy, memory, and cache measurements collected by hardware counters and OS timing. The machinery's job is to make the CPython-versus-compiler comparison clean enough that the large effect sizes can be attributed to compilation rather than to machine state, core count, or frequency scaling.","core_discovery":"The paper's central claim is that for single-threaded, library-free Python code of the kind found in standard language-benchmark suites, compiling the code instead of running it in CPython produces large, consistent improvements in execution time and energy consumption, and that the winning compilers are Codon, PyPy, and Numba. On the server, Codon's average execution-time improvement over CPython is 94.18% and its energy improvement is 94.66%; PyPy improves time by 86.67% and energy by 86.89%; Numba improves time by 85.86% and energy by 86.61%. Effect sizes are large (Cliff's $\\Delta$ = 1.0) for these three against CPython, while the other five compilers show small or negligible effects on time and energy. The paper further claims that Nuitka reliably reduces memory usage on both testbeds, that no compiler reliably lowers the LLC miss rate, and that execution time and energy are strongly correlated in this setting, so the speedups translate almost directly into energy savings. It also reports a counterexample: on n_body, the AOT compilers Nuitka, Cython, and Mypyc make execution slower and more energy-hungry than CPython.","pith_inferences":["The 85–94% figures are likely an upper bound for general Python: the benchmark suite is numerical and library-free, conditions that favor static typing and tight loops, so I would expect smaller gains on I/O-heavy, library-heavy, or dynamically typed application code. The paper itself flags external validity as a limitation.","The strong time–energy correlation suggests that, under fixed frequency and single-core execution, energy use could be approximated from runtime alone, letting future studies skip dedicated energy hardware; this is an extension the paper does not make.","A targeted follow-up varying cache pressure (for instance, workloads that exceed cache size) would test whether Nuitka's combination of low memory and high LLC misses is a genuine memory-versus-cache trade-off with energy consequences, since uncore cache components dominate energy in this data.","Running the same harness on benchmarks that exercise dictionaries, strings, file I/O, and numerical library calls would map where each compiler's advantage fades; Numba's and Codon's known compatibility constraints suggest library-heavy code may not show the same gains."],"forward_implications":["A developer with single-threaded, CPU-bound Python can expect Codon, PyPy, or Numba to cut execution time and energy by roughly 85–94% relative to CPython without changing the algorithm, provided the code is compatible with the compiler's type requirements.","Because the study finds a strong correlation between execution time and energy, speedups from compilation are also energy savings for fixed-frequency, single-core workloads.","Nuitka is the compiler to pick when memory footprint is the priority: it gives the lowest and least variable memory usage on both testbeds, though with high LLC miss rates and no large time or energy gains.","Compilation does not always help: n_body shows that some AOT compilers can regress both speed and energy on list-heavy numeric code, so compiler choice should be made per benchmark.","LLC miss rate is not a dependable indicator of compiler quality in this setting, since it varies by benchmark and testbed; most energy is consumed by uncore components such as the cache, which deserve separate study."],"supporting_citations":[{"why":"Supplies the seven single-threaded, library-free benchmark programs that are the study's subjects.","marker":"[14]"},{"why":"Establishes Python's bottom-rank performance and energy efficiency, the baseline problem the compilers are meant to solve.","marker":"[33]"},{"why":"Introduces Codon and its claimed speedups, the performance claims this study independently evaluates.","marker":"[43]"},{"why":"Documents PyPy as a JIT compiler, one of the three subjects with large measured improvements.","marker":"[37]"},{"why":"Documents Numba as a JIT compiler, another of the three high-improvement subjects.","marker":"[22]"},{"why":"Provides the RAPL-based energy and memory measurements used for the dependent variables.","marker":"[10]"},{"why":"Provides the hardware-counter data used to compute LLC miss percentages.","marker":"[26]"},{"why":"Motivates controlling core count, CPU frequency, and LLC misses in energy and performance experiments.","marker":"[49]"}],"fun_headline_variants":["Compiled Python: up to 94% faster, 86-94% less energy","Codon, PyPy, Numba beat CPython by 85-94% in speed, energy","Python compilers slash runtime and energy by ~90%","Which Python compiler wins? Codon leads, PyPy and Numba close","Compiled Python code: 94% speedup and energy savings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the seven benchmark programs, after being adjusted for Numba's and Codon's typing and printing requirements, represent Python workloads closely enough that the measured 85–94% average gains carry over to real practitioner code.","fun_headline_variants_meta":{"raw":{"variants":["Compiled Python: up to 94% faster, 86-94% less energy","Codon, PyPy, Numba beat CPython by 85-94% in speed, energy","Python compilers slash runtime and energy by ~90%","Which Python compiler wins? Codon leads, PyPy and Numba close","Compiled Python code: 94% speedup and energy savings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000303,"raw_usage":{"total_tokens":1815,"prompt_tokens":1089,"completion_tokens":726,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":705,"completion_tokens_details":{"reasoning_tokens":622}},"tokens_in":705,"tokens_out":726,"duration_ms":8730,"temperature":1.0,"reasoning_tokens":622,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:53:54.290314+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same fixed-frequency, single-core protocol on a set of unmodified Python programs that use dictionaries, strings, file I/O, and popular numerical libraries, comparing CPython against Codon, PyPy, and Numba; if the average execution-time or energy improvements over CPython fall well below the 85–94% range, or reverse on several programs, the headline gains do not generalize beyond the benchmark set.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the seven single-threaded, library-free benchmark programs that are the study's subjects."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes Python's bottom-rank performance and energy efficiency, the baseline problem the compilers are meant to solve."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Codon and its claimed speedups, the performance claims this study independently evaluates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents PyPy as a JIT compiler, one of the three subjects with large measured improvements."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents Numba as a JIT compiler, another of the three high-improvement subjects."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the RAPL-based energy and memory measurements used for the dependent variables."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the hardware-counter data used to compute LLC miss percentages."}],"review_version":1}