{"id":"8382cee4-90c9-47ee-9354-041062b37174","arxiv_id":"2506.19884","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MNN-AECS reduces on-device LLM decoding energy by about 23% on average by adaptively selecting low-power CPU cores or fewer threads during decode, while keeping decode speed within an 8% slowdown bound.","lead":"This paper introduces MNN-AECS, an engine-level system that picks fewer or smaller CPU cores during the memory-bound decode phase of on-device LLM inference to cut energy use. On seven phones, it reports 23% average energy savings over the baseline MNN engine with no slowdown, and 39% to 78% savings versus other engines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed energy saving rests on CPUFreq governors scaling down idle-cluster frequency; the paper's own Meizu 21 result shows saving drops to ~10% when this governor behavior is absent, so the 23% average is not a stable device-level guarantee.","rationale":"The reader's weakest_assumption is the same one I would stress: the energy benefit is contingent on governor-controlled frequency scaling of idle clusters. The paper deserves credit for reporting the Meizu 21 case and for validating AECS against exhaustive search, which supports search optimality on the tested devices. I do not see an internal contradiction in the arithmetic of the 23% average; the per-device ratios in Tables 9 and 10 reproduce the plotted reductions. The other limitations noted by the reader, including no confidence intervals, no released artifact, and unspecified tuning/evaluation separation, are real but secondary: they lower reproducibility confidence without pinning the mechanism as precisely as the governor dependence does. The appropriate disposition remains conditional acceptance: the work is plausible and mostly well-measured on the seven devices, but the headline claim should be framed as device- and governor-dependent until the mechanism is confirmed on additional hardware and the tuning/evaluation separation is clarified.","tokens_in":22721,"tokens_out":16088,"duration_ms":171249,"concrete_test":"Run the full MNN-AECS versus MNN dataset protocol on an additional Android device with a walt governor, and during decode record scaling_cur_freq for every CPU cluster from /sys/devices/system/cpu/cpufreq/policy*/scaling_cur_freq. If idle clusters remain at high frequency and the energy saving matches Meizu 21's ~10% rather than the 18% to 42% range, the 23% average should be reported with explicit governor dependence instead of as a general mobile-device result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is that leaving cores or clusters idle makes the OS CPUFreq governor (schedutil or walt) lower their frequency, yielding most of the energy saving. This is codified in Eq. (9) through f_i = f_max,i * s_I and in Section 2.4, where the paper argues that core selection without root works precisely because the governor scales down idle clusters. The paper's Meizu 21 result is the natural falsification: energy saving over MNN is 9.7% in Figure 11 versus 18% to 42% on the other six devices, and Section 5.3 attributes this to the OS not scaling down the idle-cluster frequency. So the mechanism that makes AECS effective is not under the engine's control and is absent on at least one shipped device. Because the headline 23% is an average over the seven devices, it is not evidence that a comparable saving will appear on other walt or schedutil devices with different frequency policies. This is a generalizability limitation, not an internal inconsistency in the reported measurements; but it is the assumption on which the central 'no root, no OS modification' claim depends.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MNN-AECS, an extension of the MNN on-device LLM inference engine that performs a once-per-device offline search over CPU core selections (core affinity on Android, thread count on iOS) to minimize the energy consumed during the memory-bound decode phase, subject to a decode-speed constraint of at most epsilon=8% slowdown relative to the fastest selection. The search (AECS) has two stages: Stage 1 greedily constructs the fastest core selection from CPU-cluster information; Stage 2 generates a small candidate tree by heuristic transformations and profiles each candidate, using an objective that blends measured energy with a power heuristic. The power heuristic models CPUFreq governor behavior via f_i = f_max,i * s_I and includes cluster scaling factors a_i, an idle-core factor b, and static power P_s. The authors integrate AECS into MNN with separate prefill/decode thread-pool configurations and build a cross-platform energy testbed that reads OS battery interfaces on Android and Xcode energy gauge on iOS. They report results on 5 Android and 2 iOS devices with 5 LLMs over 4 datasets: a 23% average energy reduction relative to MNN (versus 18-42% on individual devices, except 9.7% on Meizu 21), and 39-78% energy reduction and 12-363% speedup relative to llama.cpp, executorch, mllm, and MediaPipe. An ablation claims that AECS matches exhaustive search in 100% of cases while being about 10x faster.","tokens_in":22914,"tokens_out":8983,"duration_ms":83721,"significance":"If the measured savings are robust, the paper demonstrates a practical, root-free engine-level lever for reducing LLM decoding energy by roughly a fifth on average, a meaningful improvement for battery-constrained mobile inference. The design is well-motivated by the memory-bound nature of decode, and the use of a cross-platform testbed and a comparison against four other engines is a definite strength. The ablation against exhaustive search, if correctly reported, provides direct evidence that the heuristic search does not sacrifice solution quality. The main value is as an empirical systems contribution; the power heuristic itself is auxiliary, with the energy savings coming from the measured selection of fewer/smaller cores. However, the paper's statistical basis is thin, the heuristic parameters are undisclosed, and the central mechanism depends on OS governor behavior that is not under the engine's control, so the headline numbers should be read as device-dependent.","major_comments":[{"comment":"All energy and speed results in Tables 9 and 10 and Figures 11-13 are reported as single averages over datasets with no standard deviation, confidence interval, or number of repetitions. This matters because Section 3.3 explicitly states that empirically measured energy suffers from 'intolerable system fluctuation ... as high as 5%'. Without per-condition variance data, the 23% average energy saving over MNN cannot be statistically distinguished from run-to-run variation on individual devices, and the comparison against other engines lacks a noise floor. Please add repetition counts and variance/confidence intervals for at least the headline comparisons, or explicitly state if each reported number is a single measurement.","section":"Section 5.3/5.4, Tables 9-10, Figures 11-13"},{"comment":"The power heuristic in Eq. (9) depends on unspecified constants: the cluster scaling factors a_i, the idle-core reduction factor b, the static power P_s, and the weight alpha in the objective E_h(I) = (1-alpha)E(I) + alpha h(I)t(I). No values, fitting procedure, or sensitivity analysis is provided. Consequently AECS cannot be reimplemented from the paper, and the ablation in Table 11 that attributes better optimality to the heuristic cannot be reproduced. In addition, Section 3.3 states that 'AECS search aborts energy profiling on iOS, using only the heuristics for optimization', but the paper never says what objective is used in that case (presumably alpha=1) nor how the exhaustive-search comparison in Table 11 was conducted on iOS; please specify the exact configuration and report the parameter values.","section":"Section 3.3, Eq. (9)"},{"comment":"The energy saving over MNN is only 9.7% on Meizu 21, versus 18-42% on the other six devices, and the paper attributes this to the OS not scaling down the idle-cluster frequency. Since the design's power-reduction mechanism relies on the CPUFreq governor lowering the frequency of idle cores (Eq. (9) and Section 2.4), the 23% average reported in the abstract and conclusion is a property of the test sample, not a device-independent property of AECS. Please report the per-device savings distribution explicitly, state the governor-behavior condition under which the method is effective, and discuss how widely walt/schedutil devices can be expected to exhibit it.","section":"Section 5.3, Figure 11(a)"},{"comment":"Table 11, as presented, has columns 'exhaustive', 'AECS', and 'AECS(optimal) w/o heuristic' with optimality rates 100%, 60%-90%, and 100%, respectively; this conflicts with the text stating that 'removing the power heuristic leads to fluctuation in searching and consequently lower optimality rate'. The column layout and labeling need to be fixed so that the reader can tell which variant achieves 60%-90% and which achieves 100%; otherwise the ablation evidence for the power heuristic is ambiguous.","section":"Table 11"}],"minor_comments":[{"comment":"The claim that energy is reduced 'without slowdown' is stronger than what the speed constraint permits; Figure 11(b) reports a 6.8% slowdown on V30 Pro. Suggest rephrasing to 'with average speed preserved' or 'within an 8% slowdown bound'.","section":"Abstract and Section 1.3"},{"comment":"The energy columns have no stated unit; for iOS the values come from Xcode energy gauge and are relative, not mJ/token, so the tables should label the metric accordingly.","section":"Tables 9 and 10"},{"comment":"There is a typo: 'serach time' should be 'search time'.","section":"Table 11"},{"comment":"Reference [53] lists the year as '5555' and month 'Feb. 5555', which appears to be a citation-generation error that should be corrected.","section":"References"},{"comment":"The phrase 'The tree depth is limited less than 2' should be 'limited to at most 2' (or 'less than or equal to 2').","section":"Section 3.3"},{"comment":"The phrase 'search space scales up to 20-71' should specify that these are counts of candidate core selections, not a time or size unit.","section":"Section 5.5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical systems contribution, and the exhaustive-search ablation is valuable. My main concerns are the missing parameter values for the power heuristic, the lack of variance reporting, and the device-dependence of the claimed average. These are fixable with additional disclosures and experiments; I do not see an internal inconsistency that would require rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a serious systems paper with a real, if narrow, result. The idea—during memory-bound decode, leave big cores idle so the CPUFreq governor drops their frequency—is not groundbreaking in itself, but the authors turn it into a working engine-level feature with a per-device heuristic search and support it with a seven-device, five-engine evaluation. The ablation against exhaustive search (same selected configuration, 10x less search time) is the most convincing piece. I believe the measured 23% average energy reduction over MNN is real, and the order-of-magnitude comparisons against other engines are worth taking seriously.\n\nWhat actually merits credit: they identify decode-phase energy as the bottleneck, design the search around measured energy with a power heuristic rather than pretending the heuristic is the result, and they integrate cleanly with MNN's thread pool so prefill and decode can use different core sets. The unrooted, no-OS-modification angle is practically important. They are also honest about the Meizu 21 case, where saving drops to 10% because the OS does not scale down idle-cluster frequency.\n\nThe soft spots are real but mostly fixable. First, the energy and speed tables present single averages without variance, repetition counts, or confidence intervals, although they acknowledge ~5% measurement fluctuation. That is the biggest barrier to accepting the headline number. Second, I cannot tell from the paper whether the data used during per-device tuning is disjoint from evaluation data; if the same prompts select the config and then measure its energy, that is a leak. Third, the power heuristic has uncalibrated constants, and the sensitivity of the search to them is not reported. Fourth, on iOS the search intentionally uses the heuristic without measured energy because Xcode profiling requires developer mode; given the ablation shows heuristic-only search is 60–90% optimal, the iOS results deserve a caveat. These do not break the central claim, but they narrow how much of the 23% average generalizes. No artifact or code is released, so reproduction is not yet possible.\n\nOverall: this is a solid paper for systems venues. The mechanism is sound, the evaluation is broad, and the limitations are mostly stated rather than hidden. It deserves a serious referee, with the expectation of a revision addressing variance, tuning/eval separation, and the iOS heuristic-only caveat.","headline":"A real engine-level energy optimization for on-device LLM decode, with a believable 23% average saving; the main caveats are missing variance reporting, no tuning/eval separation shown, and device-dependent governor behavior that the paper itself documents.","tokens_in":23527,"tokens_out":2887,"would_cite":true,"duration_ms":30845,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On-device LLM decoding can cut energy use by 23% without slowing down by adaptively selecting fewer, smaller CPU cores, because the memory-bound decode phase tolerates fewer cores while the OS lowers idle-core frequency.","keywords":["on-device LLM inference","energy optimization","CPU core selection","CPU affinity","DVFS","memory-bound decoding","mobile systems"],"falsifier":"Force the CPUFreq governor on one of the paper's Android devices into 'performance' mode, so idle cores stay at maximum frequency, then re-run the MNN-AECS versus MNN comparison on the same datasets; if the 23% energy saving persists, the claimed idle-frequency mechanism is not the cause, and if it collapses, the governor is load-bearing.","tokens_in":22487,"feed_emoji":"🔋","tokens_out":8311,"duration_ms":75073,"temperature":0.7,"pith_summary":"Large language models running on phones spend most of their energy not on reading the prompt but on generating tokens one at a time, and the paper claims this decode phase is memory-bound: the CPU cores mostly wait on the memory bus. Because decode tolerates less parallelism, the authors argue that binding decoding to fewer, smaller CPU cores leaves other cores idle, which lets the operating system's frequency governor lower their clock speed and cut power. They build this into MNN, an on-device inference engine, as a tool called AECS that searches once per device for the best core selection, then uses it for all future decoding. Across seven phones and several small LLMs, the integrated engine reports 23% lower energy than the original engine at the same speed, and 39-78% less energy than four other engines while running faster. The relevance is practical: it is an engine-level change, requiring no root access or OS modification, so it could ship in ordinary apps.","feed_headline":"Two CPU cores beat eight: LLM decoding energy drops 23%","feed_subtitle":"Adaptive core selection keeps decode speed while idle cores clock down, no root access needed.","key_machinery":"The carrying mechanism is the power heuristic h(I) of Eq. (9), $$h(I)=\\sum_{i=0}^{n-1} a_i(|I_i|+(|C_i|-|I_i|)b)(f_{\\max,i}\\cdot s_I)^2+P_s,$$ which estimates decode power from the number of selected cores in each cluster, a per-cluster type factor, an idle-core factor b<1, and a governor-modeled frequency f_i = f_max,i * s_I, where s_I is the ratio of the selected biggest core's capacity to the biggest capacity on the chip. This heuristic is used in a two-stage search: stage 1 greedily builds the fastest selection starting from one prime core, and stage 2 grows a small candidate tree via transformations (remove one or two smallest cores, swap a big core for a smaller one, switch to a smaller cluster) and evaluates each candidate with a speed constraint, returning the selection with the lowest heuristically-regularized energy. The search is run once per device at tuning time, and the chosen selection is then applied to the thread pool for all future decode calls, with prefill and decode allowed different core selections.","core_discovery":"The paper's central claim is that the energy bottleneck of on-device LLM inference is the decode phase, not the prefill phase, and that this bottleneck can be attacked by choosing the right CPU core selection. It asserts that during memory-bound decoding, using all available CPU cores wastes energy because additional cores contribute little speed while keeping every cluster's frequency high. AECS therefore solves an optimization: minimize per-token energy E(I)=P(I)t(I) subject to speed(I) being at least 92% of the fastest selection, over core binding plans on Android and thread counts on iOS. The authors report that the resulting selection uses at most two cores on all seven test devices, cuts CPU utilization by 50-75%, reduces energy by 23% on average over the base engine without slowdown, and beats four other engines by 39-78% in energy while also being 12-363% faster. The energy reduction is presented as the product of leaving cores idle and the OS governor scaling down their frequency, and it is verified on 5 Android and 2 iOS devices across 5 LLMs and 4 datasets.","pith_inferences":["On devices whose governor keeps idle-cluster frequency high, the same mechanism should deliver smaller savings; forcing or emulating idle frequency scaling on such devices would test whether the 23% average is recoverable across the full hardware landscape.","The same core-undersubscription logic might apply to other memory-bound mobile workloads, such as on-device speech recognition or embedding search; a natural test is to run an AECS-style per-device tuning pass on those workloads and measure energy per inference.","From the paper's own GPU discussion, a GPU or NPU that exposes shader-core selection or frequency control could extend the energy savings beyond the CPU path, potentially compounding the reported reductions.","Since the mechanism is orthogonal to model compression, combining AECS with smaller or more aggressively quantized models should multiply per-token energy savings; direct measurement of energy per token on an AECS-tuned device at 2-bit or sparse precision would settle how the savings add."],"forward_implications":["Energy savings persist across prompt lengths from 64 to 1024 tokens, decode lengths from 128 to 512, five different LLMs, and four datasets, indicating the effect is structural rather than dataset-specific.","The tuned selection never uses more than two CPU cores on any of the seven test devices, cutting CPU utilization by 50-75%, and on six of seven devices it is as fast or faster than the base engine, with at most 7% slowdown elsewhere.","The once-and-for-all AECS search takes 1-2 minutes and matches exhaustive traversal in the chosen optimum, making per-device tuning practical enough for real deployments.","Because the change lives entirely in the inference engine, it requires no root access or OS modification and is orthogonal to quantization or sparsity methods, so it can combine with algorithmic optimizations."],"supporting_citations":[{"why":"Android kernel sched.h source that defines the capacity factor s_I used to model governor-assigned frequency in the power heuristic.","marker":"[5]"},{"why":"Cited for the claim that CPU power rises super-linearly with frequency, justifying the quadratic power-frequency term in the heuristic.","marker":"[14]"},{"why":"The schedutil governor implementation whose behavior (frequency scaled by estimated workload) the heuristic models.","marker":"[4]"},{"why":"Supports the memory-bound premise by noting that single-user decode cannot be batched, which is why the decode phase waits on the memory bus.","marker":"[9]"},{"why":"The base on-device LLM engine that the paper modifies; its thread pool and KV cache memory layout are changed to support per-phase core selection.","marker":"[33]"},{"why":"Main baseline engine for energy and speed comparisons, and an example of an engine that selects all cores during decode.","marker":"[19]"},{"why":"Baseline engine included in the cross-engine comparisons on Android for Llama models.","marker":"[8]"},{"why":"Baseline engine focused on NPU prefill acceleration; its lack of decode-energy optimization motivates the paper's design.","marker":"[51]"},{"why":"ARM idle-states documentation, basis for the reduced factor b applied to non-selected cores in a cluster.","marker":"[1]"}],"fun_headline_variants":["Decode phase eats energy: adaptive cores cut LLM drain 23%","Skip idle cores: MNN-AECS saves 23% energy in LLM decoding","Core selection trims LLM decode energy by 23% on mobile","MNN-AECS: pick few cores, cut LLM decode energy 23%","Adaptive core pick cuts LLM energy 23% without slowdown"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The energy savings depend on the operating system's frequency governor actually lowering the clock speed of CPU cores that are left idle; if a device's governor keeps idle clusters at high frequency, the 23% average saving shrinks, as the paper's own Meizu 21 result (10% instead of 18-42%) shows.","fun_headline_variants_meta":{"raw":{"variants":["Decode phase eats energy: adaptive cores cut LLM drain 23%","Skip idle cores: MNN-AECS saves 23% energy in LLM decoding","Core selection trims LLM decode energy by 23% on mobile","MNN-AECS: pick few cores, cut LLM decode energy 23%","Adaptive core pick cuts LLM energy 23% without slowdown"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000125,"raw_usage":{"total_tokens":1128,"prompt_tokens":988,"completion_tokens":140,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":38}},"tokens_in":604,"tokens_out":140,"duration_ms":2005,"temperature":1.0,"reasoning_tokens":38,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:33:33.829559+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Force the CPUFreq governor on one of the paper's Android devices into 'performance' mode, so idle cores stay at maximum frequency, then re-run the MNN-AECS versus MNN comparison on the same datasets; if the 23% energy saving persists, the claimed idle-frequency mechanism is not the cause, and if it collapses, the governor is load-bearing.","supporting_citations":[{"cited_title":"kernel/sched/sched.h","cited_arxiv_id":null,"evidence_quote":"Android kernel sched.h source that defines the capacity factor s_I used to model governor-assigned frequency in the power heuristic."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Cited for the claim that CPU power rises super-linearly with frequency, justifying the quadratic power-frequency term in the heuristic."},{"cited_title":"kernel/sched/cpufreq_schedutil.c","cited_arxiv_id":null,"evidence_quote":"The schedutil governor implementation whose behavior (frequency scaled by estimated workload) the heuristic models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The base on-device LLM engine that the paper modifies; its thread pool and KV cache memory layout are changed to support per-phase core selection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Main baseline engine for energy and speed comparisons, and an example of an engine that selects all cores during decode."},{"cited_title":"GitHub - pytorch/executorch: On-device AI across mobile, embedded and edge for PyTorch — github.com","cited_arxiv_id":null,"evidence_quote":"Baseline engine included in the cross-engine comparisons on Android for Llama models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ARM idle-states documentation, basis for the reduced factor b applied to non-selected cores in a cluster."}],"review_version":2}