{"id":"763d8bd3-2661-4208-a3ce-ab00a3399914","arxiv_id":"2605.27435","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Stage-level profiling on mobile SoC finds CPUs faster than NPUs in prefill (up to 1.6x) and only modest NPU gains in decode (1.05-1.2x), plus rising energy with greater NPU offload.","lead":"This paper benchmarks mobile LLM inference on a CPU-NPU SoC and reports that CPUs can outperform NPUs in the prefill stage while NPUs give only small gains in decode, with energy use rising as more work moves to the NPU. A smart generalist might read it to understand when specialized mobile hardware actually helps or hurts real LLM deployment.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"OPMASK decomposition validity is the least-secured link in the stage-level reversal claim","rationale":"The reader’s weakest_assumption is precisely the load-bearing methodological assumption; the full-text placeholder does not alter that the claim’s security hinges on unshown validation of the decomposition. No other internal inconsistency is visible from the supplied abstract.","tokens_in":1698,"tokens_out":313,"duration_ms":22781,"concrete_test":"Re-run the prefill and decode stages on the target SoC once with OPMASK decomposition enabled and once with a minimal non-decomposed offload path (identical model, batch size, and precision); if the CPU–NPU speedup ratios shift by >10 % or the energy ordering reverses, the isolation is incomplete.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline performance reversal (CPU 1.6× faster on prefill, NPU only 1.05–1.2× on decode) and the energy penalty of increased offloading both rest on the OPMASK-controlled pipeline decomposition correctly isolating communication, quantization, and compute overheads. If the masking or control logic itself alters scheduling, fallback behavior, or measurement timing differently across CPU vs. NPU paths, the reported stage-level differences become measurement artifacts rather than hardware properties. The abstract states the method “isolates” these factors but supplies no explicit cross-check (e.g., end-to-end vs. decomposed latency on the same run) that would rule out differential bias.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper presents the first stage-aware benchmarking study of LLM inference on mobile CPU-NPU heterogeneous SoCs. It introduces an OPMASK-based controlled pipeline decomposition to isolate communication, quantization, and computation overheads, and reports a performance reversal: CPUs outperform NPUs by up to 1.6x in the compute-intensive prefill stage while NPUs yield only 1.05-1.2x in the memory-bound decode stage. It further finds that greater NPU offloading increases energy consumption by up to 51% and derives design guidelines for NPU architects.","tokens_in":1845,"tokens_out":529,"duration_ms":29095,"significance":"If the OPMASK decomposition is shown to be free of differential measurement bias, the work supplies the first systematic operator- and stage-level characterization of NPU effectiveness for on-device LLMs. The empirical reversal and energy findings would directly inform heterogeneous scheduling policies and future NPU microarchitecture targeting LLM workloads.","major_comments":[{"comment":"Methods section (OPMASK decomposition): the central reversal claim (CPU 1.6x prefill advantage, limited NPU decode gain) rests on the assumption that the OPMASK masking and control logic isolates overheads without introducing scheduling or timing artifacts that differ between CPU and NPU paths. No explicit cross-check (end-to-end vs. decomposed latency on identical runs, or ablation of the mask itself) is described; this validation is required before the stage-level differences can be treated as hardware properties rather than measurement effects.","section":"Methods (OPMASK)"},{"comment":"Results section (performance and energy numbers): the abstract and results report concrete speedups (1.6x, 1.05-1.2x) and energy penalties (up to 51%) without error bars, model sizes, dataset details, number of runs, or raw measurement distributions. These omissions make it impossible to assess whether the reported reversals exceed measurement noise or post-hoc selection effects.","section":"Results"}],"minor_comments":[{"comment":"The paper should clarify the exact set of operators masked by OPMASK and whether fallback paths are identical across CPU-only and NPU-offloaded configurations.","section":"Methods"},{"comment":"Figure captions and axis labels for stage-level latency and energy plots should explicitly state the number of repetitions and any normalization applied.","section":"Figures"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback on our manuscript. The comments highlight important aspects of methodological validation and experimental reporting that we will address in the revision.","responses":[{"response":"We agree that explicit validation of the OPMASK decomposition is necessary to confirm the absence of differential artifacts. In the revised manuscript we will add a new subsection presenting end-to-end versus decomposed latency comparisons performed on identical runs, together with mask-ablation results. These additions will demonstrate that the reported stage-level differences arise from hardware properties rather than measurement effects.","revision_made":"yes","referee_comment":"Methods section (OPMASK decomposition): the central reversal claim (CPU 1.6x prefill advantage, limited NPU decode gain) rests on the assumption that the OPMASK masking and control logic isolates overheads without introducing scheduling or timing artifacts that differ between CPU and NPU paths. No explicit cross-check (end-to-end vs. decomposed latency on identical runs, or ablation of the mask itself) is described; this validation is required before the stage-level differences can be treated as hardware properties rather than measurement effects."},{"response":"We accept that the current reporting lacks sufficient statistical and experimental detail. The revised manuscript will include error bars (standard deviation across repeated runs), explicit model sizes and architectures, dataset specifications, the number of runs per configuration, and summary statistics of the raw measurement distributions. These changes will allow readers to evaluate whether the observed reversals exceed measurement variability.","revision_made":"yes","referee_comment":"Results section (performance and energy numbers): the abstract and results report concrete speedups (1.6x, 1.05-1.2x) and energy penalties (up to 51%) without error bars, model sizes, dataset details, number of runs, or raw measurement distributions. These omissions make it impossible to assess whether the reported reversals exceed measurement noise or post-hoc selection effects."}],"tokens_in":1411,"tokens_out":426,"duration_ms":29685,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"Hi colleague,\n\nThe main takeaway is that this study reports a stage-level reversal on heterogeneous mobile SoCs: CPUs handle the compute-heavy prefill stage up to 1.6x faster than NPUs, NPUs deliver only 1.05-1.2x in the memory-bound decode stage, and increasing NPU offloading raises energy use by up to 51%. That directly questions the default assumption that more NPU usage is always better for on-device LLMs.\n\nWhat is new is the first explicit stage-aware, multi-level breakdown at operator and pipeline granularity, plus the OPMASK controlled decomposition meant to separate communication, quantization, and compute overheads. The paper does a solid job of turning those measurements into concrete design guidelines for NPU architects, and the empirical focus on real SoC behavior is a step beyond high-level claims.\n\nThe soft spots sit in the measurement foundation. The abstract gives specific speedups and energy deltas but supplies no error bars, model sizes, dataset details, or raw traces, so it is hard to tell how sensitive the reversal is to noise or post-hoc choices. The OPMASK method carries the central claim, and the stress-test concern lands: without explicit cross-checks such as end-to-end versus decomposed latency on the same runs, it remains possible that the masking logic itself alters scheduling or timing differently across CPU and NPU paths. If the full paper includes those validations and the data, the findings strengthen; otherwise the reversal rests on an assumption that has not been shown to be neutral.\n\nThis work is for the mobile systems and hardware-software co-design crowd. A reader already running on-device LLM experiments would get usable stage-specific data points to compare against their own setups. It deserves a serious referee because the question is practical, the approach is device-grounded, and the topic matters for the growing on-device LLM market, even though the methods section will need careful review.\n\nI would send it to peer review.","headline":"The paper finds CPUs outperforming NPUs in prefill and energy rising with more offloading on mobile LLMs, but the OPMASK decomposition is the unverified link in those numbers.","tokens_in":2339,"tokens_out":486,"would_cite":false,"duration_ms":42832,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"CPUs outperform NPUs in the prefill stage of mobile LLM inference while NPUs give only marginal decode gains and raise energy use.","keywords":["mobile LLM inference","NPU acceleration","CPU vs NPU","prefill stage","decode stage","energy consumption","heterogeneous SoC","pipeline benchmarking"],"falsifier":"Repeating the stage-level benchmarks on the same mobile SoC but without applying the OPMASK decomposition and checking whether the CPU-NPU performance reversal in prefill disappears.","tokens_in":2594,"feed_emoji":"📱","tokens_out":669,"duration_ms":38383,"temperature":0.7,"pith_summary":"The paper measures LLM inference performance on a mobile CPU-NPU system by breaking execution into prefill and decode stages. It finds that CPUs run up to 1.6 times faster than NPUs during the compute-heavy prefill stage, while NPUs deliver only 1.05 to 1.2 times speedup in the memory-bound decode stage. The work also shows that moving more work to the NPU increases energy consumption by up to 51 percent. These measurements matter because on-device LLM deployment increasingly assumes NPUs will provide reliable acceleration, yet the stage-level results indicate that assumption does not hold uniformly.","feed_headline":"CPUs beat NPUs in mobile LLM prefill stage","feed_subtitle":"Stage analysis finds only 1.05-1.2x decode gains and up to 51% higher energy with more NPU offloading","key_machinery":"OPMASK-based controlled pipeline decomposition methodology that isolates communication, quantization, and computation overheads within the NPU execution path.","core_discovery":"CPUs outperform NPUs in the compute-intensive Prefill stage (up to 1.6x), while NPUs provide only limited acceleration in the memory-bound Decode stage (1.05-1.2x). Scheduling overhead and cross-backend fallback reduce the practical benefits of NPU offloading. Increasing NPU offloading leads to higher energy consumption (up to 51%).","pith_inferences":["The same stage-level reversal may appear in other heterogeneous mobile accelerators if their operator support matches the NPU pattern.","Applying the decomposition method to smaller or quantized models could test whether the prefill disadvantage shrinks with reduced compute load.","Future mobile SoC roadmaps might benefit from tighter CPU-NPU data sharing to cut the observed communication overhead."],"forward_implications":["NPU offloading decisions must be made stage by stage rather than applied uniformly across the inference pipeline.","Scheduling overhead and fallback paths must be reduced before NPU acceleration becomes practical for on-device LLMs.","NPU hardware designs for mobile LLM inference should prioritize improvements in both compute-intensive and memory-bound phases.","Energy budgets for mobile LLM deployment will be higher when NPU usage increases, requiring trade-off analysis at design time."],"fun_headline_variants":["Prefill stage favors CPUs over NPUs in mobile LLMs","Decode stage NPU gains limited to 1.05-1.2x","NPU offloading increases energy consumption up to 51 percent","Cross backend fallback limits NPU offload benefits"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The OPMASK-based controlled pipeline decomposition isolates communication, quantization, and computation overheads without introducing measurement artifacts or bias from the decomposition itself.","fun_headline_variants_meta":{"raw":{"variants":["Prefill stage favors CPUs over NPUs in mobile LLMs","Decode stage NPU gains limited to 1.05-1.2x","NPU offloading increases energy consumption up to 51 percent","Cross backend fallback limits NPU offload benefits"]},"model":"grok-4.3","cost_usd":0.00825,"raw_usage":{"total_tokens":3726,"prompt_tokens":637,"num_sources_used":0,"completion_tokens":70,"cost_in_usd_ticks":82499500,"prompt_tokens_details":{"text_tokens":637,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3019,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":637,"tokens_out":70,"duration_ms":38367,"temperature":1.0,"reasoning_tokens":3019,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-30T14:59:59.083207+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Repeating the stage-level benchmarks on the same mobile SoC but without applying the OPMASK decomposition and checking whether the CPU-NPU performance reversal in prefill disappears.","supporting_citations":[],"review_version":1}