{"id":"335e1af5-953d-4265-80d5-67c567796273","arxiv_id":"2507.23093","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A cross-platform measurement study of accuracy, speed, power, and memory for classical ML, deep learning, and LLM inference on four edge devices, with parameter tuning guidelines.","lead":"This paper benchmarks AI inference across four edge platforms: Raspberry Pi, Intel Neural Compute Stick, Google Coral, and Jetson Nano, running traditional ML, neural networks, and LLMs while measuring F1 score, inference time, power, and memory. It proposes a unified measurement scheme and reports device and model tradeoffs to guide edge AI deployment decisions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported measurements are internally inconsistent: Figure 5 vs Figures 7–8 disagree by ~10× for the same Raspberry Pi configs, and 5,376 MB on a 4 GB Pi or 93 W on a Jetson Nano is physically implausible; the central ranking is therefore unsupported.","rationale":"The paper addresses a real and useful benchmarking gap, and the authors' intention to standardize edge inference measurement is commendable. However, this is a measurement study: the numbers are the central claim, and the reported numbers are internally inconsistent. The reader's weakest-assumption analysis correctly identifies the measurement pipeline as load-bearing, and the contradictions in the data are sufficient to reject the paper in its current form. I checked whether the discrepancy could be explained by different definitions or parameter settings between the figures; the paper provides no such explanation, and the memory and power values are physically implausible for the specified hardware. This is not a disagreement with the community consensus about edge accelerators; it is an internal-correctness failure that prevents any of the reported rankings from being used for decision-making. The promised replication package, if released, could resolve the issue, but it is not currently available. Since the reader's verdict was already REJECT, my stress-test does not change that verdict.","tokens_in":19234,"tokens_out":5716,"duration_ms":66685,"concrete_test":"Run the released measurement script for a single configuration, Raspberry Pi + ResNet-50 at 224x224 under the Section V-B protocol, and verify that it reproduces both the Figure 5 values (2.62 s, 14.91 W, 334 MB) and the Figure 7 values (0.18 s, 1.2 W, 256 MB) from the same raw logs. If only one set is reproducible, the discrepancy is a data-processing or reporting error rather than a benign measurement artifact. If the promised repository is not yet available, reimplement the Section V-B script for this one cell.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central conclusion (Section VIII) is that the Jetson Nano with TensorRT is the most efficient edge platform, and the claimed gap-filling contribution depends on trustworthy cross-device measurements. The load-bearing point is the validity of the reported data themselves, not the choice of hardware. Section V-C defines memory utilization as the total memory consumed during inference, reported in MB, and Section V-B says a USB power meter samples at 16 Hz with an idle baseline subtracted to isolate inference-specific power. The reported results contradict these definitions and each other. First, Figure 6 reports phi-2-orange memory utilization of 5,376 MB on a 4 GB Raspberry Pi and 4,930 MB on a Jetson Nano; if this is physical RAM it exceeds the hardware limits, and if it includes virtual/swap memory then it does not match the Section V-C definition. Second, Figure 6 reports 93.12 W inference power for the Jetson Nano, which is above the board's power budget by roughly an order of magnitude, so the meter reading cannot be inference-specific as described. Third, the same configuration appears in Figure 5 and in Figures 7–8: for Raspberry Pi + ResNet-50 at 224×224, Figure 5 reports 2.62 s, 14.91 W, and 334 MB, while Figure 7 reports 0.18 s and 1.2 W; for MobileSSD at 300×300, Figure 5 reports 1.86 s and 20.21 W, while Figure 8 reports 0.25 s and 1.5 W. No change in batch size or measurement protocol is stated, so the cross-figure inconsistency invalidates the comparability of the rankings. Since all cross-device conclusions rely on these numbers, the central claim is not internally supported. The replication package is only promised upon paper acceptance (footnote 1), so the contradiction cannot currently be resolved by re-running the exact scripts.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper is an empirical measurement study of AI inference on four edge hardware configurations: Raspberry Pi, Raspberry Pi with Intel Neural Compute Stick, Raspberry Pi with Google Coral USB, and NVIDIA Jetson Nano (with both LiteRT and TensorRT). It reports F1 score, inference time, inference power, and memory utilization for traditional ML models, neural networks, deep CNNs, and two LLMs, and it also studies the effect of input resolution, batch size, token length, and token window size. The authors claim to fill a gap by providing a unified measurement scheme and conclude in Section VIII that the Jetson Nano with TensorRT is the most efficient platform for edge AI inference.","tokens_in":19633,"tokens_out":4062,"duration_ms":48602,"significance":"The contribution is intended to be a reproducible, hardware-in-the-loop benchmark that would help practitioners choose device-model-framework-parameter combinations. If the measurements were trustworthy, the heterogeneous device set and the breadth of model categories would be useful. However, the paper's value rests entirely on the accuracy and internal consistency of the reported numbers. The manuscript contains multiple large internal contradictions in the primary data, including factor-of-ten discrepancies between figures for identical configurations and physical values that exceed the hardware's RAM and power budgets. Because these issues undermine the central cross-device ranking and every RQ1-RQ3 conclusion, the claimed gap-filling contribution is not currently supported. The measurement scheme and replication intent are worthwhile, but the data would need to be regenerated before the claims can be assessed.","major_comments":[{"comment":"The same nominal configuration is reported with wildly different values in different figures. For Raspberry Pi + ResNet-50 at 224x224, Figure 5 reports 2.62 s, 14.91 W, and 334 MB, while Figure 7 reports 0.18 s and 1.2 W for the same resolution. For Raspberry Pi + MobileSSD at 300x300, Figure 5 reports 1.86 s and 20.21 W, while Figure 8 reports 0.25 s and 1.5 W. No change in batch size, measurement protocol, or hardware configuration is stated between these figures. A difference of roughly an order of magnitude in inference time and power for identical configurations cannot be attributed to noise or confidence intervals, and it invalidates the cross-device comparisons that are the basis for RQ1 and RQ3.","section":"§VI-A, Figure 5 vs. Figures 7-8"},{"comment":"Several reported memory and power values are physically implausible for the hardware described in Section IV-A.2. Figure 6 reports phi-2-orange memory utilization of 5,376 MB on a Raspberry Pi with 4 GB RAM and 4,930 MB on a Jetson Nano with 4 GB RAM; Table V lists 5,250 MB and Table VI lists 5,100 MB on the Raspberry Pi. If psutil-reported virtual memory including swap is meant, then the values do not match the Section V-C definition of 'total memory consumed during inference'; if physical RAM is meant, the values exceed the hardware limit. Similarly, Figure 6 reports 93.12 W inference power for the Jetson Nano, and 62.4 W for the Raspberry Pi, both far above the official power supplies for these boards. These results contradict the described USB power meter methodology and make the LLM comparisons in Tables V-VI unsupported.","section":"§VI-A, Figure 6 and Tables V-VI"},{"comment":"The definition and computation of inference power are internally inconsistent. Section V-C says inference power is 'the total power (in watts) consumed per input sample during the inference phase,' but then states that 'the total power is then computed by summing the recorded values over the entire inference duration.' Summing power samples over time yields energy (or watt-sample counts), not watts. Section V-B separately says that an idle baseline is subtracted to isolate inference-specific power, but Section V-C's definition does not mention baseline subtraction. Because power is a central metric in every RQ1-RQ3 conclusion, the units and computation must be stated precisely and consistently; as written, the reported power values cannot be interpreted.","section":"§V-B and §V-C, Power Consumption definition"},{"comment":"The batch-size claims are not backed by any reported measurements. The text asserts that Raspberry Pi supports only batch size 1, Intel NCS works with batch sizes 4-8, Google Coral with 8-16, and Jetson Nano with 16-32, and that Jetson Nano achieves up to 20% lower per-sample inference time, but no figure, table, or raw data for batch-size experiments is provided. Since batch-size tuning is one of the main conclusions of RQ3, the supporting data must be shown rather than asserted.","section":"§VI-C.1, Batch size impact"}],"minor_comments":[{"comment":"Contribution C4 states that scripts, data, and experiment configurations are available in the replication package, but footnote 1 says the package is 'to be released in GitHub upon paper acceptance'; the referenced repository was not available for verification at review time.","section":"§I, C4 and footnote 1"},{"comment":"Reference [16] is labeled 'Edgeai sustainability code repository' but the URL points to the Perf wiki; the citation appears to be for the wrong artifact.","section":"References, [16]"},{"comment":"The model name 'Phi-2-orange' is not a standard public model identifier, and no base model, repository, or version is given; this makes the LLM experiments difficult to reproduce.","section":"§IV-A.1, Large Language Models"},{"comment":"The phrase 'Respberry Pi is the slowest device' contains a typo, and the surrounding paragraph would benefit from pointing to specific subplots in Figure 6.","section":"§VI-A, LLM discussion"}],"recommendation":"reject","confidential_remarks":"The paper is a pure empirical study, and its conclusions rest on the reported measurements. The internal contradictions identified in Figures 5-8 and the physically implausible memory/power values cannot be resolved by a local revision; they require rerunning the experiments with corrected power and memory logging, with explicit validation of the measurement pipeline, and with a consistency check across all figures. If the authors redo the measurement campaign and provide the replication data, a resubmission could be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper wants to give you a cross-platform map of F1, time, power, and memory for classical ML, CNNs, and LLMs on Raspberry Pi, Intel NCS, Google Coral, and Jetson Nano. That is a genuinely useful thing to have, and the authors deserve credit for attempting a unified measurement scheme instead of one more single-device benchmark. The breadth—five hardware configs, twelve models, four metrics—is new, and the related work table is honest about what previous studies covered.\n\nUnfortunately, the numbers as reported do not support the conclusions. The same Raspberry Pi + ResNet-50 at 224×224 appears at 2.62 s in Figure 5 and at 0.18 s in Figure 7. MobileSSD similarly goes from 1.86 s to 0.25 s with no stated change in protocol. The LLM results include 5,376 MB memory utilization on a 4 GB Raspberry Pi and 93 W inference power on a Jetson Nano, both physically impossible for that hardware if the paper's own definitions (total memory consumed, inference-specific power) are taken at face value. If these numbers include swap or virtual memory, or the power meter is measuring the whole system under load, the text does not say so, and the Section V definitions contradict that reading.\n\nThe central claim of the paper—that Jetson Nano with TensorRT is the most efficient platform—depends entirely on these measurements. Right now that conclusion is unsupported. The replication package is only promised, not provided, so a reader cannot re-run the experiments to see what actually happened. That is a load-bearing problem for a measurement study.\n\nWhat is saving here is that the errors look fixable in principle: re-measure, state the conditions precisely, and the comparison could be valuable. But the current version is not a reliable reference. I would send it back for major revision or a full re-benchmark, not publish as is.\n\nFor a reading group, it could be a useful case study in how measurement studies can go wrong. I wouldn't cite it for numbers in the near term.\n\nRecommendation: deserve a serious referee because the topic matters and the problems are concrete, but the paper should not be accepted without the authors addressing the internal contradictions and releasing the data.","headline":"A useful benchmarking ambition undone by internally inconsistent numbers that make the central ranking unsupported.","tokens_in":20212,"tokens_out":2261,"would_cite":false,"duration_ms":26293,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Benchmarking four edge devices across ML, deep learning, and LLM inference, this paper claims the NVIDIA Jetson Nano with TensorRT optimizations offers the best speed and energy balance, and that parameter tuning matters as much as…","keywords":["edge AI","inference benchmarking","power consumption","energy efficiency","TensorRT","LiteRT","large language models","parameter tuning"],"falsifier":"Run phi-2-orange inference on a Jetson Nano while logging DC input power with a calibrated bench meter and process-level memory with cgroup counters; readings far below the reported 93 W and 5,376 MB would show those figures are measurement artifacts, not inference-specific consumption.","tokens_in":19076,"feed_emoji":"⚡","tokens_out":9353,"duration_ms":102385,"temperature":0.7,"pith_summary":"The paper sets out to give edge-AI practitioners a quantitative basis for choosing hardware, models, frameworks, and inference parameters, claiming that no prior study covers performance and energy together across traditional ML, deep learning, and large language models on the same edge devices. It reports measurements of F1 score, inference time, inference power, and memory for twelve models on five configurations: a Raspberry Pi alone, the same Pi with an Intel Neural Compute Stick or Google Coral USB, and an NVIDIA Jetson Nano running either LiteRT or TensorRT. Its central finding is that all devices deliver similar accuracy, but the Jetson Nano with TensorRT is the fastest and most memory-efficient while remaining competitive in power, making it the strongest all-round platform for complex edge workloads. A second claim is that device-specific parameter tuning — input resolution, batch size, token length, and token window — shifts the performance-resource balance as much as hardware choice does. If correct, the study turns edge deployment from trial-and-error into a selection problem with measured trade-offs.","feed_headline":"NVIDIA Jetson Nano with TensorRT tops edge AI speed and power","feed_subtitle":"Four-device benchmark of ML, deep learning, and LLM inference shows where speed, accuracy, and energy trade off.","key_machinery":"The load-bearing mechanism is the unified measurement scheme: a Python harness that samples USB power-meter readings at 16 Hz, subtracts a three-second idle baseline, and logs wall-clock time, psutil-reported memory, and inference outputs, so that every device reports the same four metrics. On top of this sits the device-specific optimization stack — TensorRT on Jetson, OpenVINO IR on the Neural Compute Stick, EdgeTPU on Coral, and LiteRT elsewhere — which is what the paper credits for the speed and memory differences. The scheme is what allows cross-device rankings to exist at all.","core_discovery":"The paper's discovery is a measured performance-energy landscape for edge AI inference rather than a single winner. Across the twelve models tested, F1 scores are nearly flat across devices, so accuracy is not the differentiator. The trade-offs appear in inference time, power, and memory: the Jetson Nano with TensorRT delivers the lowest inference time and memory use across deep learning models and LLMs while keeping power competitive; the Raspberry Pi draws the least power but is the slowest and most memory-hungry; the Coral USB and Neural Compute Stick sit in between, with Coral generally faster and the Stick often more power-efficient per model. The paper also shows that tuning input resolution, batch size, token length, and token window can change inference power and time by multiples, with Jetson Nano tolerating the widest ranges (up to 2048 tokens and batch sizes 16–32), while the Pi is limited to batch size 1 and 500–1000 tokens. TinyBERT consistently beats phi-2-orange on resource use, which the paper attributes to its distillation-based architecture. The implied answer to RQ1 is yes: domain-specific hardware and software design do boost edge AI, but mostly in resource use rather than accuracy.","pith_inferences":["Going beyond the paper: if the flat-F1 pattern holds generally, accuracy is set by training and data rather than by inference hardware, so hardware procurement for edge AI should be driven by energy and latency budgets, not by expected model quality.","Going beyond the paper: the device-specific batch-size sweet spots suggest a transferable rule — pick the largest batch that fits in memory without paging — that could be tested on newer edge accelerators without re-running the full battery.","Going beyond the paper: because the paper itself notes inference power does not always scale with runtime for lightweight models, per-inference energy figures measured at the USB input include fixed platform overhead; extending the scheme with device-internal power sensors would let future work separate compute power from idle draw."],"forward_implications":["For complex vision and language workloads, Jetson Nano with TensorRT is the recommended configuration: it achieves the fastest inference and lowest memory use while keeping power competitive across deep models and LLMs.","Raspberry Pi remains viable only for simple models and latency-tolerant tasks; it draws the least power but is the slowest and most memory-pressured, with batch size 1 as the only workable setting.","Coral USB and Neural Compute Stick occupy the middle ground, with Coral generally faster and the Stick often more power-efficient per model, making the choice between them model-dependent.","Increasing image resolution from the native size to 512×512 raises F1 but multiplies power and time, sharply on the Pi and roughly twofold on Coral and the Stick; input size should therefore be tuned against the application's accuracy target.","For LLMs, TinyBERT is the safer edge choice than phi-2-orange across all devices, and Jetson Nano's ability to handle up to 2048 tokens means longer-context NLP is only practical there."],"supporting_citations":[{"why":"Prior CNN inference-speed comparison on Jetson and Neural Stick that this study extends to more model types and power metrics.","marker":"[6]"},{"why":"Prior framework comparison on edge that supplies the latency-and-memory baseline the paper adds power to.","marker":"[1]"},{"why":"Prior study of framework-level energy costs that motivates the paper's model-and-device power measurements.","marker":"[5]"},{"why":"Prior work on quantization and LLM deployment on constrained hardware that informs the LLM and TinyML choices.","marker":"[9]"},{"why":"Prior survey of optimization methods such as quantization and pruning that the parameter-tuning experiments build on.","marker":"[10]"},{"why":"Energy-aware edge inference tuning approach that the paper cites as motivation for parameter-level optimization.","marker":"[20]"},{"why":"Prior LLM inference benchmarking on Jetson that the paper extends to multiple edge platforms and model types.","marker":"[21]"},{"why":"TinyBERT distillation paper that explains the observed efficiency advantage of TinyBERT over phi-2-orange.","marker":"[37]"},{"why":"LLM inference benchmark defining token length and window as key parameters, grounding the LLM tuning experiments.","marker":"[38]"}],"fun_headline_variants":["Edge AI benchmark: Jetson Nano leads, Pi sips power, Coral in between","Speed vs. power: edge AI devices trade off performance and energy","Jetson Nano with TensorRT outpaces edge rivals in inference speed","Tuning edge AI models can multiply power and time by factors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The rankings all rest on one premise: that a USB power meter sampling at 16 Hz with an idle baseline subtracted, paired with psutil's memory reading, captures the inference-specific power and memory of each device, so that if either reading is contaminated the cross-device rankings shift.","fun_headline_variants_meta":{"raw":{"variants":["Edge AI benchmark: Jetson Nano leads, Pi sips power, Coral in between","Speed vs. power: edge AI devices trade off performance and energy","Jetson Nano with TensorRT outpaces edge rivals in inference speed","Tuning edge AI models can multiply power and time by factors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000547,"raw_usage":{"total_tokens":2639,"prompt_tokens":992,"completion_tokens":1647,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":1568}},"tokens_in":608,"tokens_out":1647,"duration_ms":12237,"temperature":1.0,"reasoning_tokens":1568,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:03:27.998996+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run phi-2-orange inference on a Jetson Nano while logging DC input power with a calibrated bench meter and process-level memory with cgroup counters; readings far below the reported 93 W and 5,376 MB would show those figures are measurement artifacts, not inference-specific consumption.","supporting_citations":[{"cited_title":"Edge devices inference performance comparison,","cited_arxiv_id":null,"evidence_quote":"Prior CNN inference-speed comparison on Jetson and Neural Stick that this study extends to more model types and power metrics."},{"cited_title":"pcamp: Performance comparison of machine learning packages on the edges,","cited_arxiv_id":null,"evidence_quote":"Prior framework comparison on edge that supplies the latency-and-memory baseline the paper adds power to."},{"cited_title":"Optimizing large language models for edge devices: A comparative study on reputation analysis,","cited_arxiv_id":null,"evidence_quote":"Prior work on quantization and LLM deployment on constrained hardware that informs the LLM and TinyML choices."},{"cited_title":"Enabling deep learning on iot edge: Approaches and evaluation,","cited_arxiv_id":null,"evidence_quote":"Prior survey of optimization methods such as quantization and pruning that the parameter-tuning experiments build on."},{"cited_title":"DynaSplit: A Hardware-Software Co-Design Framework for Energy-Aware Inference on Edge","cited_arxiv_id":"2410.23881","evidence_quote":"Energy-aware edge inference tuning approach that the paper cites as motivation for parameter-level optimization."},{"cited_title":"TinyBERT: Distilling BERT for natural language understanding,","cited_arxiv_id":null,"evidence_quote":"TinyBERT distillation paper that explains the observed efficiency advantage of TinyBERT over phi-2-orange."},{"cited_title":"Llm-inference- bench: Inference benchmarking of large language models on ai acceler- ators,","cited_arxiv_id":null,"evidence_quote":"LLM inference benchmark defining token length and window as key parameters, grounding the LLM tuning experiments."}],"review_version":1}