{"id":"9b3fcede-dc79-42d4-b5ca-6c6f469d8eba","arxiv_id":"2412.15352","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A measurement study characterizing LLM inference latency, power, memory, and energy on Jetson Orin devices across model sizes, power modes, and quantization, with a public testing utility.","lead":"Researchers benchmarked five Pythia language models (70 million to 1.4 billion parameters) on six NVIDIA Jetson Orin device configurations, varying power limits and 4-bit quantization. They provide a public batch testing tool and a dataset of latency, power, memory, and energy measurements for edge LLM inference.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All device-specific tables rest on one devkit's emulation of five retail Orin devices; if NVIDIA's emulation guarantee diverges under sustained LLM load, every comparison in Tables II/III loses external validity.","rationale":"The reader's weakest-assumption pick is correct: the entire empirical contribution is external validity, and that validity hinges on the devkit emulation being faithful for every measured quantity, not just for a demo workload. The paper cites NVIDIA's documentation for the emulation claim but provides no independent confirmation. This is not an internal inconsistency, but it is a correctness risk: if the emulation is not faithful under sustained load, Tables II and III describe a single devkit in software modes, not six commercial devices. The quantization contradiction in Section IV-A2 is real and worth fixing, but it is a localized interpretive error; the emulation question can invalidate all tables. Missing variance bars and the lack of a commit hash are reproducibility weaknesses, but they do not threaten the central claim as directly. A conditional verdict remains appropriate: the methods are documented well enough to reproduce, but the hardware-representativeness premise needs a direct check before the numbers can be treated as a baseline for retail Orin hardware.","tokens_in":12475,"tokens_out":3817,"duration_ms":33498,"concrete_test":"Acquire one retail device from a different family, e.g., Jetson Orin Nano 8GB, and run the public orin-llm-testing suite on it for a representative subset: pythia-70m, pythia-410m, and pythia-1.4b, each with and without 4-bit quantization, at the default 15W and 7W power models, with 5 iterations. Compare median time-per-token, total generation latency, peak power, and estimated energy against the corresponding devkit-emulated Orin Nano 8GB rows in Tables II/III and Figures 5/6. If median time-per-token or total latency differs by more than ~10% (or energy by >15%) on any configuration, the emulation premise fails and the device-specific claims cannot be accepted as characterizing retail hardware. Also record thermal throttling frequency (e.g., CPU/GPU clocks) to isolate cause.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that Tables II-III and Figures 1-7 characterize six Jetson Orin devices. But every entry was measured on a single Jetson AGX Orin developer kit re-flashed to emulate the other five configurations (Section III, relying on NVIDIA's statement [23]). The paper never independently validates that emulation. The NVIDIA claim that the devkit 'can emulate performance and power' is not a license to assume equivalence for all measured quantities. In particular, sustained token generation drives thermal/power management; the Orin Nano and NX modules have different physical boards, cooling, and power delivery than the devkit, so voltage/frequency trajectories under load can differ even if idle/peak specs match. Since latency, power, energy, and memory are all measured on the emulated configuration, a failure of emulation under load would infect every device-specific number. This is therefore more load-bearing than the quantization finding or the missing variance bars: it determines whether the dataset describes the hardware it names or only one devkit in six software modes. The paper's own text notes that flashing requires a full re-flash and that no additional storage was used, but it contains no check against a physical device.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical benchmark study of six NVIDIA Jetson Orin configurations (the AGX Orin developer kit plus five configurations intended to emulate retail Orin devices) running five Pythia LLMs with and without 4-bit quantization under each device's default NV power modes. It measures model-load and token-generation latency, power, memory, energy (via idle-baseline subtraction), and accuracy (via the LM Evaluation Harness), presents median results in Tables II and III and Figures 1-7, and provides a public GitHub repository for reproducing and extending the tests. The authors position the work as a baseline characterization and hardware/software trade-off analysis for resource-constrained LLM inference.","tokens_in":12633,"tokens_out":6669,"duration_ms":47785,"significance":"If the data are trustworthy, the paper is a useful empirical reference: it covers currently relevant embedded hardware and LLM sizes, reports 204 successful configurations with 1,020 logs, and ships reproducible tooling and analysis scripts. The energy-estimation method is standard, and the choice of medians over means is a reasonable robustness measure. However, the value of the dataset depends on two points that are not established in the manuscript: whether developer-kit emulation of five retail devices is faithful under sustained LLM execution, and whether the reported quantization trends are stated accurately. The paper contains no theoretical derivation or machine-checked proof; its contribution is purely experimental, so the credibility of the measurements is the central issue.","major_comments":[{"comment":"All device-specific measurements were taken on a single AGX Orin developer kit re-flashed into six configurations, relying on NVIDIA's statement that the devkit 'can emulate performance and power for any of the [devices]' [23]. The paper provides no independent validation that this emulation preserves thermal throttling, voltage/frequency trajectories, or PMIC power readings under sustained token generation. Since every latency, power, energy, and memory number in Tables II and III and Figures 1-7 is attributed to a named retail device, this assumption is load-bearing for the paper's central claim. Please either validate emulation on at least one physical non-devkit device (e.g., an Orin Nano 8GB) under the same workload, or reframe the results as measurements of emulated configurations rather than physical devices and soften the device-specific claims accordingly.","section":"Section III, Tables II and III"},{"comment":"The text states that 'the expected behavior returns for the larger models (1b, 1.4b) and quantization reduces the latency from the baseline' and that this pattern 'can be seen in Figure 1a when sliced, showing a correlation across each device.' This is contradicted by Table III for the AGX Orin Devkit and AGX Orin 32GB rows. For example, at MAXN power on the devkit, pythia-1b generation latency is 20.171 s with 4-bit quantization versus 17.157 s without, and pythia-1.4b is 28.229 s versus 23.596 s; similar inversions appear in the AGX Orin 32GB rows and at several power modes. The claimed cross-device trend actually holds only for the Orin NX and Orin Nano families. Please re-analyze the quantization effect and restrict the claim to the configurations where it is supported by the data.","section":"Section IV-A2, Table III"}],"minor_comments":[{"comment":"The sentence beginning 'To isolate our testing as much as possible and to reduce extraneous processes from using resrou' is incomplete and contains a typo; it should be rewritten.","section":"Section III-B"},{"comment":"There are several typographical errors, including 'out tests' and 'use to to compare'; please proofread this subsection.","section":"Section IV-A2"},{"comment":"The paper mentions that five iterations were run per configuration 'to allow us to compare the behavior of the initial run to subsequent loading and generation,' but no such comparison is actually presented. Either include that analysis or remove the stated motivation.","section":"Section IV-A"},{"comment":"Only median values are reported, with no indication of dispersion across the five iterations. Given the text's own observation that some configurations show a markedly different first iteration, adding IQR or min-max ranges (at least in supplementary material) would materially strengthen the empirical claims.","section":"Tables II and III and Figures 1-7"},{"comment":"The phrase 'when sliced' in reference to Figure 1a is vague; please specify exactly which slice of the data is being compared and how it supports the claimed cross-device pattern.","section":"Section IV-A2 and Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The emulation validity issue is the main gate for this paper: if the authors cannot provide a validation experiment on at least one physical retail device, the manuscript should be reframed as a study of devkit emulation modes rather than physical devices. The quantization contradiction is easy to fix in text but signals that the results section needs a careful pass. The public repository and breadth of configurations are strengths; with the two major issues resolved, the paper could become a solid empirical contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things before anything else. This is a real measurement contribution: 204 configurations of Pythia models across six Orin device configurations, with latency, power, memory, and energy, plus a public batch-testing tool. That alone makes it worth having around. But the entire device-specific dataset rests on one Orin devkit re-flashed to imitate the other five boards. NVIDIA says the devkit can emulate power and performance, and the authors say they relied on that. There's no independent check against a physical Nano or NX. If the emulation drifts under sustained token generation—thermal throttling, power-supply behavior, memory controller contention—then every column in Tables II and III describes the devkit in a software mode, not the retail hardware.\n\nThe good parts first. The methodology is honestly documented: idle-power subtraction for energy, jtop polling, five iterations per config, medians reported, and a clear account of the OOM freezes on the 4GB Nano. The repository is real and the setup scripts are modular. The authors also explicitly note that accuracy was measured on remote hardware, which is the right call.\n\nThe soft spots are proportionate to the claims. The quantization finding is the clearest problem: the text says 4-bit quantization reduces latency for the 1b and 1.4b models 'across each device,' but Table III shows the opposite on both AGX configurations. Figure 3 only shows the NX 16GB, where the pattern does hold. That is a fixable but real error in the narrative. The missing variance is minor for medians over five runs, but it would be nice to know the spread given the first-run effect they mention. Reproducibility is decent but not pinned: no commit hash and no explicit statement that the raw logs are in the repo.\n\nThe emulation premise is the soft spot with real weight. I don't think it's fatal—benchmarks on eval boards are always one step removed from production hardware—but the paper should say, plainly, that these are devkit emulation results unless a ground-truth check on at least one physical device is done. That is the one thing I would require before trusting the dataset as a baseline.\n\nWho is this for? Anyone doing LLM inference on Jetson Orin and needing a starting point for configuration trade-offs. It deserves a serious referee, with the expectation that the authors either validate the emulation or soften the device claims.","headline":"Useful benchmark data and a working test harness for LLMs on Jetson Orin, but every device-specific number inherits the trust placed in a single devkit's emulation.","tokens_in":13208,"tokens_out":2262,"would_cite":true,"duration_ms":19713,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One Jetson Orin developer kit, re-flashed to emulate six Orin devices, benchmarks five Pythia LLMs and maps their latency, power, memory, and energy trade-offs.","keywords":["large language models","edge inference","NVIDIA Jetson Orin","performance benchmarking","quantization","power consumption","embedded systems","latency measurement"],"falsifier":"Take a retail Orin Nano 8GB and a retail Orin NX 16GB, run the same five Pythia models and power models with the paper's utility, and compare the median loading and generation latencies and peak power to the corresponding entries in Tables II and III; emulation is falsified if the retail medians fall outside the variation seen across the paper's five iterations.","tokens_in":12224,"feed_emoji":"⚡","tokens_out":9701,"duration_ms":74816,"temperature":0.7,"pith_summary":"The paper's goal is to give edge-computing practitioners a baseline for running large language models on current embedded hardware, specifically the NVIDIA Jetson Orin family, without relying on cloud servers. It benchmarks five Pythia models (70 million to 1.4 billion parameters) across six Orin device configurations, varying the NVIDIA power model and 4-bit quantization, and records latency, power, memory, and estimated energy. The authors argue that such a baseline matters because local inference is needed for privacy, connectivity, and security-sensitive applications, and because prior studies target older hardware. They also release a batch-testing utility so other researchers can reproduce or extend the measurements on Orin devices.","feed_headline":"One dev kit benchmarks five LLMs on six Jetson Orin devices","feed_subtitle":"Measurements of latency, power, memory, and energy show which device and settings fit real edge constraints.","key_machinery":"The load-bearing mechanism is the Jetson Orin developer kit's flash-based emulation: NVIDIA's SDK manager re-flashes the kit with configurations that enable or disable CPU/GPU cores and memory so that one physical unit can emulate performance and power for the other Orin devices. Around that premise the paper builds a three-phase measurement loop (IDLE, MODEL LOAD, GENERATE) using the jetson-stats power and memory poller, with median aggregation over five sequential iterations, trapezoidal integration of idle-subtracted power to estimate energy, and a separate accuracy measurement from the LM evaluation harness. The utility ties these together to automate batch testing and to filter results into constraint-based use cases.","core_discovery":"On the paper's own terms, the central discovery is a measured trade-off space for running small large language models on current embedded hardware. Across 204 of 210 possible combinations of six Orin device configurations, five Pythia models (70 million to 1.4 billion parameters), four or five NVIDIA power models, and two quantization levels, the authors report median model-loading and token-generation latencies, peak memory, peak power, and estimated energy, with every configuration repeated five times. The headline results are that 4-bit quantization raises generation latency for the three smallest Pythia models but lowers it for the 1b and 1.4b models; that lower power models consistently slow generation; and that the 4 GB Orin Nano cannot load the 1b or 1.4b models without quantization because the system freezes on out-of-memory errors. The paper then converts these measurements into three constraint-based use cases that select the configuration with the best accuracy or lowest latency under limits on power, energy, memory, and accuracy.","pith_inferences":["A direct spot-check on retail units, comparing one physical Orin Nano or Orin NX against the emulated medians in Tables II and III, would settle whether flash-based emulation reproduces power and thermal behavior; the paper never independently validates this.","The quantization result suggests a testable hypothesis: on these unified-memory devices, 4-bit dequantization overhead outweighs memory savings for small models; repeating the sweep with pre-quantized checkpoints or a different quantization backend would show whether the reversal is an artifact of the authors' on-device quantization path.","Because accuracy was measured on remote hardware rather than on the Jetson devices, the use-case tables that mix accuracy constraints with on-device latency and power implicitly assume quantization loss is hardware-independent; an on-device accuracy run would test that assumption.","The three-phase logging and median aggregation could be applied to other model families and to variable output lengths, which would reveal whether the observed device rankings and use-case recommendations persist beyond 512-token generation."],"forward_implications":["Practitioners can use the published medians and the released analysis script to choose a device, power model, quantization level, and Pythia model that satisfy given latency, power, memory, and accuracy limits, as demonstrated by the three use-case tables.","Because all six device configurations were measured on the same physical developer kit with the same logging stack, the sweep provides a consistent cross-device comparison that separate single-device studies cannot offer.","The counterintuitive quantization result implies that 4-bit quantization should not be applied blindly on unified-memory Jetson devices: for models below roughly one billion parameters it increases generation latency rather than reducing it.","The five-iteration, 512-token protocol is sufficient to establish relative ordering and rough magnitudes across configurations, but the authors note that more iterations would tighten the medians and make the baseline more precise."],"supporting_citations":[{"why":"Supplies the claim that the developer kit can emulate performance and power for other Orin devices, which lets one physical unit stand in for all six.","marker":"[23]"},{"why":"Provides the five deduplicated Pythia models from 70m to 1.4b parameters that every configuration in the sweep runs.","marker":"[24]"},{"why":"Supplies the model format and the 16-bit parameter precision that serves as the no-quantization baseline.","marker":"[25]"},{"why":"Explains the unified RAM/VRAM memory pool that underlies the memory measurements and the out-of-memory freezes.","marker":"[26]"},{"why":"Provides the jtop power and memory polling used for the measured latency, power, and memory metrics.","marker":"[28]"},{"why":"Supplies the accuracy scores used in the use-case constraints.","marker":"[29]"},{"why":"Provides the flashing tool required to reconfigure the developer kit between device configurations.","marker":"[30]"},{"why":"Defines the flash.sh emulation configurations that enable or disable hardware to emulate each device.","marker":"[31]"}],"fun_headline_variants":["4-bit quantization speeds big LLMs, slows small ones on Jetson","Edge LLM benchmark: 4-bit quantization has opposite effects by size","Jetson Orin LLM trade-offs: power limits slow generation","5 LLMs, 6 Jetson, 204 configs: trade-off space mapped"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole device comparison rests on NVIDIA's assertion that flashing the Jetson AGX Orin developer kit into different hardware configurations faithfully reproduces the power and performance of the retail Orin devices; the paper does not independently verify that emulation.","fun_headline_variants_meta":{"raw":{"variants":["4-bit quantization speeds big LLMs, slows small ones on Jetson","Edge LLM benchmark: 4-bit quantization has opposite effects by size","Jetson Orin LLM trade-offs: power limits slow generation","5 LLMs, 6 Jetson, 204 configs: trade-off space mapped"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000528,"raw_usage":{"total_tokens":2566,"prompt_tokens":983,"completion_tokens":1583,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":1500}},"tokens_in":599,"tokens_out":1583,"duration_ms":11757,"temperature":1.0,"reasoning_tokens":1500,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:29:52.650621+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a retail Orin Nano 8GB and a retail Orin NX 16GB, run the same five Pythia models and power models with the paper's utility, and compare the median loading and generation latencies and peak power to the corresponding entries in Tables II and III; emulation is falsified if the retail medians fall outside the variation seen across the paper's five iterations.","supporting_citations":[{"cited_title":"”Jetson AGX Orin Developer Kit User Guide”","cited_arxiv_id":null,"evidence_quote":"Supplies the claim that the developer kit can emulate performance and power for other Orin devices, which lets one physical unit stand in for all six."},{"cited_title":"Accessed: 2024-08-20","cited_arxiv_id":null,"evidence_quote":"Supplies the model format and the 16-bit parameter precision that serves as the no-quantization baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Explains the unified RAM/VRAM memory pool that underlies the memory measurements and the out-of-memory freezes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the jtop power and memory polling used for the measured latency, power, and memory metrics."},{"cited_title":"Accessed: 2024-09-10","cited_arxiv_id":null,"evidence_quote":"Provides the flashing tool required to reconfigure the developer kit between device configurations."},{"cited_title":"Accessed: 2024- 09-10","cited_arxiv_id":null,"evidence_quote":"Defines the flash.sh emulation configurations that enable or disable hardware to emulate each device."}],"review_version":1}