Pith. sign in

REVIEW 4 major objections 5 minor 32 references

Selecting a lightweight language model by size or FLOPs can pick the wrong deployment candidate, because those proxies track running cost but not accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 09:18 UTC pith:W3LRBH6X

load-bearing objection Solid, reproducible measurement study with useful Pareto results; just treat the pairwise precision-equivalence claims as lower-confidence than they look. the 4 major comments →

arxiv 2607.20806 v1 pith:W3LRBH6X submitted 2026-07-23 cs.AI

Profiling Lightweight Large Language Models

Tomohiro Harada , Enrique Alba , Gabriel Luque This is my paper
classification cs.AI
keywords PTME profilinglightweight LLM inferenceenergy measurementresource envelopesPareto analysismodel selectionproxy descriptors
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the usual way of choosing a lightweight language model for a laptop or edge device—looking at parameter count, FLOPs, or model size—can estimate how much a model will cost to run but says almost nothing about whether it will answer correctly. The authors introduce a PTME profiling methodology that measures precision, execution time, peak memory, and energy directly on constrained hardware, and apply it to six small models on code, math, and general-knowledge benchmarks. They find that cost metrics track proxies closely (correlations around 0.95–1.0) while precision does not (below 0.5), that no single model wins on all four dimensions at once, and that tightening the resource budget raises time and energy without changing accuracy, hitting larger models hardest. If correct, this means deployment choices based on size, FLOPs, latency, or accuracy alone can pick the wrong model.

Core claim

The paper's central claim is that static proxy descriptors—parameter count, theoretical FLOPs, and loaded-memory footprint—are useful first-order predictors of inference cost (token-normalized time, peak memory, energy) but are not predictors of task precision, and that lightweight-LLM comparison must therefore be done jointly on accuracy, time, memory, and energy at the target execution envelope. On six models from 1.1B to 7.2B parameters run CPU-only under five resource envelopes, the authors report Pearson correlations of roughly 0.95–1.0 between each proxy and the cost metrics, versus roughly 0.36–0.48 with precision. A four-objective Pareto analysis then shows that single-metric selecti

What carries the argument

The central object is the PTME profiling framework: four metrics measured together on the same inference run—Precision (benchmark accuracy), Time (end-to-end wall-clock), Memory (peak resident set size), and Energy (CPU-package hardware counter). It is paired with controlled resource envelopes (CPU core counts, frequency caps, RAM limits) and analyzed with paired non-parametric tests plus four-objective Pareto dominance. This combination is what lets the paper separate what static proxies can predict (cost) from what they cannot (precision), and what single-metric views hide (non-dominated configurations).

Load-bearing premise

The load-bearing premise is that the constrained desktop setup (fewer cores, lower clock, RAM cap) reproduces the relative cost behavior of real edge/mobile devices, so that the ordering of models and the envelope-sensitivity conclusions transfer beyond the test bench.

What would settle it

Run the same PTME protocol on an actual edge or mobile device (for example, a Raspberry Pi or an ARM smartphone) and compare the relative ordering of models and the ratio of time to energy across envelopes. If a smaller model becomes more energy-hungry than a larger one, or if tightening the envelope no longer amplifies time more than energy, the paper's central conclusions about proxies and envelope sensitivity would fail.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Choosing a model by parameter count, FLOPs, or loaded memory is safe for estimating time, memory, and energy per token, but not for estimating accuracy; a small model picked for speed may be useless on a real task.
  • No model dominates across precision, time, memory, and energy at once, so any ranking built on a single metric is incomplete.
  • Qwen2.5 and Phi-3 are non-dominated alternatives that can match the largest model's precision statistically while using substantially less time and energy on MMLU-Pro.
  • Tightening the resource envelope raises physical cost without changing output quality, and it penalizes larger models more than smaller ones, so efficiency is a property of the model-envelope pair, not of the model alone.
  • Pareto analysis is a decision-support tool: it removes dominated choices and exposes precision-cost trade-offs, rather than naming a single best model.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the proxy-precision decoupling extends beyond these six models, then any leaderboard that ranks lightweight models by size, latency, or accuracy alone is incomplete; deployment-oriented rankings should report a Pareto front over measured PTME at a stated envelope.
  • The same protocol could be applied to quantization variants, GPU execution, or actual edge devices; a testable extension is whether the non-dominated set shifts when memory bandwidth, thermal throttling, or accelerator power profiles differ.
  • Because precision is invariant across envelopes under greedy decoding, envelope choice can be optimized separately from accuracy once a model is fixed—an implication the paper leaves implicit.
  • The asymmetry that tightening resources amplifies time more than energy suggests that on battery-powered devices, latency and energy are not interchangeable; direct energy measurement matters most precisely in the tight-envelope regime.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces a PTME (Precision, Time, Memory, Energy) measurement framework for profiling lightweight LLMs under resource-constrained envelopes. Six quantized models (TinyLlama 1.1B, Qwen2.5 1.5B, Gemma 2B, Phi-2, Phi-3-mini, Mistral-7B) are evaluated on HumanEval, GSM8K, and MMLU-Pro under five desktop-simulated edge envelopes, using deterministic decoding and direct RAPL-based energy measurements. The central claims are: (RQ1) static proxy descriptors (parameter count, FLOPs, loaded memory) correlate strongly with measured cost metrics (R≈0.95–1.0) but weakly with precision (R≈0.36–0.48); (RQ2) no single model dominates across all PTME dimensions, and Pareto analysis reveals non-dominated configurations (e.g., Qwen2.5 and Phi-3 as statistically comparable, lower-cost alternatives to Mistral on MMLU-Pro); (RQ3) tightening the resource envelope leaves precision unchanged but increases time more than energy, with larger models penalized most. The paper concludes that size-, FLOPs-, latency-, or accuracy-only selection can choose the wrong deployment candidate, and joint precision-resource profiling is needed.

Significance. The study has real strengths: direct hardware-level energy metering, matched benchmark instances across all conditions, deterministic greedy decoding, paired non-parametric tests, and a public code/data repository. If the empirical claims hold, the Pareto-based decision framework would be a practically useful contribution to lightweight-LLM deployment, and the RQ1 result that proxies track cost but not precision is a clean, falsifiable finding. The main weakness is that the precision axis—on which several load-bearing conclusions rest—is measured on small subsets (HumanEval 164; GSM8K 200 of 1319; MMLU-Pro 280 of 12032) with no confidence intervals and no assessment of test power. For a deployment-oriented paper, the distinction between 'no significant difference' and 'demonstrated equivalence' is material. The envelope-transferability limitation is explicitly acknowledged, but the RQ3 conclusions are still framed more broadly than the evidence on one x86 desktop platform supports.

major comments (4)
  1. [§VI-C, §VII-C, Table VIII] The central RQ2 guidance that 'Qwen2.5 and Phi-3 provide viable alternatives to Mistral... without a statistically confirmed loss in precision' rests on McNemar non-significance with very low power. On MMLU-Pro, Qwen2.5's 28.2% precision (n=280) has a 95% CI of about ±5.3 points against Mistral's 34.6%; for HumanEval (n=164) and GSM8K (n=200) the intervals are comparably wide. A non-significant McNemar test here cannot distinguish equivalence from insufficient evidence. Please report exact p-values, discordant-pair counts, and confidence intervals for all precision differences used in Table VIII, or perform full-benchmark precision evaluation. Without this, the wording 'viable alternatives... without a statistically confirmed loss' overstates what the data support.
  2. [§VII-B, Table VII] RQ1's headline precision finding uses Pearson correlations computed over 18 points (6 models × 3 benchmarks) pooled across benchmarks, with no confidence intervals, no adjustment for benchmark clustering, and no account of measurement noise. Precision is also invariant across envelopes, so the E1–E5 rows reuse the same 18 precision values and are not independent evidence. The claim that proxies 'fail to predict precision' needs at least bootstrap CIs for R and ideally a mixed-effects model with benchmark as a random effect. As reported, R≈0.36–0.48 could be substantially attenuated by the small sample and by per-benchmark differences.
  3. [§V-A, §VI-A, §VII-D] Time, energy, and memory are measured once per inference instance, with no repeated executions logged. The reported means over instances therefore carry unquantified run-to-run variation, and the RQ3 ratio statements (e.g., 'E1 requires about 5.3–6.3 times longer than E5') have no uncertainty bounds. Since these ratio claims are load-bearing for the envelope-sensitivity conclusion, the authors should either report per-configuration standard deviations/confidence intervals or demonstrate that repeated-run variation is negligible for the cross-envelope ordering.
  4. [§VI-B, §VII-E] The RQ3 conclusions are explicitly limited to relative trends on a frequency- and core-capped desktop platform, yet the abstract and conclusion state that 'tightening the resource envelope... penalizes larger models the most' as a general finding. Whether this ordering survives on actual edge/mobile hardware (ARM memory bandwidth, thermal throttling, GPU/NPU presence) is untested. Please validate on at least one representative edge device, or else restrict the claim in the abstract/conclusion to the constrained desktop platform and clearly state that transfer to edge hardware is a hypothesis.
minor comments (5)
  1. [§VII-A vs. Supplementary S.I] The main text defines T/tok as normalized by total input+output tokens, while Supplementary Table S.I defines T/tok as 'time per output token.' Please unify the definition.
  2. [Supplementary figures] Several regression equations in the supplementary material contain a '□' placeholder where a minus sign should be (e.g., 'y = 1.304x □ 0.091'). These appear to be rendering artifacts and should be corrected.
  3. [Table VIII caption] The caption says models in parentheses are 'not significantly different from the best,' but the statistical test and adjustment are not described there. Please add the test name and note that McNemar p-values are uncorrected or specify the correction.
  4. [Table II] Envelope E1 lists two logical CPU IDs (2–3) with one P-core; clarify whether these are two hardware threads of a single physical core and whether Ollama was pinned to one or two threads.
  5. [General] The phrase 'statistically comparable' is used for models that are not significantly different from the best on a metric; consider using 'not significantly different' throughout to avoid implying equivalence.

Circularity Check

0 steps flagged

No circularity: PTME is a measurement protocol and the results are direct empirical observations, not fitted or self-referential claims.

full rationale

The paper's claims are supported by direct measurements rather than by a derivation that reduces to its own inputs. PTME metrics are defined by direct measurement equations (Eqs. 1-4: precision as benchmark correctness, time as wall-clock, memory as peak RSS, energy via RAPL), and the results are descriptive analyses of those measured values. RQ1's regression lines and Pearson correlations (Section VII-B, Table VII) are in-sample descriptive summaries; the phrase 'fail to predict precision' is a shorthand for low observed correlation and is not a case where a parameter is fitted to a target and then relabeled as a prediction. RQ2's Pareto sets (Section VII-C, Table VIII) are computed from the measured metrics using the standard definition of dominance; they are analyses of the data, not assumptions smuggled in to produce the conclusion. RQ3 (Section VII-D) compares measured values across envelopes, and the fact that precision is identical across envelopes confirms deterministic decoding rather than assuming envelope-invariance. The self-citations [13]-[15] for the PTME framework are background context and do not carry the LLM-specific empirical findings, which are supported by the paper's own measurements. The paper explicitly acknowledges its limitations, including subset sampling (Section VI-C: 'Since the objective of this study is not to provide a fine-grained performance ranking of LLMs...') and desktop-only envelopes (Section VII-E: 'Our conclusions are therefore limited to relative trends...'), and these are validity/statistical-power concerns rather than circularity. No self-definitional step, no fitted-input-called-prediction, no load-bearing self-citation chain, and no renaming of a known result as a new derivation was found.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

No fitted mathematical model is used; all support comes from direct measurements. The key assumptions are environmental (desktop caps represent edge devices, RAPL captures relevant energy, subsets represent full benchmarks) and therefore domain assumptions rather than standard math or ad hoc constructs.

axioms (5)
  • domain assumption RAPL CPU-package energy counters capture the energy cost relevant to inference.
    Used in Eq. 4 (E = E_pkg(t_end) - E_pkg(t_start)); excludes DRAM and peripherals, with the caveat acknowledged in Section VII-E.
  • domain assumption Constrained desktop caps reproduce the key resource limits of edge/mobile devices.
    Section VI-B asserts this to justify the five envelopes; Section VII-E limits conclusions to relative trends. Transferability to real devices is assumed.
  • domain assumption Random subsets (GSM8K 200/1319, MMLU-Pro 280/12032) reliably estimate full-benchmark precision.
    Section VI-C cites tinybenchmarks [30] to justify small subsets; the precision numbers feeding the RQ1 correlations and Pareto sets depend on this.
  • domain assumption Each model's default Ollama quantization (Q4_0 vs Q4_K_M) represents practical local deployment.
    Section II states models use their default Ollama quantization; different quantizations affect both cost and accuracy but match realistic use.
  • domain assumption Deterministic greedy decoding (temperature=0) makes precision invariant across envelopes.
    Section VII-D/RQ3 states precision and token counts are identical across E1–E5; this is verified in the reported tables and required for the 'envelope affects cost, not precision' claim.

pith-pipeline@v1.3.0-alltime-deepseek · 39060 in / 11561 out tokens · 113767 ms · 2026-08-01T09:18:12.785729+00:00 · methodology

0 comments
read the original abstract

Lightweight large language models (LLMs) are increasingly being deployed locally on personal computers and are expected to play a growing role in resource-constrained edge and mobile environments. In such settings, energy consumption, execution time, and memory usage directly affect practical usability, yet existing evaluations of LLM efficiency largely rely on proxy descriptors such as parameter count or FLOPs, often decoupled from task precision. This paper introduces a PTME-based experimental framework for the precision-aware profiling of lightweight LLM inference, jointly measuring Precision, execution Time, peak Memory usage, and Energy consumption through direct hardware-level measurements. The methodology is applied to a representative set of lightweight LLMs executed locally under edge-class resource envelopes on a controlled desktop platform, using benchmarks spanning code generation, mathematical reasoning, and multi-task understanding. We find that static proxy descriptors approximate inference cost well but fail to predict precision. Tightening the resource envelope increases cost without affecting precision, amplifying execution time more strongly than energy and penalizing larger models the most. Moreover, no single model dominates across all PTME dimensions, and a Pareto analysis reveals non-dominated configurations that would be hidden by accuracy-only or efficiency-only assessments, providing practical guidance for selecting models under different resource envelopes. These results show that selecting lightweight LLMs by size, FLOPs, latency, or accuracy alone can select the wrong deployment candidate; PTME profiling exposes configurations that preserve useful accuracy at lower physical cost.

Figures

Figures reproduced from arXiv: 2607.20806 by Enrique Alba, Gabriel Luque, Tomohiro Harada.

Figure 1
Figure 1. Figure 1: Relationship between total tokens and time, memory, and energy metrics on envelope E5 across all benchmarks [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Relationship between proxies and precision, time, memory, and energy on envelope E5 across all benchmarks [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Relation between precision and energy/token, time/token, and memory metrics under envelope E1 across all benchmarks [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Relation between precision and energy/token, time/token, and memory metrics under envelope E5 across all benchmarks [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: 3D visualizations of precision, time per token, and [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Ratios of token-normalized time, energy, and TTFT relative to E1 across resource envelopes, averaged across benchmarks [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Precision, time per token, and energy per token 3D plots for each model under different envelopes across all benchmarks [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 9 linked inside Pith

  1. [1]

    Energy and Policy Consid- erations for Deep Learning in NLP,

    E. Strubell, A. Ganesh, and A. McCallum, “Energy and Policy Consid- erations for Deep Learning in NLP,” inProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, A. Korhonen, D. Traum, and L. M `arquez, Eds. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 3645–3650

    2019

  2. [2]

    Green AI,

    R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni, “Green AI,” Commun. ACM, vol. 63, no. 12, p. 54–63, Nov. 2020

    2020

  3. [3]

    A Comprehensive Survey on TinyML,

    Y . Abadade, A. Temouden, H. Bamoumen, N. Benamar, Y . Chtouki, and A. S. Hafid, “A Comprehensive Survey on TinyML,”IEEE Access, vol. 11, pp. 96 892–96 922, 2023

    2023

  4. [4]

    On-Device Language Models: A Comprehensive Review,

    J. Xu, Z. Li, W. Chen, Q. Wang, X. Gao, Q. Cai, and Z. Ling, “On-Device Language Models: A Comprehensive Review,” 2024. [Online]. Available: https://arxiv.org/abs/2409.00088

    Pith/arXiv arXiv 2024

  5. [5]

    Privacy issues in Large Language Models: A survey,

    H. Kibriya, W. Z. Khan, A. Siddiqa, and M. K. Khan, “Privacy issues in Large Language Models: A survey,”Computers and Electrical Engineering, vol. 120, p. 109698, 2024

    2024

  6. [6]

    Edge Computing: Vision and Challenges,

    W. Shi, J. Cao, Q. Zhang, Y . Li, and L. Xu, “Edge Computing: Vision and Challenges,”IEEE Internet of Things Journal, vol. 3, no. 5, pp. 637–646, 2016

    2016

  7. [7]

    The Emergence of Edge Computing,

    M. Satyanarayanan, “The Emergence of Edge Computing,”Computer, vol. 50, no. 1, pp. 30–39, 2017

    2017

  8. [8]

    LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale,

    T. Dettmers, M. Lewis, Y . Belkada, and L. Zettlemoyer, “LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale,” inAdvances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 30 318–30 332

    2022

  9. [9]

    GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers,

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers,”

  10. [10]

    LLM-Pruner: On the Structural Pruning of Large Language Models,

    X. Ma, G. Fang, and X. Wang, “LLM-Pruner: On the Structural Pruning of Large Language Models,” inAdvances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 21 702–21 720

    2023

  11. [11]

    A Review on Edge Large Language Models: Design, Execution, and Applications,

    Y . Zheng, Y . Chen, B. Qian, X. Shi, Y . Shu, and J. Chen, “A Review on Edge Large Language Models: Design, Execution, and Applications,” ACM Computing Surveys, vol. 57, no. 8, Mar. 2025

    2025

  12. [12]

    1.1 Computing’s energy problem (and what we can do about it),

    M. Horowitz, “1.1 Computing’s energy problem (and what we can do about it),” in2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), 2014, pp. 10–14

    2014

  13. [13]

    Energy-aware metaheuristics,

    E. Alba, T. Harada, and G. Luque, “Energy-aware metaheuristics,” in Proceedings of the Genetic and Evolutionary Computation Conference, ser. GECCO ’26. New York, NY , USA: Association for Computing Machinery, 2026, p. 672–680

    2026

  14. [14]

    Green optimization: Energy-aware design of metaheuristics by using machine learning surrogates to cope with real problems,

    T. Harada, E. Alba, and G. Luque, “Green optimization: Energy-aware design of metaheuristics by using machine learning surrogates to cope with real problems,”Future Generation Computer Systems, vol. 182, p. 108482, 2026

    2026

  15. [15]

    White-box execution refactoring of transformers for lower energy,

    E. Alba and H. D. Menendez, “White-box execution refactoring of transformers for lower energy,” inSearch-Based Software Engineering, ser. Lecture Notes in Computer Science. Springer, 2026, In press, SSBSE 2026 SSBSE Challenge track

    2026

  16. [16]

    LLaMA: Open and Efficient Foundation Language Models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “LLaMA: Open and Efficient Foundation Language Models,” 2023. [Online]. Available: https: //arxiv.org/abs/2302.13971

    Pith/arXiv arXiv 2023

  17. [17]

    Mistral 7B,

    A. Q. Jiang, A. Sablayrolleset al., “Mistral 7B,” 2023. [Online]. Available: https://arxiv.org/abs/2310.06825

    Pith/arXiv arXiv 2023

  18. [18]

    TinyLlama: An Open-Source Small Language Model,

    P. Zhang, G. Zeng, T. Wang, and W. Lu, “TinyLlama: An Open-Source Small Language Model,” 2024. [Online]. Available: https://arxiv.org/abs/2401.02385

    Pith/arXiv arXiv 2024

  19. [19]

    Qwen2.5 Technical Report,

    Qwen Team, “Qwen2.5 Technical Report,” 2025. [Online]. Available: https://arxiv.org/abs/2412.15115

    Pith/arXiv arXiv 2025

  20. [20]

    Gemma: Open models based on gemini research and technology,

    Gemma Team, “Gemma: Open models based on gemini research and technology,” 2024. [Online]. Available: https://arxiv.org/abs/2403.08295

    Pith/arXiv arXiv 2024

  21. [21]

    Phi-2: The Surprising Power of Small Language Models,

    Microsoft Research, “Phi-2: The Surprising Power of Small Language Models,” https://www.microsoft.com/en-us/research/blog/ phi-2-the-surprising-power-of-small-language-models/, 2023, accessed: 2026-06-08

    2023

  22. [22]

    Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone,

    M. Abdin, J. Aneja, H. Awadalla, A. Awadallah, A. A. Awanet al., “Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone,” 2024. [Online]. Available: https://arxiv.org/abs/2404.14219

    Pith/arXiv arXiv 2024

  23. [23]

    Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning,

    P. Henderson, J. Hu, J. Romoff, E. Brunskill, D. Jurafsky, and J. Pineau, “Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning,”Journal of Machine Learning Research, vol. 21, no. 248, pp. 1–43, 2020

    2020

  24. [24]

    Evaluating Large Language Models Trained on Code,

    M. Chen, J. Tworek, H. Junet al., “Evaluating Large Language Models Trained on Code,” 2021. [Online]. Available: https://arxiv.org/abs/2107. 03374

    2021

  25. [25]

    Training Verifiers to Solve Math Word Problems,

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, “Training Verifiers to Solve Math Word Problems,” 2021. [Online]. Available: https://arxiv.org/abs/2110.14168

    Pith/arXiv arXiv 2021

  26. [26]

    MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark,

    Y . Wang, X. Ma, G. Zhang, Niet al., “MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark,” in Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, Eds., vol. 37. Curran Associates, Inc., 2024, pp. 95 266–95 290

    2024

  27. [27]

    Measuring Massive Multitask Language Understanding,

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring Massive Multitask Language Understanding,” inInternational Conference on Learning Representations, 2021

    2021

  28. [28]

    A Survey on Edge Performance Benchmarking,

    B. Varghese, N. Wang, D. Bermbach, C.-H. Hong, E. D. Lara, W. Shi, and C. Stewart, “A Survey on Edge Performance Benchmarking,”ACM Computing Surveys, vol. 54, no. 3, pp. 1–33, Apr. 2021

    2021

  29. [29]

    Intelligent Edge Computing and Machine Learning: A Survey of Optimization and Applications,

    S. A. Cajas Ordo ˜nez, J. Samanta, A. L. Su ´arez-Cetrulo, and R. S. Carbajo, “Intelligent Edge Computing and Machine Learning: A Survey of Optimization and Applications,”Future Internet, vol. 17, no. 9, 2025

    2025

  30. [30]

    tinybenchmarks: evaluating LLMs with fewer examples,

    F. M. Polo, L. Weber, L. Choshen, Y . Sun, G. Xu, and M. Yurochkin, “tinybenchmarks: evaluating LLMs with fewer examples,” inForty-first International Conference on Machine Learning, 2024

    2024

  31. [31]

    THOP: PyTorch-OpCounter,

    Ultralytics, “THOP: PyTorch-OpCounter,” https://github.com/ultralytics/ thop, 2025, accessed: 2026-06-08. PREPRINT 1 Supplementary material for “Profiling Lightweight Large Language Models” Tomohiro Harada,Member , IEEE,Enrique Alba,Member , IEEE,and Gabriel Luque,Member , IEEE Abstract This document is supplementary material for “Profiling Lightweight La...

    arXiv 2025

  32. [2023]

    Available: https://arxiv.org/abs/2210.17323

    [Online]. Available: https://arxiv.org/abs/2210.17323

    Pith/arXiv arXiv