Pith. sign in

REVIEW 3 major objections 5 minor 35 references

A backend dispatcher that remembers the previously selected execution backend reduces replayed edge LLM inference latency by 17.4% on average, with corresponding energy and energy-delay-product gains.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 18:00 UTC pith:QZOC67LF

load-bearing objection Honest, well-scoped replay study with a genuinely new transition-aware framing—but the headline gains are in-sample upper bounds until an integrated runtime validates them. the 3 major comments →

arxiv 2607.17415 v1 pith:QZOC67LF submitted 2026-07-19 cs.AR

Transition-Aware Backend Dispatch for Edge LLM Inference

Alaaddin Goktug Ayar , Martin Margala This is my paper
classification cs.AR
keywords edge intelligencelarge language modelsinference schedulingheterogeneous systemsbackend dispatchtrace replayenergy-delay productbackend transitions
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that backend dispatch for edge transformer inference should be transition-aware: instead of picking the cheapest backend for each operator in isolation, the dispatcher should condition on the backend used by the preceding operator and pay the measured switch penalty as a cost. Because operator performance on an NVIDIA Jetson platform varies with exact tensor shape, static assignment leaves latency and energy on the table, while operator-local selection pays too much in backend switches. Combining current operator features with the previous backend state, the approach cuts replayed latency, energy, and energy-delay product by 17.4%, 14.4%, and 28.5% on average relative to the best static backend across seven transformer model traces (9,584 ordered operator instances), while also reducing backend, device, and framework switches relative to operator-local selection. Leave-one-model-out evaluation transfers to held-out models for six of seven on all objectives and to all seven on energy.

Core claim

The central claim is that adding the previous backend as a dispatch feature, and evaluating each candidate backend using the sum of its measured operator cost and a directed transition penalty, produces better selective backend dispatch for edge transformer workloads than either a fixed backend or independent per-operator choice. This is demonstrated through measurement-backed trace replay: 278 exact-shape groups from seven full-model traces are benchmarked on PyTorch eager CPU, PyTorch eager CUDA, and ONNX Runtime CPU on Jetson, and 9,584 ordered instances are replayed with measured per-operator and transition costs. On average, transition-aware dispatch lowers replayed latency, energy, and

What carries the argument

The central mechanism is the transition-aware cost model C_i(b_i) = C_op_i(b_i) + C_trans_i(b_{i-1}, b_i): the total cost of assigning operator i to backend b_i is the measured per-shape operator cost plus a directed transition penalty that is zero when the backend is unchanged. A classifier (histogram-based gradient boosting in the reported results) is trained on operator features such as operator type, phase, sequence length, hidden size, matrix dimensions, and output bytes, plus the previous backend, to predict the lowest-cost candidate. The resulting policy is then evaluated via measurement-backed trace replay that preserves model-derived operator order while isolating the effect of back

Load-bearing premise

The load-bearing assumption is that the cost of a backend switch for a supported operator is exactly the measured directed-pair penalty, and that these costs add together with no extra overhead from an integrated dispatcher or from unsupported operators between supported ones.

What would settle it

Build an integrated runtime that actually executes a mixed-backend model on Jetson and compare end-to-end latency and energy to the replay prediction; if the measured end-to-end saving versus the best static backend is materially below the replayed 17.4% and 14.4%, or if realized transition costs diverge from the measured pair penalties in real execution, the additive replay model is wrong.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Deployed edge systems can exploit shape-dependent backend preferences by changing per-operator backend at runtime, without needing to retrain or recompile the model graph.
  • A learned selector can reach within 0.41% regret of the best possible backend choice under the measured-cost model, so cheap learned dispatch is nearly as good as an oracle for these workloads.
  • Because transition-aware dispatch cuts backend switches by about 14% versus operator-local selection, mixed-framework execution pipelines become more predictable in terms of synchronization and data movement.
  • The leave-one-model-out results suggest that a dispatcher trained on one set of decoder-style transformer models can be reused for new models with similar shape distributions, reducing per-model benchmarking effort.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the replay excludes integrated dispatcher and runtime overhead, real deployments are likely to see smaller net gains; the reported percentages are upper bounds unless per-call dispatch cost is negligible.
  • The same previous-backend conditioning could apply to other expensive transition points, such as precision or quantization-level switches, where the cost of changing data formats is analogous to a backend transition.
  • If directed transition penalties generalize across shapes for the same backend pair, a compact transition table could replace exhaustive per-shape benchmarking, making the approach feasible for models with many more shapes.
  • The negative ModernBERT transfer implies that training sets need encoder-shaped workloads; a dispatcher trained only on decoder traces may systematically mis-select for bidirectional encoders.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a transition-aware backend dispatch policy for edge transformer inference. It collects ordered operator traces from seven transformer models, benchmarks four operator classes across PyTorch CPU, PyTorch CUDA, and ONNX Runtime CPU on an NVIDIA Jetson Orin Nano, and trains selectors that use current operator features plus the previous backend. The policies are evaluated through measurement-backed trace replay: replayed latency, energy, and EDP are computed by summing measured per-operator costs and directed transition penalties over the 9,584 supported operator instances. The central quantitative claim is that transition-aware dispatch reduces replayed latency, energy, and EDP by 17.4%, 14.4%, and 28.5% on average relative to the best static backend, while reducing backend switches relative to operator-local selection. Leave-one-model-out evaluation is also reported.

Significance. If the replay-based results transferred to a real integrated runtime, the work would offer a practical, measurement-driven method for exploiting shape-dependent backend performance on heterogeneous edge platforms. The paper is unusually transparent about the scope of its evaluation: Sec. IV-B explicitly states the numbers 'do not represent end-to-end measurements from an integrated mixed-backend runtime,' and Sec. III-E excludes integrated dispatcher and model-level overheads. The released artifact and the detailed measurement methodology (exact-shape grouping, correctness validation, adaptive repetition) are strengths that support reproducibility. However, the significance is currently tempered by the gap between the replay model and real end-to-end mixed-backend execution, and by the fact that the headline gains are computed on shapes used to train the selectors.

major comments (3)
  1. [Sec. IV-B and Sec. III-D/III-E] The headline 17.4% latency gain (and the energy/EDP gains) is computed on traces whose constituent shapes were used to train the selectors. The evaluation protocol in Sec. III-G reports gains over static for the same seven models used in training; the final HistGBM models are selected by grouped cross-validation, but the replay itself is in-sample. The only out-of-sample evidence is the leave-one-model-out evaluation in Table II, where ModernBERT-base exhibits a -15.49% latency and -6.10% EDP gain. The paper should either report the main result on a held-out split of shape groups/models or clearly label the 17.4% gain as a training-fit upper bound, and discuss why the in-sample result is not sufficient to support the abstract's unqualified 'demonstrate' claim.
  2. [Sec. III-E, Eq. (1)-(2)] The replay model L_T = sum_i [L_i(b_i) + L_trans(b_{i-1}, b_i)] covers only the 9,584 supported operator instances. Unsupported operators (attention, embeddings, softmax, etc.) retain static assignment and contribute no dynamic cost. In a real model, switching backends at supported/unsupported boundaries would incur data movement and tensor-format conversion costs that are neither measured nor added. The paper explicitly excludes these costs, so the reported gains are upper bounds on the dispatch policy's contribution, not on end-to-end inference improvement. To make the practical claim about 'edge LLM inference' convincing, the paper should add a sensitivity analysis estimating the magnitude of boundary conversion costs, or implement at least a proof-of-concept integrated runtime to measure one end-to-end trace.
  3. [Sec. III-A and Sec. IV-D] The extracted feature vector includes the model name (Sec. III-A: 'Extracted features include model name, phase, ...'). This raises two problems. First, the main in-sample gains may partly reflect memorizing per-model backend preferences rather than learning shape-dependent transferable structure. Second, in the leave-one-model-out experiment the held-out model has a previously unseen model-name value; for tree-based models, this unseen categorical may be mapped arbitrarily, which could contribute to the poor ModernBERT-base result. The paper should either exclude model name from the features, show the effect of including it by ablation, or explain how the selectors handle novel model names.
minor comments (5)
  1. [Abstract and Sec. I] The abstract uses 'demonstrate' without the 'replayed' qualifier in the final sentence; consider adding 'in replay' or 'under the measured-cost replay model' to the abstract to match the body's careful phrasing in Sec. IV-B.
  2. [Sec. III-D] The transition penalty is described as 'estimated from directed backend-pair measurements' but the exact measurement procedure is not fully specified. Clarify how the pair measurement is separated into per-operator transition cost, and whether the transition penalty is independent of the operator shape. This would help reproducibility.
  3. [Sec. III-C] The power formula uses P_net = P_active - P_idle. It would be helpful to state whether P_active and P_idle are measured over the same time window and how the 50 ms tegrastats sampling interacts with very short operator runs. The adaptive repetition is mentioned, but a few more details on the idle baseline would strengthen the energy methodology.
  4. [Table I] The accuracy values for operator-local and transition-aware selectors are computed on different decision contexts (278 vs 834), and the paper notes they are not directly comparable. This is fine, but consider adding a sentence in the caption to avoid readers misinterpreting the higher transition-aware accuracy as a uniform improvement.
  5. [Fig. 4] The figure shows 'average replay gain' and 'switch reduction.' The y-axis labels are not fully legible in the provided text; ensure the axes are clearly labeled in the camera-ready version. Also define 'switch reduction' (relative to operator-local) in the caption.

Circularity Check

1 steps flagged

Main replay gains are in-sample evaluations of the selector's own training objective; leave-one-model-out provides the only independent check.

specific steps
  1. fitted input called prediction [Sec. III-D (Eq. 1) and Sec. III-E (Eqs. 2-3)]
    "Sec. III-D: "The transition-aware selector additionally receives previous backend b_{i-1} and evaluates candidate backend b_i using C_i(b_i)=C^op_i(b_i)+C^trans_i(b_{i-1}, b_i)" and "The lowest-cost candidate for each shape and previous-backend state defines the label." Sec. III-E: "replayed latency and energy are L_T = sum_i [L_i(b_i)+L^trans_i(b_{i-1},b_i)], E_T = sum_i [E_i(b_i)+E^trans_i(b_{i-1},b_i)].""

    The transition-aware selector is trained on labels defined as the argmin of C_i(b_i) = C^op_i(b_i) + C^trans_i(b_{i-1}, b_i), and the replay total is the sum of exactly the same measured operator cost plus the same directed transition penalty. The main replay gain is therefore an in-sample evaluation of the training objective: it measures how well the fitted classifier reproduces the table's argmin, not an independent prediction of runtime speedup. The paper's caveat that the values "do not represent end-to-end measurements from an integrated mixed-backend runtime" confirms this is a modeled potential. The leave-one-model-out replay is the only held-out check and is weaker (latency/EDP improve for 6/7, ModernBERT negative).

full rationale

This is a borderline case. The paper is transparent: it labels the evaluation "measurement-backed trace replay" and explicitly excludes integrated dispatcher overhead (Sec. III-E) and end-to-end integration (Sec. IV-B). Self-citations ([6], [17], [23]) are not load-bearing; no uniqueness theorem or ansatz is imported from prior work. The circular component is the identity between the selector's training objective and the replay metric: Eq. 1 in Sec. III-D defines candidate cost as measured operator cost plus transition penalty, and Eqs. 2-3 in Sec. III-E sum those same quantities. Thus the headline 17.4%/14.4%/28.5% gains are the fitted policy's in-sample improvement on the table it was trained to minimize, not an external validation. The leave-one-model-out results provide the independent content: they use held-out model traces (though still the same measured-cost table as ground truth) and show positive transfer for 6/7 (latency/EDP) and 7/7 (energy), with ModernBERT negative, demonstrating the check is non-vacuous. Because the central headline is partially forced by the training/evaluation identity, score 6; the independent held-out results prevent a higher score.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The paper adds no new physical entities. Its central claim rests on domain assumptions about the faithfulness of the replay model: additivity of measured costs, independence of unsupported operators, representativeness of the one-step decode trace, and measurement fidelity of the power sampling.

free parameters (1)
  • Selector model type and hyperparameters = HistGBM; 200 estimators, min leaf 1, seed 42
    Chosen by grouped validation regret (Table I). These choices influence reported accuracy/gains but are standard ML hyperparameters rather than physically fit constants.
axioms (4)
  • domain assumption Measured operator and transition costs are additive and generalize to integrated full-model execution.
    Replay sums per-operator costs and transition penalties; integrated dispatcher/model overheads are explicitly excluded (Sec. III-E). If real overheads differ, the headline gains change.
  • domain assumption Unsupported operators' costs are independent of the backend choices made for supported operators.
    Sec. III-E states static assignments for unsupported operators are common and not modified; however, in a real mixed-backend graph, data placement may create additional transfers between supported and unsupported operators.
  • domain assumption A single length-1 decode forward call represents decode-phase operator shapes.
    Sec. III-A says the decode trace contains one length-1 forward call rather than a complete autoregressive KV-cache loop; KV-cache effects on shape and cost are absent.
  • domain assumption Power sampling at 50 ms intervals with baseline correction yields accurate energy values.
    Sec. III-F describes adaptive repetition for short operators to obtain sufficient samples, but no uncertainty is propagated to the reported gain percentages.

pith-pipeline@v1.3.0-alltime-deepseek · 11162 in / 11645 out tokens · 109248 ms · 2026-08-01T18:00:23.549009+00:00 · methodology

0 comments
read the original abstract

Efficient large language model (LLM) inference on edge platforms is limited not only by model size, but also by shape-dependent performance differences across execution backends. Static backend assignment cannot exploit this variation, while independent per-operator selection can introduce costly device and framework switches. This paper presents a transition-aware backend dispatch approach for edge transformer inference. The approach combines current operator features with the previously selected backend to preserve beneficial shape-specific choices while avoiding unnecessary transitions. Ordered traces are collected from full-model inference runs of seven transformer models, and four common operator classes are benchmarked across PyTorch eager CPU, PyTorch eager CUDA, and ONNX Runtime CPU on an NVIDIA Jetson platform. The dispatch policies are evaluated through measurement-backed trace replay using observed operator costs and transition costs measured from actual backend switches. Supported operators are selected dynamically, while operators outside the dispatch scope retain a static assignment. Across 9,584 ordered operator instances and 278 exact shape groups, transition-aware dispatch reduces replayed latency, energy, and energy-delay product relative to the best static policy by 17.4%, 14.4%, and 28.5% on average, respectively. It also reduces switching relative to operator-local selection. Leave-one-model-out evaluation improves all three objectives for six of seven held-out models and improves energy for all seven. These results demonstrate that incorporating operator shape and backend-transition context can improve selective backend dispatch for edge transformer workloads.

Figures

Figures reproduced from arXiv: 2607.17415 by Alaaddin Goktug Ayar, Martin Margala.

Figure 1
Figure 1. Figure 1: Overview of the proposed transition-aware backend dispatch framework, from full-model inference trace collection to exact-shape benchmarking, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of static backend assignment, operator-local selection, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Transition-aware replay gains over the best static backend across seven [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ablation of transition-aware dispatch against operator-local selection, [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 15 linked inside Pith

  1. [1]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” inAdvances in Neural Information Processing Systems, I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017. [Online]. Available: https://pr...

    2017

  2. [2]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” 2023. [Online]. Available: https: //arxiv.org/abs/2302.13971

    Pith/arXiv arXiv 2023

  3. [3]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” 2023. [Online]. Available: https://arxiv.org/abs/2309.06180

    Pith/arXiv arXiv 2023

  4. [4]

    Powerinfer: Fast large language model serving with a consumer-grade gpu,

    Y . Song, Z. Mi, H. Xie, and H. Chen, “Powerinfer: Fast large language model serving with a consumer-grade gpu,” 2024. [Online]. Available: https://arxiv.org/abs/2312.12456

    Pith/arXiv arXiv 2024

  5. [5]

    Powerinfer-2: Fast large language model inference on a smartphone,

    Z. Xue, Y . Song, Z. Mi, X. Zheng, Y . Xia, and H. Chen, “Powerinfer-2: Fast large language model inference on a smartphone,” 2024. [Online]. Available: https://arxiv.org/abs/2406.06282

    Pith/arXiv arXiv 2024

  6. [6]

    Word2hypervec: From word embeddings to hypervectors for hyperdimensional computing,

    A. G. Ayar, S. Aygun, M. H. Najafi, and M. Margala, “Word2hypervec: From word embeddings to hypervectors for hyperdimensional computing,” inProceedings of the Great Lakes Symposium on VLSI 2024, ser. GLSVLSI ’24. New York, NY , USA: Association for Computing Machinery, 2024, p. 355–356. [Online]. Available: https://doi.org/10.1145/3649476.3658795

    arXiv 2024

  7. [7]

    Deepx: a software accelerator for low-power deep learning inference on mobile devices,

    N. D. Lane, S. Bhattacharya, P. Georgiev, C. Forlivesi, L. Jiao, L. Qen- dro, and F. Kawsar, “Deepx: a software accelerator for low-power deep learning inference on mobile devices,” inProceedings of the 15th International Conference on Information Processing in Sensor Networks, ser. IPSN ’16. IEEE Press, 2016

    2016

  8. [8]

    Deepmon: Mobile gpu-based deep learning framework for continuous vision applications,

    L. N. Huynh, Y . Lee, and R. K. Balan, “Deepmon: Mobile gpu-based deep learning framework for continuous vision applications,” in Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services, ser. MobiSys ’17. New York, NY , USA: Association for Computing Machinery, 2017, p. 82–95. [Online]. Available: https://doi....

    arXiv 2017

  9. [9]

    Neurosurgeon: Collaborative intelligence between the cloud and mobile edge,

    Y . Kang, J. Hauswald, C. Gao, A. Rovinski, T. Mudge, J. Mars, and L. Tang, “Neurosurgeon: Collaborative intelligence between the cloud and mobile edge,”SIGPLAN Not., vol. 52, no. 4, p. 615–629, Apr

  10. [10]

    Axonn: energy-aware execution of neural network inference on multi-accelerator heterogeneous socs,

    I. Dagli, A. Cieslewicz, J. McClurg, and M. E. Belviranli, “Axonn: energy-aware execution of neural network inference on multi-accelerator heterogeneous socs,” inProceedings of the 59th ACM/IEEE Design Automation Conference, ser. DAC ’22. New York, NY , USA: Association for Computing Machinery, 2022, p. 1069–1074. [Online]. Available: https://doi.org/10.1...

    arXiv 2022

  11. [11]

    Tvm: an automated end-to-end optimizing compiler for deep learning,

    T. Chenet al., “Tvm: an automated end-to-end optimizing compiler for deep learning,” inProceedings of the 13th USENIX Conference on Operating Systems Design and Implementation, ser. OSDI’18. USA: USENIX Association, 2018, p. 579–594

    2018

  12. [12]

    Ansor: generating high-performance tensor programs for deep learning,

    L. Zhenget al., “Ansor: generating high-performance tensor programs for deep learning,” inProceedings of the 14th USENIX Conference on Operating Systems Design and Implementation, ser. OSDI’20. USA: USENIX Association, 2020

    2020

  13. [13]

    Tensor program optimization with probabilistic programs,

    J. Shao, X. Zhou, S. Feng, B. Hou, R. Lai, H. Jin, W. Lin, M. Masuda, C. H. Yu, and T. Chen, “Tensor program optimization with probabilistic programs,” inAdvances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 35 783–35 796. [Online]. Available:...

    2022

  14. [14]

    Slo-aware inference scheduler for heterogeneous processors in edge platforms,

    W. Seo, S. Cha, Y . Kim, J. Huh, and J. Park, “Slo-aware inference scheduler for heterogeneous processors in edge platforms,”ACM Trans. Archit. Code Optim., vol. 18, no. 4, Jul. 2021. [Online]. Available: https://doi.org/10.1145/3460352

    doi:10.1145/3460352 2021

  15. [15]

    Lalarand: Flexible layer-by-layer cpu/gpu scheduling for real-time dnn tasks,

    W. Kang, K. Lee, J. Lee, I. Shin, and H. S. Chwa, “Lalarand: Flexible layer-by-layer cpu/gpu scheduling for real-time dnn tasks,” in2021 IEEE Real-Time Systems Symposium (RTSS), 2021, pp. 329–341

    2021

  16. [16]

    Band: coordinated multi-dnn inference on heterogeneous mobile processors,

    J. S. Jeong, J. Lee, D. Kim, C. Jeon, C. Jeong, Y . Lee, and B.-G. Chun, “Band: coordinated multi-dnn inference on heterogeneous mobile processors,” inProceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services, ser. MobiSys ’22. New York, NY , USA: Association for Computing Machinery, 2022, p. 235–247. [Online]. A...

    arXiv 2022

  17. [17]

    Power-aware edge inference public repository,

    A. Goktug Ayar, Martin Margala, “Power-aware edge inference public repository,” 2026. [Online]. Available: https://anonymous.4open.science/ r/power aware edge inference public-3B71/README.md

    2026

  18. [18]

    Orca: A distributed serving system for transformer-based generative models,

    G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A distributed serving system for transformer-based generative models,” in16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). Carlsbad, CA: USENIX Association, Jul. 2022, pp. 521–538. [Online]. Available: https://www.usenix.org/ conference/osdi22/presentation/yu

    2022

  19. [19]

    Deepspeed-inference: enabling efficient in- ference of transformer models at unprecedented scale,

    R. Y . Aminabadiet al., “Deepspeed-inference: enabling efficient in- ference of transformer models at unprecedented scale,” inProceedings of the International Conference on High Performance Computing, Net- working, Storage and Analysis, ser. SC ’22. IEEE Press, 2022

    2022

  20. [20]

    Flexgen: High-throughput generative inference of large language models with a single gpu,

    Y . Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, D. Y . Fu, Z. Xie, B. Chen, C. Barrett, J. E. Gonzalez, P. Liang, C. R ´e, I. Stoica, and C. Zhang, “Flexgen: High-throughput generative inference of large language models with a single gpu,” 2023. [Online]. Available: https://arxiv.org/abs/2303.06865

    Pith/arXiv arXiv 2023

  21. [21]

    Llm in a flash: Efficient large language model inference with limited memory,

    K. Alizadehet al., “Llm in a flash: Efficient large language model inference with limited memory,” 2024. [Online]. Available: https://arxiv.org/abs/2312.11514

    Pith/arXiv arXiv 2024

  22. [22]

    Edgeshard: Efficient llm inference via collaborative edge computing,

    M. Zhang, J. Cao, X. Shen, and Z. Cui, “Edgeshard: Efficient llm inference via collaborative edge computing,” 2024. [Online]. Available: https://arxiv.org/abs/2405.14371

    Pith/arXiv arXiv 2024

  23. [23]

    Real-time hy- perdimensional computing for lightweight language model classifiers,

    A. G. Ayar, S. Aygun, M. H. Najafi, and M. Margala, “Real-time hy- perdimensional computing for lightweight language model classifiers,” in2026 IEEE 19th Dallas Circuits and Systems Conference (DCAS), 2026, pp. 1–4

    2026

  24. [24]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” 2019. [Online]. Available: https://arxiv.org/abs/1810.04805

    Pith/arXiv arXiv 2019

  25. [25]

    Learning to optimize tensor programs,

    T. Chenet al., “Learning to optimize tensor programs,” 2019. [Online]. Available: https://arxiv.org/abs/1805.08166

    Pith/arXiv arXiv 2019

  26. [26]

    Tensorir: An abstraction for automatic tensorized program optimization,

    S. Fenget al., “Tensorir: An abstraction for automatic tensorized program optimization,” inProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, V olume 2, ser. ASPLOS 2023. New York, NY , USA: Association for Computing Machinery, 2023, p. 804–817. [Online]. Available: https://doi.or...

    arXiv 2023

  27. [27]

    Taso: optimizing deep learning computation with automatic generation of graph substitutions,

    Z. Jia, O. Padon, J. Thomas, T. Warszawski, M. Zaharia, and A. Aiken, “Taso: optimizing deep learning computation with automatic generation of graph substitutions,” inProceedings of the 27th ACM Symposium on Operating Systems Principles, ser. SOSP ’19. New York, NY , USA: Association for Computing Machinery, 2019, p. 47–62. [Online]. Available: https://do...

    arXiv 2019

  28. [28]

    Rammer: Enabling holistic deep learning compiler optimizations with rtasks,

    L. Maet al., “Rammer: Enabling holistic deep learning compiler optimizations with rtasks,” inUSENIX Symposium on Operating Systems Design and Implementation, 2020. [Online]. Available: https://api.semanticscholar.org/CorpusID:228990152

    2020

  29. [29]

    Qwen2.5 Technical Report,

    Qwen Team, “Qwen2.5 Technical Report,”arXiv preprint arXiv:2412.15115, 2024. [Online]. Available: https://arxiv.org/abs/ 2412.15115

    Pith/arXiv arXiv 2024

  30. [31]

    Tinyllama: An open-source small language model,

    P. Zhang, G. Zeng, T. Wang, and W. Lu, “Tinyllama: An open-source small language model,” 2024. [Online]. Available: https://arxiv.org/abs/2401.02385

    Pith/arXiv arXiv 2024

  31. [34]

    Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference,

    B. Warneret al., “Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference,”arXiv preprint arXiv:2412.13663, 2024. [Online]. Available: https://arxiv.org/abs/2412.13663

    Pith/arXiv arXiv 2024

  32. [35]

    Available: https://arxiv.org/abs/2502.02737

    [Online]. Available: https://arxiv.org/abs/2502.02737

    Pith/arXiv arXiv

  33. [37]

    Available: https://arxiv.org/abs/2501.00656

    [Online]. Available: https://arxiv.org/abs/2501.00656

    Pith/arXiv arXiv

  34. [2017]

    Available: https://doi.org/10.1145/3093336.3037698

    [Online]. Available: https://doi.org/10.1145/3093336.3037698

    arXiv

  35. [2025]

    Available: https://arxiv.org/abs/2505.09388

    [Online]. Available: https://arxiv.org/abs/2505.09388

    Pith/arXiv arXiv