Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Predicting expert routing speeds MoE inference by 40 percent

desk verdict Real systems work with a genuine speedup, but the accuracy claim needs a caveat: GSM8K drops over 40% at low cache ratios, and the abstract hides it. read the letter →

arxiv 2501.10375 v2 pith:NUS4P5UE submitted 2024-12-16 cs.DC cs.LG

Yujie Zhang , Shivam Aggarwal , Tulika Mitra This is my paper · ORCID
classification cs.DCcs.LG
keywords Mixture-of-ExpertsGPU-CPUoffloadingexpertcachingpredictioninferenceaccelerationedgeMoEengine
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the biggest bottleneck in running Mixture-of-Experts (MoE) language models on memory-limited GPUs is not computation but the constant shuffling of expert weights between GPU and CPU. It introduces DAOP, an inference engine that watches which experts each input sequence activates during prefill, keeps the hot experts on the GPU, and pre-calculates the next layer's likely experts on the CPU one layer ahead. The two predictions together let CPU and GPU work in parallel, hiding transfer latency. The paper reports that DAOP boosts token generation over the Fiddler offloading baseline by 40.4% on Mixtral 8x7B and by up to 8.20x over caching/prefetching baselines, while keeping accuracy close to the official model on most benchmarks. If true, this would make large MoE models practical on ordinary workstations and edge devices.

What carries the argument

The load-bearing objects are two empirical regularities plus a scheduling algorithm. The first is the prefill-decode expert activation matrix $P$ and $D$, each of size $L \times E$ where $L$ is the number of MoE layers and $E$ the experts per layer; their row-wise cosine similarity, averaged over layers, is claimed to be ~90.72%. The second is the one-layer-ahead predictor: applying block $i+1$'s gating function to block $i$'s non-MoE hidden states yields ~84.11% accuracy. The algorithm itself is sequence-specific allocation: during prefill, count tokens per expert, form tuples of most-active CPU experts versus least-active GPU experts, and swap when the CPU expert's token count exceeds the GPU expert's by a threshold $\mathrm{SwapInOut}=1.05$. During decode, if the predicted next-layer experts are on CPU, send hidden states to CPU and pre-compute them while the GPU works on the current layer, with a graceful-degradation rule that replaces a low-score CPU expert by the best available GPU expert.

What would settle it

Measure the one-layer-ahead expert prediction accuracy and the prefill-decode activation similarity on a dataset of long, multi-step mathematical proofs (e.g., GSM8K-style problems with multi-token derivations) and correlate with DAOP's speedup and accuracy; if prediction accuracy falls below about 70% and similarity below about 80%, DAOP's advantage over Fiddler should shrink sharply and its accuracy gap should widen beyond the currently reported 25-point drop on GSM8K.

Watch

Extended reading notes

Core claim

The central discovery is that expert routing in MoE models is predictable enough to drive a scheduling policy. Using Mixtral 8x7B, the paper measures a 90.72% average cosine similarity between the expert activation matrices of prefill and decode phases across C4, MATH, and GSM8K, and an 84.11% average one-layer-ahead prediction accuracy when the next layer's gating function is applied to current hidden states. DAOP exploits these patterns by swapping experts between CPU and GPU during prefill based on per-sequence activation counts, and during decode, predicting the next layer's experts and pre-computing the CPU-resident ones with approximate hidden states, with a graceful-degradation fallback that substitutes a lower-scoring GPU expert when both predicted experts are on CPU. The empirical payoff is a 40.4% speedup over Fiddler on Mixtral 8x7B at [256,512] I/O length and up to 8.20x over expert caching/prefetching baselines, with an average 35.4% improvement across cache ratios.

Load-bearing premise

The fragile premise is that an expert's future use can be predicted from prefill activations and from one layer ahead; if routing is unstable, as the GSM8K results show, the cache placement and pre-computation decisions lose their value.

Editorial extensions

If this is right

  • If the one-layer-ahead prediction accuracy of 84.11% holds across diverse workloads, the same predictive pre-calculation can be applied to other top-2 MoE models, such as Phi-3.5 MoE, yielding comparable speedups.
  • The prefill-to-decode activation similarity means expert allocation decisions can be made once per sequence rather than per token, eliminating most cache-miss migration overhead during decoding.
  • DAOP's accuracy preservation on commonsense, world-knowledge, and summarization benchmarks suggests that approximate CPU execution of offloaded experts is a viable substitute for exact GPU execution at small cache ratios, down to 25% ECR.
  • The graceful-degradation mechanism provides a template for trading a few accuracy points for large latency reductions when both predicted experts are unavailable.
  • The approach requires no model fine-tuning or architecture changes, so it can be dropped into existing MoE deployment pipelines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test how the one-layer-ahead predictor behaves on long, multi-step chain-of-thought sequences; a topic shift could collapse prediction accuracy and erase the speedup, which is a testable boundary condition.
  • The prefill-decode similarity is measured over the whole decode phase, but a sliding-window analysis might reveal drift as generation progresses; if drift exists, periodic re-allocation during decode could improve cache utilization further.
  • The swap threshold of 1.05 is presented without sensitivity analysis; a data-dependent threshold tuned to sequence length and batch size might yield even better cache utilization.
  • The reported speedups are measured on a PCIe 4.0 platform; on faster interconnects the transfer-hiding benefit of CPU pre-calculation would shrink, so the method's advantage may be specific to bandwidth-constrained edge hardware.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents DAOP, an on-device inference engine for MoE models that combines data-aware expert allocation between GPU and CPU with one-layer-ahead expert prediction and selective CPU pre-calculation. The motivating observations are that expert activation patterns are similar between prefill and decode and that expert routing can be predicted one layer in advance. DAOP uses prefill patterns to swap experts between GPU and CPU, then during decode pre-computes predicted CPU-resident experts using current hidden states, with a graceful-degradation fallback to next-best GPU experts. Experiments on Mixtral 8x7B and Phi-3.5 MoE on an NVIDIA A6000 show consistent speedups over Fiddler (40.4% at input/output length [256,512]; 35.4% average across cache ratios), large speedups over migration-heavy baselines, and improved energy efficiency. Accuracy is reported on several downstream tasks at cache ratios down to 25%.

Significance. If the empirical claims hold, DAOP is a useful systems contribution: it is training-free, does not require model modification, has a clear algorithmic description (Algorithm 1), and the code is public. The speedup over Fiddler is consistent across cache ratios and both models, and the energy efficiency gains are concrete. The main weakness is that the accuracy claim is stated too broadly: the routing-predictability premise, which is load-bearing for the pre-calculation mechanism, degrades sharply on GSM8K at low cache ratios, and the paper's own limitation section acknowledges this. With the accuracy claims appropriately qualified and the missing GSM8K prediction data supplied, the contribution would be solid but narrower than the abstract and conclusion advertise.

major comments (4)
  1. [Abstract, §VII, Table VI] The abstract and conclusion claim that DAOP maintains model accuracy, but Table VI directly contradicts this for GSM8K. For Mixtral 8x7B, GSM8K ExactMatch falls from 58.91 (official) to 51.48 at ECR 62.5%, 41.77 at ECR 37.5%, and 33.51 at ECR 25%—a 43% relative drop at the lowest ratio. Phi-3.5 MoE similarly drops from 86.88 to 74.07 at ECR 25%. Section VI.B acknowledges the limitation, but the abstract's "while maintaining model accuracy" and the conclusion's "without compromising accuracy" are unqualified. Since ECR 25% is one of DAOP's advertised operating points (see §V.B, "even with only 25% of experts cached"), the accuracy claim must be restricted to the benchmarks that actually support it, and GSM8K should be reported as a known failure case in the abstract-level claims.
  2. [§III, Observation 3; Fig. 5; Table II] The central prediction mechanism is validated only on average: Fig. 5 reports 84.11% one-layer-ahead prediction accuracy averaged over Alpaca, MATH, and C4, and GSM8K is excluded from that figure even though Table II includes GSM8K in the prefill-decode similarity measurement. This is precisely the load-bearing point that breaks in Table VI. The authors should report per-dataset one-layer-ahead prediction accuracy for all accuracy benchmarks, including GSM8K, and relate those numbers to the Table VI accuracy drops. Without this, the claim that expert prediction is "generally high" (§III) is not established for the workload where DAOP's accuracy degrades most.
  3. [§II.B, §V.A] The abstract claims DAOP outperforms "traditional expert caching and prefetching methods" by up to 8.20×, but Pre-gated MoE—a prefetching method directly based on prediction—is explicitly excluded from the baseline comparison in §V.A on the grounds of smaller-scale experts and fine-tuning requirements. This makes the prefetching comparison incomplete: the most relevant prediction-based prefetching competitor is not evaluated. Either add a comparison to Pre-gated MoE (possibly on a suitable configuration) or rephrase the speedup claim to refer to the specific evaluated baselines rather than to the broad category of prefetching methods.
  4. [§V.D, Table V] The sentence "The results confirm that our approximate optimizations during decoding do not affect task performance" is not supported by Table V, which the text says evaluates only the first output token (prefill stage). Decoding-time approximations are therefore not tested by that table. The full-sequence results in Table VI show nonzero effects on several tasks and large effects on GSM8K. The sentence should be reworded to say that prefill-stage accuracy is unaffected, and the decode-stage effects should be summarized from Table VI with the GSM8K caveat.
minor comments (6)
  1. [§V.A(d)] The list of baselines omits MoE-Infinity and EdgeMoE even though they are discussed in §II.B; clarify why these are not compared and why MoE-OnDemand is chosen as a proxy.
  2. [Fig. 6] The figure caption says "expert parallelism based on predication"; this should be "prediction".
  3. [Fig. 10] The y-axis label "Impro. (%)" is ambiguous; it should state clearly that it is the percentage improvement of DAOP over Fiddler.
  4. [Algorithm 1] The fixed constants SwapInOut=1.05 and swap fraction 0.5 are not accompanied by a sensitivity analysis; a short study varying these values would help show that the reported speedups are not tuned to a narrow operating point.
  5. [Table IV] Several entries for Phi-3.5 MoE are marked with a dash without explanation; state why DeepSpeed-MII and Mixtral-Offloading results are unavailable for that model.
  6. [§V.B] The paper reports speedups without error bars or repeated-run statistics; adding standard deviations across runs would strengthen the end-to-end comparison.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity found: DAOP's speedups and prediction-accuracy figures are empirical measurements against external baselines and the model's own gating, not quantities forced by construction.

full rationale

DAOP's derivation chain is empirical rather than circular. The load-bearing premise — that expert routing can be predicted one layer ahead — is established by direct measurement: the predicted top-2 experts (from the (i+1)-th gate applied to the i-th block's non-MoE output) are compared against the actual top-2 experts chosen by the fully-updated hidden states, yielding 84.11% agreement (Fig. 5), and prefill/decode activation-matrix similarity is measured at 90.72% (Table II). Neither quantity is defined in terms of the other; the prediction is an approximation of the model's own gating with an independently measured error rate, and it is not fitted to the reported speedups. The end-to-end claims (40.4% over Fiddler at [256,512]; up to 8.20x over caching/prefetching baselines; 35.4% average improvement in Fig. 10) are measured tokens/s against external systems (Fiddler, MoE-OnDemand, DeepSpeed-MII, Mixtral-Offloading) on fixed hardware, not quantities forced by construction. The design constants (SwapInOut = 1.05, swap fraction 0.5, prediction start at layer 4) are stated design choices, not parameters fitted to reproduce the reported results. The ShareGPT calibration set is disjoint from the downstream accuracy tasks, so the accuracy numbers are not self-referential, and Table V is explicitly scoped to the prefill phase only. The two co-author self-citations ([3], [5]) concern quantization and pruning and are not load-bearing for DAOP's central claim. The disclosed GSM8K limitation (Table VI: ExactMatch falls from 58.91 to 33.51 at ECR 25%, with the 3.43% cosine-similarity drop reported in Sec. VI.B) is a genuine accuracy-degradation concern on math-heavy workloads and makes the unqualified "maintaining model accuracy" phrasing an overclaim, but that is a correctness and evaluation-scope issue, not a circular derivation.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central result depends on three hand-chosen parameters and four domain assumptions about GPU/CPU speed and expert-routing predictability. No new physical entities are introduced.

free parameters (3)
  • SwapInOut = 1.05
    Algorithm 1 threshold comparing hot and cold expert activation counts; chosen by hand to avoid unnecessary swaps.
  • SwapNum fraction = 0.5
    Algorithm 1 swaps 0.5 times the number of experts per layer; hand-chosen design choice.
  • Prediction start layer = 4
    Section IV.C: one-layer-ahead prediction is applied only when i >= 4; chosen because early layers show lower prediction accuracy (Fig. 5).
assumptions (5)
  • domain assumption GPU memory is insufficient to store all expert weights simultaneously.
    Section VI.A states DAOP assumes GPU memory is limited for storing model weights; the entire offloading scheme relies on this.
  • domain assumption GPU execution is faster and more energy-efficient than CPU execution for experts.
    Section VI.A and Table I assume GPU is faster; if CPU were not slower, offloading would not be necessary, but if CPU were much slower than the transfer cost, pre-computation would not help.
  • domain assumption CPU-GPU transfer latency for expert weights exceeds the time to execute the expert on the CPU.
    Section VI.A condition (3) and Table I show expert migration at 39.87 ms; DAOP's CPU pre-computation is only beneficial under this condition.
  • domain assumption Expert activation patterns are predictable across prefill/decode and one layer ahead.
    Observations 2 and 3: similarity 90.72% (Table II) and prediction accuracy 84.11% (Fig. 5); this is empirically validated but not guaranteed and is the main failure point on GSM8K.
  • domain assumption ShareGPT calibration yields a representative initial expert cache.
    Section V.A uses ShareGPT solely for initialization; if this distribution differs sharply from the deployment task, the initial allocation may be suboptimal.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference." pith.science (2026). https://pith.science/paper/NUS4P5UE

@misc{pith2026250110375,
  author       = {Pith},
  title        = {Pith review of: DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NUS4P5UE}},
  note         = {Machine review of arXiv:2501.10375}
}
read the original abstract

Mixture-of-Experts (MoE) models, though highly effective for various machine learning tasks, face significant deployment challenges on memory-constrained devices. While GPUs offer fast inference, their limited memory compared to CPUs means not all experts can be stored on the GPU simultaneously, necessitating frequent, costly data transfers from CPU memory, often negating GPU speed advantages. To address this, we present DAOP, an on-device MoE inference engine to optimize parallel GPU-CPU execution. DAOP dynamically allocates experts between CPU and GPU based on per-sequence activation patterns, and selectively pre-calculates predicted experts on CPUs to minimize transfer latency. This approach enables efficient resource utilization across various expert cache ratios while maintaining model accuracy through a novel graceful degradation mechanism. Comprehensive evaluations across various datasets show that DAOP outperforms traditional expert caching and prefetching methods by up to 8.20x and offloading techniques by 1.35x while maintaining accuracy.

Figures

Figures reproduced from arXiv: 2501.10375 by the authors.

Figure 1
Figure 1. ). This sparse activation balances model capacity with computational efficiency, significantly reducing computational load compared to traditional dense models. However, deploying these models in low-resource environ￾ments is challenging due to limited GPU memory capacity ( [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. The decoder-only MoE-based LLM inference procedure with top-2 ex￾perts activated per token. This study examines decoder-only MoE-based LLMs [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 6
Figure 6. Design Overview of DAOP. utilize expert parallelism based on predication to leverage CPU resources, accelerating inference speed. A. Memory Initialization We initially allocate non-MoE and dominant experts to the GPU to maximize memory utilization, with others assigned to the CPU. Utilizing the calibration dataset, we identify the expert activation pattern through the decode phase and prioritize experts based on lay… view at source ↗
Figures from the paper (4 more)
Figure 8
Figure 8. Figure 8: Execution timeline of DAOP and related works, showing two consecu [PITH_FULL_IMAGE:figures/full_fig_p004_8.png]
Figure 7
Figure 7. Figure 7: Computation flow of expert pre-calculation based on prediction. [PITH_FULL_IMAGE:figures/full_fig_p004_7.png]
Figure 9
Figure 9. Figure 9: Inference speed comparison with full GPU memory utilization. [PITH_FULL_IMAGE:figures/full_fig_p005_9.png]
Figure 10
Figure 10. Figure 10: Inference speed improvement of DAOP over Fiddler for input/output length 256 [PITH_FULL_IMAGE:figures/full_fig_p005_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HD-MoE: Hybrid and Dynamic Parallelism for Mixture-of-Expert LLMs with 3D Near-Memory Processing

    cs.PF 2025-09 conditional novelty 6.0 of 10

    HD-MoE combines an offline linear-programming placement search with online expert pre-broadcast, cutting simulated MoE inference latency on 3D near-memory processors by 1.1-1.8x over tensor parallelism.

Reference graph

Works this paper leans on

32 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [1]

    Adaptive mixtures of local experts,

    R. A. Jacobs et al., “Adaptive mixtures of local experts,” Neural compu- tation, vol. 3, no. 1, pp. 79–87, 1991

    work page 1991

  2. [2]

    Mixtral of experts,

    A. Q. Jiang et al., “Mixtral of experts,” arXiv preprint arXiv:2401.04088, 2024

    arXiv 2024

  3. [3]

    Shedding the bits: Pushing the boundaries of quantization with minifloats on FPGAs,

    S. Aggarwal et al. , “Shedding the bits: Pushing the boundaries of quantization with minifloats on FPGAs,” in FPL. IEEE, 2024, pp. 297– 303

    work page 2024

  4. [4]

    Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models,

    X. Lu et al. , “Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models,” arXiv preprint arXiv:2402.14800, 2024

    arXiv 2024

  5. [5]

    CRISP: Hybrid structured sparsity for class-aware model pruning,

    S. Aggarwal et al. , “CRISP: Hybrid structured sparsity for class-aware model pruning,” in DATE, 2024, pp. 1–6

    work page 2024

  6. [6]

    SiDA: Sparsity-inspired data-aware serving for efficient and scalable large mixture-of-experts models,

    Z. Du et al. , “SiDA: Sparsity-inspired data-aware serving for efficient and scalable large mixture-of-experts models,” Proc. of MLSys , vol. 6, pp. 224–238, 2024

    work page 2024

  7. [7]

    Pre-gated MoE: An algorithm-system co-design for fast and scalable mixture-of-expert inference,

    R. Hwang et al. , “Pre-gated MoE: An algorithm-system co-design for fast and scalable mixture-of-expert inference,” in ISCA. IEEE, 2024, pp. 1018–1031

    work page 2024

  8. [8]

    AdapMoE: Adaptive sensitivity-based expert gating and management for efficient moe inference,

    S. Zhong et al., “AdapMoE: Adaptive sensitivity-based expert gating and management for efficient moe inference,” ICCAD, 2024

    work page 2024

Show all 32 references
  1. [9]

    A survey of GPT-3 family large language models including ChatGPT and GPT-4,

    K. S. Kalyan, “A survey of GPT-3 family large language models including ChatGPT and GPT-4,” Natural Language Processing Journal, p. 100048, 2023

    2023

  2. [10]

    LLaMA: Open and efficient foundation language models,

    H. Touvron et al. , “LLaMA: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971 , 2023

    2023 arXiv

  3. [11]

    Fast inference of mixture-of-experts language models with offloading,

    A. Eliseev et al., “Fast inference of mixture-of-experts language models with offloading,” arXiv preprint arXiv:2312.17238 , 2023

    2023 arXiv

  4. [12]

    MoE-Infinity: Activation-aware expert offloading for efficient moe serving,

    L. Xue et al. , “MoE-Infinity: Activation-aware expert offloading for efficient moe serving,” arXiv preprint arXiv:2401.14361 , 2024

    2024 arXiv

  5. [13]

    EdgeMoE: Fast on-device inference of moe-based large language models,

    R. Yi et al. , “EdgeMoE: Fast on-device inference of moe-based large language models,” arXiv preprint arXiv:2308.14352 , 2023

    2023 arXiv

  6. [14]

    Fiddler: CPU-GPU orchestration for fast inference of mixture-of-experts models,

    K. Kamahori et al., “Fiddler: CPU-GPU orchestration for fast inference of mixture-of-experts models,” arXiv preprint arXiv:2402.07033 , 2024

    2024 arXiv

  7. [15]

    A review of sparse expert models in deep learning,

    W. Fedus et al. , “A review of sparse expert models in deep learning,” arXiv preprint arXiv:2209.01667 , 2022

    2022 arXiv

  8. [16]

    A survey on mixture of experts,

    W. Cai et al. , “A survey on mixture of experts,” arXiv preprint arXiv:2407.06204, 2024

    2024 arXiv

  9. [17]

    Hellaswag: Can a machine really finish your sentence?

    R. Zellers et al., “Hellaswag: Can a machine really finish your sentence?” arXiv preprint arXiv:1905.07830 , 2019

    1905 arXiv

  10. [18]

    Think you have solved question answering? try arc, the ai2 reasoning challenge,

    P. Clark et al., “Think you have solved question answering? try arc, the ai2 reasoning challenge,” arXiv preprint arXiv:1803.05457 , 2018

    2018 arXiv

  11. [19]

    PIQA: Reasoning about physical commonsense in natural language,

    Y . Bisk et al., “PIQA: Reasoning about physical commonsense in natural language,” in Proc. of AAAI , vol. 34, no. 05, 2020, pp. 7432–7439

    2020

  12. [20]

    Winogrande: An adversarial winograd schema challenge at scale,

    K. Sakaguchi et al. , “Winogrande: An adversarial winograd schema challenge at scale,” Communications of the ACM , vol. 64, no. 9, pp. 99–106, 2021

    2021

  13. [21]

    Training verifiers to solve math word problems,

    K. Cobbe et al., “Training verifiers to solve math word problems,” arXiv preprint arXiv:2110.14168, 2021

    2021 arXiv

  14. [22]

    Measuring massive multitask language understand- ing,

    D. Hendrycks et al., “Measuring massive multitask language understand- ing,” arXiv preprint arXiv:2009.03300 , 2020

    2009 arXiv

  15. [23]

    TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension,

    M. Joshi et al., “TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension,” arXiv preprint arXiv:1705.03551 , 2017

    2017 arXiv

  16. [24]

    Rouge: A package for automatic evaluation of sum- maries,

    C.-Y . Lin et al. , “Rouge: A package for automatic evaluation of sum- maries,” in Text summarization branches out , 2004, pp. 74–81

    2004

  17. [25]

    Transformers: State-of-the-art natural language process- ing,

    T. Wolf et al. , “Transformers: State-of-the-art natural language process- ing,” in Proc. of EMNLP: system demonstrations , 2020, pp. 38–45

    2020

  18. [26]

    A framework for few-shot language model evaluation,

    L. Gao et al. , “A framework for few-shot language model evaluation,”

  19. [27]

    ShareGPT,

    S. Team, “ShareGPT,” 2024. [Online]. Available: https://sharegpt.com/

    2024

  20. [28]

    DeepSpeed-MII,

    Microsoft, “DeepSpeed-MII,” 2024. [Online]. Available: https://github. com/microsoft/DeepSpeed-MII

    2024

  21. [29]

    GeForce RTX 4090,

    NVIDIA, “GeForce RTX 4090,” 2024. [Online]. Available: https: //www.nvidia.com/en-sg/geforce/graphics-cards/40-series/rtx-4090

    2024

  22. [30]

    [Online]

    AMD, “MI300X,” 2024. [Online]. Available: https://www.amd.com/en/ products/accelerators/instinct/mi300/mi300x.html

    2024

  23. [31]

    NVIDIA Hopper H100 GPU: Scaling performance,

    J. Choquette, “NVIDIA Hopper H100 GPU: Scaling performance,” IEEE Micro, vol. 43, no. 3, pp. 9–17, 2023

    2023

  24. [2023]

    Available: https://zenodo.org/records/10256836

    [Online]. Available: https://zenodo.org/records/10256836

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.