Pith. sign in

REVIEW 3 major objections 4 minor 52 references

FluidML: Fast and Memory Efficient Inference Optimization

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FluidML argues that a model's execution blueprint—memory layout for every tensor plus loop order for every kernel—can be globally optimized to cut inference latency by up to 25.38% and peak memory by up to 41.47% on CPU platforms.

desk verdict A serious CPU-inference optimization system whose own tables contradict the abstract's 'consistently reduce' latency and memory claims. read the letter →

arxiv 2411.09242 v1 pith:QDBLPWXN submitted 2024-11-14 cs.LG

Jinjie Liu , Hang Qiu This is my paper · ORCID
classification cs.LG
keywords memorylayoutschedulinginferenceruntimeoptimizationtransformeredgedeploymentMLIRcompilerdynamicprogrammingstaticallocationloopreordering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FluidML is a compiler-and-virtual-machine framework that argues a consequential point: for modern neural networks, the memory layout of every tensor and the loop order of every kernel form a global optimization problem, not a per-operator detail. The paper's method splits the computation graph into long operator sequences, uses dynamic programming over each sequence with real measured kernel timings to choose input and output memory layouts, and resolves conflicting layout choices across sequences by majority vote; a greedy-by-size static allocator then packs tensors with non-overlapping lifetimes into shared buffers. On three CPU platforms, FluidML reports end-to-end latency reductions up to 25.38% for MatMul-heavy transformers and peak-memory reductions up to 41.47% relative to ONNX-MLIR, ONNXRuntime, PyTorch, and TVM, with the largest gains on BERT-family models and small or negative gains on MatMul-light GPT-NeoX and VGG. If the reported numbers hold, edge deployments gain a model-agnostic way to speed up inference and shrink footprint without touching the model weights.

What carries the argument

The load-bearing object is the per-edge memory layout schedule produced by the recursive longest-sequence decomposition and the dynamic-programming recurrence $$t_{v,l} = \min_{u \in N(v),\, l' \in L(u)} \left( t_{u,l'} + T_{(u,l'),\ldots,(v,l)} \right),$$ where $T$ is the measured time of one kernel given its input and output layouts. The DP assumes those measured kernel times are stable and additive, so the cost of the whole schedule is the sum of per-kernel costs. After each sequence gets its optimal layout plan, a majority-vote pass resolves conflicts on shared edges, and a Greedy-by-Size allocator packs tensors into shared buffers by descending size subject to lifetime overlap. The virtual machine's role is to supply the real-world timings that make the cost model concrete.

What would settle it

Compile BERT with the released FluidML code on the paper's Intel i7-13700 setup and compare end-to-end latency and peak resident-set size against ONNX-MLIR and the unoptimized version; if the latency reduction does not approach 25.38% or the memory reduction does not approach 41.47%, the central performance claim fails.

Watch

Extended reading notes

Core claim

The central claim is that a neural network's execution blueprint can be transformed globally and cheaply. Instead of tuning each operator's kernel in isolation, FluidML decomposes the graph into the longest linear sequences of operators, applies a dynamic program whose cost model is populated by the virtual machine's real-world kernel timings (the recurrence in Eq. 1), and then reconciles the sequences' conflicting layout choices by majority vote. The resulting schedule fixes one memory layout per tensor edge and, for MatMul and GEMM, a cache-friendly loop order; the schedule is lowered through MLIR to LLVM IR and run on a JIT engine. The authors report that this blueprint transformation reduces end-to-end BERT-family inference latency by up to 25.38% and peak memory by up to 41.47% against the compared frameworks, while acknowledging that MatMul-light graphs such as GPT-NeoX and VGG see little or no latency benefit. The memory reduction is achieved by a static Greedy-by-Size allocator that reuses memory across non-overlapping tensor lifetimes and performs only a handful of allocation calls at runtime.

Load-bearing premise

The dynamic program assumes that a kernel's measured execution time with a chosen memory layout is stable and additive—that it does not change depending on neighboring kernels, other memory traffic, cache state, or operating-system scheduling.

Editorial extensions

If this is right

  • On MatMul-heavy transformer models like BERT, ConvBERT, and I-BERT, the joint layout and loop-order schedule should cut end-to-end latency by up to 25.38% relative to ONNX-MLIR without changing model weights.
  • Static Greedy-by-Size allocation should reduce peak memory by up to 41.47% compared with allocator-based frameworks, because non-overlapping tensor lifetimes share buffers and runtime allocation calls drop to single digits.
  • The longest-sequence decomposition plus per-sequence DP keeps the scheduling overhead linear, so the same compile-time machinery should scale to large graphs.
  • Ablations imply that loop reordering for MatMul/GEMM is the dominant source of speedup in transformer graphs, contributing roughly 67–77% of the MatMul gains, while layout DP alone contributes smaller gains.
  • For MatMul-light graphs such as GPT-NeoX and VGG, the framework's latency advantage should be small or negative, which the paper attributes to few, rectangular, or bias-carrying matrix multiplications.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same blueprint transformation could be retargeted to GPUs or NPUs by swapping the virtual machine's eval() oracle for device-specific timing, but the additivity assumption would need revalidation because cache hierarchies and concurrent streams make per-kernel timings less stable.
  • Because the cost model is populated by measured timings, FluidML could be extended into a per-device autotuner that recompiles the layout schedule for each target hardware and batch shape, using the JIT engine's measurements as feedback.
  • The majority-vote conflict-resolution policy is a heuristic; weighting votes by the time lost in each sequence or by sequence length might close part of the 9–17% gap to the brute-force optimum that the paper reports on toy graphs.
  • Extending loop reordering to bias-carrying GEMM and rectangular MatMul shapes is the most direct route to improving the weak GPT-NeoX and VGG results.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents FluidML, an MLIR/LLVM-based framework that takes ONNX models and applies a two-part optimization: a dynamic-programming schedule that chooses per-tensor memory layouts using kernel timings collected from a custom virtual machine, and a greedy-by-size static memory allocator that reuses memory blocks across non-overlapping tensor lifetimes. The evaluation reports end-to-end latency and peak memory usage for five models (BERT, ConvBERT, GPT-NEOX, I-BERT, VGG) on Intel, AMD, and Apple platforms, comparing against ONNX-MLIR for latency and against ONNXRuntime, PyTorch, and TVM for memory.

Significance. If the reported gains were consistent and reproducible, FluidML would be a genuinely useful addition to the edge-inference toolchain, particularly for transformer models where layout and memory-access optimization can matter as much as kernel arithmetic. The idea of collecting real-world kernel timings through a VM and feeding them into a scheduling DP is pragmatic and reasonably standard as autotuning practice, and the static-allocation result is a sensible engineering contribution. The toy-model brute-force comparison in Table 3 is also a welcome attempt to bound the optimality loss from conflict resolution. However, the paper's central quantitative claims are not supported by its own data, and the transferability of the DP cost model to large real graphs is not adequately validated, so the current contribution is significantly overstated.

major comments (3)
  1. [Abstract vs. §5.1, Table 2] The abstract claims that FluidML can 'consistently reduce the end-to-end inference latency by up to 25.38%' for popular language models compared to state-of-the-art approaches. Table 2 directly contradicts this: on GPT-NEOX, FluidML takes 9 ms versus ONNX-MLIR's 6 ms on AMD and 8 ms versus 4 ms on Intel, and on VGG it takes 19.0 s versus 10.1 s on AMD and 10.0 s versus 5.5 s on Intel. The paper itself acknowledges in §5.1 that FluidML 'slightly lags behind on GPT-NEOX and VGG.' The claim of consistency is false as stated, and the 25.38% figure does not match the much larger or differently-directioned numbers in the table. Please correct the abstract to describe model-dependent results or provide additional evidence that reconciles these discrepancies.
  2. [Abstract vs. §5.4, Table 4] The abstract claims a peak memory reduction 'by up to 41.47%' compared to state-of-the-art approaches, but Table 4 shows that TVM uses 142.1 MB on I-BERT while FluidML uses 348.0 MB, making FluidML more than twice as memory-hungry on that model. Furthermore, the 41.47% figure does not correspond to any comparison in Table 4; for example, versus PyTorch on BERT the reduction is about 86.7%, and versus ONNXRuntime on GPT-NEOX it is about 94.8%. The memory claim should be recomputed, qualified by which baseline and which model, and explicitly acknowledge the I-BERT/TVM case.
  3. [§3, Eq. (1), Alg. 3 and §5.3] The scheduling DP relies on eval(node, layouts) returning real-world kernel execution times and assumes that the cost T(u,l'),(v,l) is stable and additive, i.e., independent of neighboring kernels, cache state, and concurrent memory traffic. The paper validates this only through two toy graphs in Table 3, reporting optimality gaps of about 17% and 9%. Since the paper claims a 'holistic' and 'graph-agnostic' optimization, a stronger validation is needed on the actual benchmark models, for example by comparing the DP-chosen layout schedule against a per-operator greedy layout choice or against a second heuristic, to establish that the cost model transfers beyond the toy settings. Without that, the end-to-end speedups on large graphs could be influenced by factors the DP does not capture.
minor comments (4)
  1. [Throughout] The paper uses 'FLUID ML' in the body but the title and abstract use 'FluidML'; please standardize the spelling.
  2. [Alg. 2, §2, §5.4] There are several typos: 'sequnece' in Algorithm 2, 'untractable' in Section 2, 'FLU-IDML' in Section 5.4, and 'GPT-NEXO' in the Table 4 caption. These should be fixed.
  3. [Table 2] The table header is difficult to parse: the units are mixed (BERT, I-BERT, ConvBERT, and VGG in seconds, GPT-NEOX in milliseconds) and the column labels wrap confusingly. Please make the model names and units explicit in each column.
  4. [§5.2] The sentence 'At the same time, we noticed that for the Gemm operator. However, the loop reorder has also brought significant improvements;' is grammatically incomplete and should be rewritten.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the scheduling DP is empirical autotuning and the evaluation is anchored to external baselines.

full rationale

FluidML's core optimization is a layout-scheduling dynamic program whose cost function is the VM's measured per-kernel execution time (Eq. 1, Alg. 3). This is empirical autotuning, not circular: the DP selects a schedule by minimizing a cost measured on the same hardware, and the paper then reports the resulting end-to-end latency against external baselines (ONNX-MLIR, TVM, PyTorch, ONNXRuntime). No parameter is fitted to a subset of the reported results and then 'predicted' on a closely related quantity; the claimed speedups are measured outcomes, not forecasts derived from inputs. The memory allocator (Alg. 4) is a standard greedy-by-size scheme cited to prior work (Pisarchyk & Lee, 2020) and used as a component, not as evidence for the speedup claim. Self-citations (Liu et al., 2019) are used only to motivate the problem, not to justify the central result. The abstract's 'consistently' and the 25.38% figure are not supported by Table 2 (FluidML is slower than ONNX-MLIR on GPT-NEOX and VGG), but this is a correctness or consistency defect, not circularity. I find no step in the derivation chain that reduces, by the paper's own equations or by self-citation, to its inputs.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper does not introduce new physical or mathematical entities; its contributions are algorithmic and systemic. The main unstated premises are the DAG assumption, the additivity of kernel timing, the stability of the VM timing signal, and the effectiveness of majority voting for conflict resolution. Only one design choice (the per-operator layout candidate set) functions as an unparameterized free choice that materially affects the reported results.

free parameters (1)
  • Per-operator layout candidate set = unspecified
    The DP search space and final latency depend on which layouts (row-major, column-major, tiled, etc.) are enumerated for each operator; the paper never specifies this set, so the reported speedups cannot be reproduced or compared fairly across frameworks.
assumptions (4)
  • domain assumption The ONNX computation graph is a directed acyclic graph (DAG) with well-defined successor relationships.
    Longest-sequence extraction in Algorithm 1 assumes a DAG with no cycles; control flow or dynamic shapes would break the recursive splitting and DP scheduling.
  • domain assumption Execution time of an operator with a given input/output layout is independent and additive across kernels.
    Equation 1 sums per-kernel times assuming no interactions between kernels beyond layout; cache effects, memory bandwidth contention, and OS interference are ignored.
  • ad hoc to paper Real-world timings from the VM are a stable, representative measure of end-to-end performance.
    The DP uses eval() to choose layouts; if these timing measurements are noisy or not representative of the final deployment environment, the schedule is suboptimal. No error bars or repeat counts are given.
  • ad hoc to paper Majority voting conflict resolution preserves near-optimality on large real graphs.
    Only validated on two toy models with 17% and 9% loss versus brute force; no evidence is provided for larger or more interleaved graphs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FluidML: Fast and Memory Efficient Inference Optimization." pith.science (2026). https://pith.science/paper/QDBLPWXN

@misc{pith2026241109242,
  author       = {Pith},
  title        = {Pith review of: FluidML: Fast and Memory Efficient Inference Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QDBLPWXN}},
  note         = {Machine review of arXiv:2411.09242}
}
read the original abstract

Machine learning models deployed on edge devices have enabled numerous exciting new applications, such as humanoid robots, AR glasses, and autonomous vehicles. However, the computing resources available on these edge devices are not catching up with the ever-growing number of parameters in these models. As the models become bigger and more complicated, the novel yet sophisticated structure challenges the inference runtime optimization. We present FluidML, a generic runtime memory management and optimization framework that can flexibly transform the model execution blueprint to achieve faster and more memory-efficient inference. Evaluations across different platforms show that FluidML can consistently reduce the end-to-end inference latency by up to 25.38% for popular language models and reduce peak memory usage by up to 41.47%, compared to state-of-the-art approaches. FluidML is of ~30K line of codes, built for general-purpose usage, and will be released as an open-source inference runtime optimization framework to the community.

Figures

Figures reproduced from arXiv: 2411.09242 by the authors.

Figure 1
Figure 1. Memory access patterns for matrix multiplication can be easily converted into executable files suitable for running on these platforms. At runtime, the compiled intermediate products can be stored in the read-only area (e.g.,, the ".rodata" segment for x86-64) for fast access, while the inference engine must load dynamically allocate buffered on demand and load the model parameters into these buffers. Also, modern o… view at source ↗
Figure 2
Figure 2. Mockup example for the optimal memory layout chal￾lenge: graphs with simple connections (left) can be solved opti￾mally in linear time, while complicated connections (right) and dependencies make the search untractable. erators, each with a few diverse memory layout options. Selecting the optimal layout for each tensor to ultimately achieve global optimality is NP-hard. Early work (Liu et al., 2019) attempted on sim… view at source ↗
Figure 3
Figure 3. Memory Allocation Strategy. The left is the naive ver￾sion, and the right is the dynamic programming version. tensor. However, we can see a potential optimization. Some tensors do not overlap in their lifetime, so they can share the same memory space at different periods. FLUIDML leverages a Greedy-by-Size strategy to solve this problem as shown in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Normalized Latency on AMD [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 8
Figure 8. Figure 8: Computation-Intensive Operators Latency Reduction on Different Models Normalized by Unoptimized we set the unoptimized version’s time consumption as the unit’s baseline and calculate the optimized time cost on this basis. For ordinary operators, their optimization is b…
Figure 9
Figure 9. Figure 9: Time Cost Ratio of Different Kernels in BERT [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: Time Cost Ratio of Different Kernels in GPT-NEOX not being noticeable enough. However, the most crucial matrix multiplication overhead has been significantly op￾timized for BERT and I-BERT, so FLUIDML surpassed ONNX-MLIR. However, the small matrix multiplication optim…
Figure 12
Figure 12. Figure 12: Memory Overhead Comparison models, respectively. This loss rate is acceptable, so we can consider this method effective. 5.4 Memory Management Evaluation To evaluate how our memory management works in the real world, we choose three popular frameworks as the baselines…
Figure 13
Figure 13. Figure 13: Normalized Latency Reduction per Operator [PITH_FULL_IMAGE:figures/full_fig_p013_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 23 canonical work pages

  1. [1]

    Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Man\' e , D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., T...

    2015

  2. [2]

    GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch , 9 2023

    Andonian, A., Anthony, Q., Biderman, S., Black, S., Gali, P., Gao, L., Hallahan, E., Levy-Kramer, J., Leahy, C., Nestler, L., Parker, K., Pieler, M., Phang, J., Purohit, S., Schoelkopf, H., Stander, D., Songz, T., Tigges, C., Thérien, B., Wang, P., and Weinbach, S. GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch , 9 2023. URL https://www...

    work page 2023

  3. [3]

    High performance code generation in MLIR: an early case study with GEMM

    Bondhugula, U. High performance code generation in MLIR: an early case study with GEMM . CoRR, abs/2003.00532, 2020. URL https://arxiv.org/abs/2003.00532

    arXiv 2003

  4. [4]

    The slab allocator: an object-caching kernel memory allocator

    Bonwick, J. The slab allocator: an object-caching kernel memory allocator. In Proceedings of the USENIX Summer 1994 Technical Conference on USENIX Summer 1994 Technical Conference - Volume 1, USTC'94, pp.\ 6, USA, 1994. USENIX Association

    work page 1994

  5. [5]

    J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., Vander P las, J., Wanderman- M ilne, S., and Zhang, Q

    Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., Vander P las, J., Wanderman- M ilne, S., and Zhang, Q. JAX : composable transformations of P ython+ N um P y programs, 2018. URL http://github.com/jax-ml/jax

    2018

  6. [6]

    Q., Wang, L., Hu, Y., Ceze, L., Guestrin, C., and Krishnamurthy, A

    Chen, T., Moreau, T., Jiang, Z., Shen, H., Yan, E. Q., Wang, L., Hu, Y., Ceze, L., Guestrin, C., and Krishnamurthy, A. TVM: end-to-end optimization stack for deep learning. CoRR, abs/1802.04799, 2018. URL http://arxiv.org/abs/1802.04799

    arXiv 2018

  7. [7]

    Clint Whaley , R., Petitet, A., and Dongarra, J. J. Automated empirical optimizations of software and the atlas project. Parallel Computing, 27 0 (1): 0 3--35, 2001. ISSN 0167-8191. doi:https://doi.org/10.1016/S0167-8191(00)00087-9. URL https://www.sciencedirect.com/science/article/pii/S0167819100000879. New Trends in High Performance Computing

    doi:10.1016/s0167-8191(00)00087-9 2001

  8. [8]

    Intel(r) math kernel library for deep neural networks (intel(r) mkl-dnn)

    Corporation, I. Intel(r) math kernel library for deep neural networks (intel(r) mkl-dnn). https://oneapi-src.github.io/oneDNN/v0/index.html

    work page

Show all 52 references
  1. [9]

    J., Jeffries, N., Li, J., Kreeger, N., Nappier, I., Natraj, M., Regev, S., Rhodes, R., Wang, T., and Warden, P

    David, R., Duke, J., Jain, A., Reddi, V. J., Jeffries, N., Li, J., Kreeger, N., Nappier, I., Natraj, M., Regev, S., Rhodes, R., Wang, T., and Warden, P. Tensorflow lite micro: Embedded machine learning on tinyml systems. CoRR, abs/2010.08678, 2020. URL https://arxiv.org/abs/2010.08678

    2010 arXiv

  2. [10]

    Tensorflow lite micro: Embedded machine learning for tinyml systems

    David, R., Duke, J., Jain, A., Janapa Reddi, V., Jeffries, N., Li, J., Kreeger, N., Nappier, I., Natraj, M., Wang, T., Warden, P., and Rhodes, R. Tensorflow lite micro: Embedded machine learning for tinyml systems. In Smola, A., Dimakis, A., and Stoica, I. (eds.), Proceedings ...

    2021

  3. [11]

    developers, O. R. Onnx runtime. https://onnxruntime.ai/, 2021. Version: x.y.z

    2021

  4. [12]

    BERT: pre-training of deep bidirectional transformers for language understanding

    Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018. URL http://arxiv.org/abs/1810.04805

    2018 arXiv

  5. [13]

    Algorithms for compile-time memory optimization

    Gergov, J. Algorithms for compile-time memory optimization. In Proceedings of the Tenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA '99, pp.\ 907–908, USA, 1999. Society for Industrial and Applied Mathematics. ISBN 0898714346

    1999

  6. [14]

    Google. Xnnpack. https://github.com/google/XNNPACK

  7. [15]

    and Geijn, R

    Goto, K. and Geijn, R. A. v. d. Anatomy of high-performance matrix multiplication. ACM Trans. Math. Softw., 34 0 (3), May 2008. ISSN 0098-3500. doi:10.1145/1356052.1356053. URL https://doi.org/10.1145/1356052.1356053

    2008

  8. [16]

    Gustavson, F. G. New generalized data structures for matrices lead to a variety of high performance dense linear algebra algorithms. In Proceedings of the 7th International Conference on Applied Parallel Computing: State of the Art in Scientific Computing, PARA'04, pp.\ 11–20,...

    2004 doi

  9. [17]

    Han, S., Mao, H., and Dally, W. J. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In Bengio, Y. and LeCun, Y. (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, ...

    2016 arXiv

  10. [18]

    E., Vinyals, O., and Dean, J

    Hinton, G. E., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. CoRR, abs/1503.02531, 2015

    2015 arXiv

  11. [19]

    Intel. Onemkl. https://www.intel.com/content/www/us/en/docs/onemkl/developer-reference-c/2024-0/overview.html, a

    2024

  12. [20]

    Openvino

    Intel. Openvino. https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/overview.html, b

  13. [21]

    Convbert: Improving BERT with span-based dynamic convolution

    Jiang, Z., Yu, W., Zhou, D., Chen, Y., Feng, J., and Yan, S. Convbert: Improving BERT with span-based dynamic convolution. CoRR, abs/2008.02496, 2020. URL https://arxiv.org/abs/2008.02496

    2008 arXiv

  14. [22]

    S., Huang, J., Basu, P., Deng, S., Liu, H., Park, J., and Smelyanskiy, M

    Khudia, D. S., Huang, J., Basu, P., Deng, S., Liu, H., Park, J., and Smelyanskiy, M. FBGEMM: enabling high-performance low-precision deep learning inference. CoRR, abs/2101.05615, 2021. URL https://arxiv.org/abs/2101.05615

    2021 arXiv

  15. [23]

    W., and Keutzer, K

    Kim, S., Gholami, A., Yao, Z., Mahoney, M. W., and Keutzer, K. I-bert: Integer-only bert quantization. International Conference on Machine Learning (Accepted), 2021

    2021

  16. [24]

    Knuth, D. E. The Art of Computer Programming, Vol. 1: Fundamental Algorithms. Addison-Wesley, Reading, Mass., third edition, 1997. ISBN 0201896834 9780201896831

    1997

  17. [25]

    CMSIS-NN: efficient neural network kernels for arm cortex-m cpus

    Lai, L., Suda, N., and Chandra, V. CMSIS-NN: efficient neural network kernels for arm cortex-m cpus. CoRR, abs/1801.06601, 2018. URL http://arxiv.org/abs/1801.06601

    2018 arXiv

  18. [26]

    A., Amini, M., Bondhugula, U., Riddle, R., Cohen, A., Shpeisman, T., Davis, A., Vasilache, N., and Zinenko, O

    Lattner, C., Pienaar, J. A., Amini, M., Bondhugula, U., Riddle, R., Cohen, A., Shpeisman, T., Davis, A., Vasilache, N., and Zinenko, O. MLIR: A compiler infrastructure for the end of moore's law. CoRR, abs/2002.11054, 2020. URL https://arxiv.org/abs/2002.11054

    2002 arXiv

  19. [27]

    MLIR : Scaling compiler infrastructure for domain specific computation

    Lattner, C., Amini, M., Bondhugula, U., Cohen, A., Davis, A., Pienaar, J., Riddle, R., Shpeisman, T., Vasilache, N., and Zinenko, O. MLIR : Scaling compiler infrastructure for domain specific computation. In 2021 IEEE/ACM International Symposium on Code Generation and Optimiza...

    2021

  20. [28]

    D., Bercea, G., Chen, T., Eichenberger, A

    Le, T. D., Bercea, G., Chen, T., Eichenberger, A. E., Imai, H., Jin, T., Kawachiya, K., Negishi, Y., and O'Brien, K. Compiling ONNX neural network models using MLIR . CoRR, abs/2008.08272, 2020. URL https://arxiv.org/abs/2008.08272

    2008 arXiv

  21. [29]

    On-device neural net inference with mobile gpus

    Lee, J., Chirkov, N., Ignasheva, E., Pisarchyk, Y., Shieh, M., Riccardi, F., Sarokin, R., Kulik, A., and Grundmann, M. On-device neural net inference with mobile gpus. CoRR, abs/1907.01989, 2019. URL http://arxiv.org/abs/1907.01989

    1907 arXiv

  22. [30]

    Mcunetv2: Memory-efficient patch-based inference for tiny deep learning

    Lin, J., Chen, W., Cai, H., Gan, C., and Han, S. Mcunetv2: Memory-efficient patch-based inference for tiny deep learning. CoRR, abs/2110.15352, 2021. URL https://arxiv.org/abs/2110.15352

    2021 arXiv

  23. [31]

    and Deng, W

    Liu, S. and Deng, W. Very deep convolutional neural network based image classification using small training sample size. In 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), pp.\ 730--734, 2015. doi:10.1109/ACPR.2015.7486599

    2015

  24. [32]

    Optimizing CNN model inference on CPUs

    Liu, Y., Wang, Y., Yu, R., Li, M., Sharma, V., and Wang, Y. Optimizing CNN model inference on CPUs . In 2019 USENIX Annual Technical Conference (USENIX ATC 19), pp.\ 1025--1040, Renton, WA, July 2019. USENIX Association. ISBN 978-1-939133-03-8. URL https://www.usenix.org/confe...

    2019

  25. [33]

    Rammer: Enabling holistic deep learning compiler optimizations with rTasks

    Ma, L., Xie, Z., Yang, Z., Xue, J., Miao, Y., Cui, W., Hu, W., Yang, F., Zhang, L., and Zhou, L. Rammer: Enabling holistic deep learning compiler optimizations with rTasks . In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pp.\ 881--897. USENI...

    2020

  26. [34]

    Meta. Qnnpack. https://github.com/pytorch/QNNPACK

  27. [35]

    Nash, J. C. The (dantzig) simplex method for linear programming. In Computing in Science and Engg., volume 2, pp.\ 29--31. IEEE Educational Activities Department, Piscataway, NJ, USA, 2000. doi:10.1109/5992.814654

    2000

  28. [36]

    Tensorrt

    NVIDIA. Tensorrt. https://developer.nvidia.com/tensorrt

  29. [37]

    Z., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., K \" o pf, A., Yang, E. Z., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: A...

    1912 arXiv

  30. [38]

    and Lee, J

    Pisarchyk, Y. and Lee, J. Efficient memory management for deep neural net inference. CoRR, abs/2001.03288, 2020. URL https://arxiv.org/abs/2001.03288

    2001 arXiv

  31. [39]

    Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines

    Ragan-Kelley, J., Barnes, C., Adams, A., Paris, S., Durand, F., and Amarasinghe, S. Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines. SIGPLAN Not., 48 0 (6): 0 519–530, June 2013. ISSN 0362-1340. doi:10.1145/...

    2013

  32. [40]

    Glow: Graph lowering compiler techniques for neural networks

    Rotem, N., Fix, J., Abdulrasool, S., Deng, S., Dzhabarov, R., Hegeman, J., Levenstein, R., Maher, B., Satish, N., Olesen, J., Park, J., Rakhov, A., and Smelyanskiy, M. Glow: Graph lowering compiler techniques for neural networks. CoRR, abs/1805.00907, 2018. URL http://arxiv.or...

    2018 arXiv

  33. [41]

    Xla : Compiling machine learning for peak performance, 2020

    Sabne, A. Xla : Compiling machine learning for peak performance, 2020

    2020

  34. [42]

    Efficient transformers: A survey

    Tay, Y., Dehghani, M., Bahri, D., and Metzler, D. Efficient transformers: A survey. CoRR, abs/2009.06732, 2020. URL https://arxiv.org/abs/2009.06732

    2009 arXiv

  35. [43]

    IREE , September 2019

    The IREE Authors . IREE , September 2019. URL https://github.com/iree-org/iree

    2019

  36. [44]

    N., Kaiser, L., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. CoRR, abs/1706.03762, 2017. URL http://arxiv.org/abs/1706.03762

    2017 arXiv

  37. [45]

    Augem: Automatically generate high performance dense linear algebra kernels on x86 cpus

    Wang, Q., Zhang, X., Zhang, Y., and Yi, Q. Augem: Automatically generate high performance dense linear algebra kernels on x86 cpus. In SC '13: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, pp.\ 1--12, 2013. doi:10....

    2013

  38. [46]

    Whaley, R. C. and Dongarra, J. J. Automatically tuned linear algebra software. In Proceedings of the 1998 ACM/IEEE Conference on Supercomputing, SC '98, pp.\ 1–27, USA, 1998. IEEE Computer Society. ISBN 089791984X

    1998

  39. [47]

    Model-driven level 3 blas performance optimization on loongson 3a processor

    Xianyi, Z., Qian, W., and Yunquan, Z. Model-driven level 3 blas performance optimization on loongson 3a processor. In 2012 IEEE 18th International Conference on Parallel and Distributed Systems, pp.\ 684--691, 2012. doi:10.1109/ICPADS.2012.97

    2012 doi

  40. [48]

    DeepCPU : Serving RNN-based deep learning models 10x faster

    Zhang, M., Rajbhandari, S., Wang, W., and He, Y. DeepCPU : Serving RNN-based deep learning models 10x faster. In 2018 USENIX Annual Technical Conference (USENIX ATC 18), pp.\ 951--965, Boston, MA, July 2018. USENIX Association. ISBN 978-1-939133-01-4. URL https://www.usenix.or...

    2018

  41. [49]

    H., Haj-Ali, A., Wang, Y., Yang, J., Zhuo, D., Sen, K., Gonzalez, J

    Zheng, L., Jia, C., Sun, M., Wu, Z., Yu, C. H., Haj-Ali, A., Wang, Y., Yang, J., Zhuo, D., Sen, K., Gonzalez, J. E., and Stoica, I. Ansor: Generating High-Performance tensor programs for deep learning. In 14th USENIX Symposium on Operating Systems Design and Implementation (OS...

    2020

  42. [50]

    vmcu: Coordinated memory management and kernel optimization for dnn inference on mcus, 2024

    Zheng, S., Chen, R., Li, M., Ye, Z., Ceze, L., and Liang, Y. vmcu: Coordinated memory management and kernel optimization for dnn inference on mcus, 2024. URL https://arxiv.org/abs/2406.06542

    2024 arXiv

  43. [51]

    and Le, Q

    Zoph, B. and Le, Q. V. Neural architecture search with reinforcement learning. CoRR, abs/1611.01578, 2016. URL http://arxiv.org/abs/1611.01578

    2016 arXiv

  44. [52]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.