Pith. sign in

REVIEW 3 major objections 5 minor 46 references

MoPEQ: Mixture of Mixed Precision Quantized Experts

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Per-expert sensitivity, not activation frequency, is the right signal for quantizing mixture-of-experts vision-language models.

desk verdict Useful first mapping of expert-level mixed-precision quantization for VLM-MoEs, but the 'Hessian sensitivity' metric is analytically just inverse weight norm, so the central interpretive claim does not hold as stated. read the letter →

arxiv 2509.02512 v1 pith:VBPHCONB submitted 2025-09-02 cs.LG

Krishna Teja Chitty-Venkata , Jie Ye , Murali Emani This is my paper · ORCID
classification cs.LG
keywords Mixture-of-Expertsmixed-precisionquantizationpost-trainingvision-languagemodelsHessiantracesensitivityexpertclusteringmodelcompressionbit-widthallocation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that mixture-of-experts vision-language models can be compressed more effectively by giving each expert its own bit width, chosen by a data-free sensitivity score, than by quantizing all experts uniformly or by allocating bits according to how often experts are used. The proposed algorithm, MoPEQ, estimates each expert's sensitivity with a Hessian-trace proxy on the weights and clusters experts with similar importance into 2-, 3-, and 4-bit groups. On four open vision-language MoE models, the method keeps accuracy within a few points of uniform 4-bit baselines while cutting model size by roughly 1.5x, and on one perception benchmark it improves the score from 1300 to 1338 at a smaller size. The broader claim is that expert specialization, not usage frequency, should drive which experts deserve high precision.

What carries the argument

The load-bearing object is the per-expert Hessian trace $\mathrm{Tr}(\mathbf{H})$ of a proxy loss $L = \|\mathbf{W}\|_F$ for each expert's feed-forward weight tensor, estimated by a stochastic trace estimator that forms Hessian-vector products $\nabla_W(g_1^\top v)$ with random sign vectors $v$ and averages $v \odot \mathrm{HVP}$. This score is meant to quantify how much an expert's output would move under quantization, without requiring input data. The assignment machinery then K-means clusters experts by the score, either within each layer or across the whole model, sorts clusters by mean importance, and maps the top cluster to 4 bits, the middle to 3, and the bottom to 2.

What would settle it

Measure, for every expert in one of the tested models, the actual output perturbation after rounding its weights to 2 and 3 bits on a held-out calibration set; compute Spearman correlation between that measured error and the paper's Hessian-trace score. If the correlation is near zero or negative, the bit assignments are not tracking true sensitivity, and the reported accuracy advantages would not be explained by the claimed mechanism.

Watch

Extended reading notes

Core claim

The central claim is that per-expert precision assignment for MoE layers improves the accuracy-per-byte trade-off of post-training quantized vision-language models, and that Hessian-trace sensitivity is a better importance signal than activation frequency. For each expert, the method computes a Hessian trace estimate of the Frobenius-norm loss $\|\mathbf{W}\|_F$ of its feed-forward weight matrices, without any calibration data, then K-means clusters experts by those scores and assigns 4, 3, or 2 bits to high-, mid-, and low-importance clusters. Evidence includes that this sensitivity-based scheme beats the activation-frequency baseline on the largest tested model in most tasks, that model-wise assignment outperforms layer-wise assignment in 63 of 105 comparison scenarios, and that the largest model reaches a 10.485 GB size with competitive scores versus 14.353 GB for uniform 4-bit quantization. If this is right, expert importance is discoverable from weights alone, and compressing MoE models need not wait for calibration data.

Load-bearing premise

The method assumes that the Hessian trace of the Frobenius norm of each expert's weight matrix ranks experts by how much their quantized outputs will damage the model's accuracy, although this proxy is never calibrated against measured quantization error.

Editorial extensions

If this is right

  • If per-expert sensitivity is the right signal, uniform-precision quantization leaves memory on the table for every MoE vision-language model, and data-free bit assignment can deliver most of the savings.
  • On models trained with load-balancing losses, activation frequency is nearly flat, so frequency-based methods cannot differentiate experts; sensitivity-based assignment should be the default there.
  • Model-wide clustering of experts tends to outperform layer-wise clustering because it can concentrate high bits in the layers that matter most to overall output.
  • At fixed bit budgets, the method gives a family of Pareto points: smaller models with mild accuracy loss or larger models with near-baseline accuracy, selectable by the bit set $\{2,3,4\}$.
  • The approach is orthogonal to the underlying quantizer, so it can ride on any post-training rounding routine that accepts per-module bit widths.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because $\mathrm{Tr}(\mathbf{H})$ for $L=\|\mathbf{W}\|_F$ simplifies, for an $n$-parameter weight matrix, to $(n-1)/\|\mathbf{W}\|_F$, the "sensitivity" ranking may effectively order experts by weight norm; a parsimonious test is to compare MoPEQ against magnitude-based bit assignment with the same cluster fractions.
  • The paper does not measure actual per-expert output perturbation before assigning bits; a direct check is to compute per-expert quantization error at 2 and 3 bits and see whether it matches the Hessian ranking.
  • If mixed-precision kernels become available, the scheme's habit of assigning low bits to frequently activated experts could cut GPU-CPU transfer in offloaded MoE serving, which the paper notes but does not measure.
  • The hybrid frequency-times-sensitivity score may be most useful for unevenly routed models, where it trims the model most while keeping accuracy, suggesting the two signals are complementary rather than substitutes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes MoPEQ, a post-training mixed-precision quantization method for mixture-of-experts (MoE) layers in vision-language models. The method assigns 2-, 3-, or 4-bit precision to each expert based on a per-expert importance score, which is computed either from expert activation frequency, from a 'Hessian trace approximation' of the Frobenius-norm loss on the expert weights, or from a normalized product of the two. Similar experts are grouped by K-means clustering and bit widths are assigned to clusters by descending mean importance. The authors evaluate MoPEQ on four VLM-MoEs (DeepSeek-VL2 tiny/small/base and MolmoE-1B) across nine VLMEvalKit tasks, comparing against uniform-precision quantization and activation-frequency-based assignment. The main claims are that sensitivity-based assignment achieves competitive or better accuracy than uniform 4-bit and activation-frequency baselines while reducing model size, and that the approach is data-free because the sensitivity score uses only weights.

Significance. If the sensitivity proxy is valid, the paper would make a useful empirical contribution: it is, to my knowledge, the first mixed-precision quantization study targeting expert granularity in VLM-MoEs, and it includes a broad evaluation across four models and nine tasks. The paper also compares against the activation-frequency baseline and releases code, which are strengths. A notable positive is that the expert-importance assignment is computed from weights alone (plus optionally activation frequency) and is not fitted to the target accuracy, so the central comparison is not circular. However, the validity of the proposed 'Hessian trace' sensitivity metric is the load-bearing assumption of the paper, and as shown below it is not supported by the manuscript's derivation or experiments.

major comments (3)
  1. [Section 3.3, Algorithm 1] The quantity called the 'Hessian trace approximation' is analytically identical to the inverse Frobenius norm of the weight tensor, not a measure of task sensitivity. For L = ||W||_F, the Hessian is H = (I - ww^T/||W||^2)/||W||, so Tr(H) = (n-1)/||W||_F exactly; the Hutchinson estimator in Algorithm 1 merely recovers this deterministic value. Thus Figures 3, 6, and 9 and the 'Hessian sensitivity' assignments in Tables 2-5 are monotone inverse-Frobenius-norm rankings of the expert weights. The manuscript's central claim that this captures 'expert sensitivity' to quantization is unvalidated: no experiment shows a correlation between this quantity and actual per-expert quantization error or task-level accuracy drop. The authors should either replace this proxy with a calibration-based Hessian trace (e.g., on a reconstruction loss) or demonstrate empirically that the inverse-weight-norm ranking correlates with quantization sensitivity. Without that, the observed accuracy differences cannot be attributed to the stated mechanism.
  2. [Tables 2-5] No error bars or multiple-seed results are reported for any of the VLMEvalKit tasks, several of which are known to be noisy. Many of the reported improvements are small (e.g., Table 3 shows DocVQA 87.334 vs. 87.652 for Hessian layer-wise vs. activation-frequency layer-wise, and Table 5 shows MMMU 37.0 vs. 36.0 for Hessian model-wise vs. activation-frequency model-wise). Without variance estimates, the claim that sensitivity-based assignment 'outperforms' activation frequency in '63 different scenarios' (Section 5.3) is not statistically supported. Please provide standard errors, multiple seeds, or at least significance tests for the main comparisons.
  3. [Abstract, Section 1 Contribution (3), Section 6 Conclusion] The manuscript repeatedly claims 'a model size reduction of approximately 1.5x' (also phrased as 'substantial improvements in memory footprint' and 'smaller model size'), but the tables show at most a 1.37x reduction compared to the uniform 4-bit baseline (Table 5: 14.353 GB to 10.485 GB), with other models in the 1.12-1.26x range. If the 1.5x figure is relative to the 8-bit uniform baseline, that baseline is not the one used in the accuracy comparisons and should be stated explicitly. The abstract, contributions, and conclusion should be corrected to match the reported numbers.
minor comments (5)
  1. [Section 3.4] The normalization formula for I_i has no epsilon guard; if max_j H_j equals min_j H_j, the denominator is zero. Please add a small constant or state the assumption that the ranges are non-degenerate.
  2. [Reference list] Reference [42] is listed as a survey on multimodal large language models but is used in Table 2 for the MME benchmark; the MME dataset citation should be [16] (Fu et al., 2023). Please correct the citation.
  3. [Table 4] Some cells in the DeepSeek-VL2-Small results show extreme degradations (e.g., activation-frequency layer-wise DocVQA 13.193 vs. uniform 4-bit 62.58, and MMMU 9.333 vs. 34.0). These are not discussed in the text; a sentence explaining whether these reflect a systematic failure mode of layer-wise assignment would help the reader interpret the comparison.
  4. [Section 4.1] The K-means clustering step requires a pre-specified number of clusters C equal to the number of bit widths, and the mapping from clusters to bit widths is by sorted cluster means; the sensitivity of the assignment to the random initialization of K-means is not reported. Please state the number of runs or the fixed seed used.
  5. [Section 5.1] The paper says the activation-frequency baseline is profiled on the MME dataset and then evaluated on MME among other tasks; this gives the activation-frequency baseline an in-distribution advantage and should be stated explicitly in the setup, since MoPEQ's Hessian-based assignment is data-free.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MoPEQ's bit assignment is computed from weights alone and evaluated on held-out benchmarks, so the central comparisons are external; the Hessian-trace closed form is a methodological concern, not circularity.

full rationale

The paper's derivation chain is self-contained: expert importance is computed from Algorithm 1 (Hutchinson estimate with L=||W||_F) and optionally activation frequency; no parameter is fitted to the VLMEvalKit accuracies, and the final comparisons (Tables 2-5) are obtained by quantizing the models with the assigned precisions and then evaluating on held-out tasks. The assignment step (Algorithm 2) is a fixed K-means clustering with a fixed bit list {2,3,4}; it contains no objective fitted to the reported accuracy. I find no self-citation that is load-bearing, no imported uniqueness theorem, and no prediction that is equivalent to an input by construction. The skeptic's observation is analytically correct: for L=||W||_F, Tr(H) = (n-1)/||W||_F, so the 'Hessian sensitivity' ranking is essentially an inverse-weight-norm ranking. This is a validity/interpretability weakness—the metric may not measure task-level quantization sensitivity—but it is not circularity, because the metric's ranking is not constructed from the benchmark outcomes being predicted. The paper's own evaluation is external to the assignment procedure.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method has no invented entities. Free parameters are limited to the fixed bit schema, cluster count, and unreported estimator/initialization details. The key axiomatic load is the assumption that the Frobenius-norm Hessian trace is a valid proxy for quantization sensitivity, which is ad hoc to this paper.

free parameters (5)
  • Bit width set P = {2, 3, 4}
    The precision search space is fixed to 2-4 bits by the authors; this choice defines the compression-exploration tradeoff.
  • Number of clusters C = 3 = len(P)
    K-means cluster count equals number of bit widths; no sensitivity analysis of this choice is reported.
  • Hutchinson sample count m = not reported in text
    Algorithm 1 requires m, but the paper never states the value or any sensitivity test. This affects trace estimate variance.
  • K-means random seed and initialization = not reported
    K-means depends on initialization; without a fixed seed the expert-to-bit assignment may vary.
  • Min-max normalization bounds = per-model min/max
    The hybrid importance metric normalizes activation frequency and Hessian trace to [0,1]; this is a scale choice.
assumptions (4)
  • standard math Hutchinson's trace estimator converges: Tr(H) = E_{v~N(0,I)}[v^T H v] for symmetric H
    Used in Section 3.3 to justify trace approximation with random Rademacher vectors.
  • ad hoc to paper The Hessian trace of the Frobenius norm loss is a valid proxy for an expert's quantization sensitivity
    Introduced in Section 3.3; no validation against task loss Hessians or per-layer quantization error. This is the load-bearing assumption.
  • domain assumption K-means clustering on scalar importance yields partitions whose mean importance ordering justifies bit assignment
    Algorithm 2 assumes sorted cluster means correspond to optimal precision groups; no optimality argument is given.
  • domain assumption SignRound/AutoRound is an adequate quantization backend such that assignment comparisons are backend-independent
    Section 5.1 uses AutoRound; the paper claims generality but tests only one backend.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MoPEQ: Mixture of Mixed Precision Quantized Experts." pith.science (2026). https://pith.science/paper/VBPHCONB

@misc{pith2026250902512,
  author       = {Pith},
  title        = {Pith review of: MoPEQ: Mixture of Mixed Precision Quantized Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VBPHCONB}},
  note         = {Machine review of arXiv:2509.02512}
}
read the original abstract

Large Language and Vision Models using a Mixture-of-Experts (MoE) architecture pose significant challenges for deployment due to their computational and memory demands. Mixed Precision Quantization assigns different precisions to different layers of an LLM/VLM based on layer sensitivity and importance within the model. In this work, we propose a Post Training Quantization algorithm, MoPEQ, that assigns optimal bit width to each expert. Our method balances accuracy and model size by analyzing each expert's sensitivity using Hessian trace approximation instead of relying on the activation frequency of the expert. This per-expert granularity approach clusters similar experts to maintain model performance while reducing memory requirements. The experimental results on VLMEvalKit benchmark datasets using State-of-the-art VLMs Deepseek-VL2 -tiny, -small, -base, and MolmoE models demonstrate that our mixed precision quantized MoEs achieve competitive accuracy with substantial improvements in memory footprint compared to uniform-precision baseline methods. We perform a comprehensive study to analyze the impact of expert activation frequency and sensitivity using Hessian trace approximation at both layer-wise and model-wide expert precision allocation of 2, 3, and 4 bits to provide a thorough understanding of mixed precision quantization of VLM-MoEs.

Figures

Figures reproduced from arXiv: 2509.02512 by the authors.

Figure 1
Figure 1. MoPEQ: Mixed Precision Quantization of MoE [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Expert Activation Frequency map of MolmoE-1B and DeepSeek VL2 family Models on MME task [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Hessian Trace Approximation Map of MolmoE-1B and DeepSeek VL2 family Models [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Normalized Activation Frequency-Hessian Trace Approximation Importance Map [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Layer-wise Precision Assignment Map based on Expert Activation Frequency [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Layer-wise Precision Assignment Map based Hessian Trace Approximation [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Layer-wise Precision Assignment Map based on Normalized Hessian Trace & Expert Activation Frequency [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Model-wise Precision Assignment Map based on Expert Activation Frequency [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: Model-wise Precision Assignment Map based on Hessian Trace Approximation [PITH_FULL_IMAGE:figures/full_fig_p006_9.png]
Figure 10
Figure 10. Figure 10: Model-wise Precision Assignment Map based on Normalized Hessian Trace & Expert Activation Frequency [PITH_FULL_IMAGE:figures/full_fig_p006_10.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 11 canonical work pages

  1. [1]

    Molmoe-1b-0924

    allenai. Molmoe-1b-0924. https://huggingface.co/allenai/MolmoE-1B-0924, 2024. 3

    work page 2024

  2. [2]

    Randomized algorithms for estimating the trace of an implicit symmetric positive semi- definite matrix

    Haim Avron and Sivan Toledo. Randomized algorithms for estimating the trace of an implicit symmetric positive semi- definite matrix. Journal of the ACM (JACM) , 58(2):1–34,

    work page

  3. [3]

    Lan- guage models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. Advances in neural in- formation processing systems, 33:1877–1901, 2020. 1

    1901

  4. [4]

    A survey on mixture of experts.arXiv preprint arXiv:2407.06204, 2024

    Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts.arXiv preprint arXiv:2407.06204, 2024. 1

    arXiv 2024

  5. [5]

    Optimize weight round- ing via signed gradient descent for the quantization of llms

    Wenhua Cheng, Weiwei Zhang, Haihao Shen, Yiyang Cai, Xin He, Kaokao Lv, and Yi Liu. Optimize weight round- ing via signed gradient descent for the quantization of llms. arXiv preprint arXiv:2309.05516, 2023. 3, 7

    arXiv 2023

  6. [6]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1–113, 2023. 1

    2023

  7. [7]

    Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models

    Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066, 2024. 1

    arXiv 2024

  8. [8]

    Deepseek-vl2

    deepseek ai. Deepseek-vl2. https://huggingface.co/deepseek-ai/deepseek-vl2, 2024. 3

    work page 2024

Show all 46 references
  1. [9]

    Deepseek-vl2 small

    deepseek ai. Deepseek-vl2 small. https://huggingface.co/deepseek-ai/deepseek-vl2-small,

  2. [10]

    Deepseek-vl2 tiny

    deepseek ai. Deepseek-vl2 tiny. https://huggingface.co/deepseek-ai/deepseek-vl2-tiny,

  3. [11]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tri- pathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146, 2...

    2024 arXiv

  4. [12]

    Hawq: Hessian aware quantization of neural networks with mixed-precision

    Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Ma- honey, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precision. In Proceedings of the IEEE/CVF international conference on computer vision, pages 293–302, 2019. 3, 5

    2019

  5. [13]

    Hawq-v2: Hessian aware trace-weighted quantization of neural networks

    Zhen Dong, Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq-v2: Hessian aware trace-weighted quantization of neural networks. Ad- vances in neural information processing systems, 33:18518– 18529, 2020. 3

    2020

  6. [14]

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM International Conference on Multimedia...

    2024

  7. [15]

    Gptq: Accurate post-training quantization for generative pre-trained transformers

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022. 1, 3

    2022 arXiv

  8. [16]

    Mme: A comprehensive evaluation bench- mark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation bench- mark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 3

    2023 arXiv

  9. [17]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. 2

    2025 arXiv

  10. [18]

    Mc-moe: Mixture compressor for mixture-of-experts llms gains more

    Wei Huang, Yue Liao, Jianhui Liu, Ruifei He, Haoru Tan, Shiming Zhang, Hongsheng Li, Si Liu, and Xiaojuan Qi. Mc-moe: Mixture compressor for mixture-of-experts llms gains more. arXiv preprint arXiv:2410.06270 , 2024. 2, 5, 7

    2024 arXiv

  11. [19]

    A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines

    Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Com- munications in Statistics-Simulation and Computation , 18 (3):1059–1076, 1989. 3

    1989

  12. [20]

    Intel autoround

    Intel. Intel autoround. https://github.com/intel/auto-round,

  13. [21]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Deven- dra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. 2

    2024 arXiv

  14. [22]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14 , pages 235–

    2016

  15. [23]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Pro- ceedings of the 29th Symposium on Operating Systems Prin- ciple...

    2023

  16. [24]

    Ex- amining post-training quantization for mixture-of-experts: A benchmark

    Pingzhi Li, Xiaolong Jin, Yu Cheng, and Tianlong Chen. Ex- amining post-training quantization for mixture-of-experts: A benchmark. arXiv preprint arXiv:2406.08155, 2024. 2, 7

    2024 arXiv

  17. [25]

    Moe-llava: Mixture of experts for large vision-language models

    Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947, 2024. 2

    2024 arXiv

  18. [26]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems , 6:87–100,

  19. [27]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024. 3

    2024 arXiv

  20. [28]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. 1

    2024 arXiv

  21. [29]

    A survey on inference optimization techniques for mixture of experts models

    Jiacheng Liu, Peng Tang, Wenfeng Wang, Yuhang Ren, Xi- aofeng Hou, Pheng-Ann Heng, Minyi Guo, and Chao Li. A survey on inference optimization techniques for mixture of experts models. arXiv preprint arXiv:2412.14219, 2024. 1

    2024 arXiv

  22. [30]

    Llm-qat: Data-free quantization aware training for large language models.arXiv preprint arXiv:2305.17888, 2023

    Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large language models.arXiv preprint arXiv:2305.17888, 2023. 1

    2023 arXiv

  23. [31]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems , 35:2507–2521,

  24. [32]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceed- ings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021. 7

    2021

  25. [33]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rub `en Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision, pages 1697–1706, 2022. 7

    2022

  26. [34]

    Olmoe: Open mixture-of-experts language models

    Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al. Olmoe: Open mixture-of-experts language models. arXiv preprint arXiv:2409.02060, 2024. 3

    2024 arXiv

  27. [35]

    Outra- geously large neural networks: The sparsely-gated mixture- of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outra- geously large neural networks: The sparsely-gated mixture- of-experts layer. arXiv preprint arXiv:1701.06538, 2017. 1

    2017 arXiv

  28. [36]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 7

    2019

  29. [37]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 1

    2023 arXiv

  30. [38]

    Efficient large language models: A survey

    Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, et al. Efficient large language models: A survey. arXiv preprint arXiv:2312.03863, 2023. 1

    2023 arXiv

  31. [39]

    Deepseek-vl2: Mixture-of- experts vision-language models for advanced multimodal understanding

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-vl2: Mixture-of- experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302, 2024. 2, 3

    2024 arXiv

  32. [40]

    Smoothquant: Accurate and effi- cient post-training quantization for large language models

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and effi- cient post-training quantization for large language models. In International Conference on Machine Learning , pages 38087–38099. PMLR, 2023. 3

    2023

  33. [41]

    Zeroquant: Ef- ficient and affordable post-training quantization for large- scale transformers

    Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xi- aoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Ef- ficient and affordable post-training quantization for large- scale transformers. Advances in Neural Information Process- ing Systems, 35:27168–27183, 2022. 3

    2022

  34. [42]

    A survey on multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549, 2023. 7

    2023 arXiv

  35. [43]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi. In Proceedings of the IEEE/CVF Conference on...

    2024

  36. [44]

    Vision-language models for vision tasks: A survey

    Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence,

  37. [45]

    Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans? arXiv preprint arXiv:2408.13257, 2024

    Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qing- song Wen, Zhang Zhang, et al. Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans? arXiv preprint arXiv...

    2024 arXiv

  38. [46]

    A survey on model compression for large language models

    Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for large language models. Transactions of the Association for Computational Linguis- tics, 12:1556–1577, 2024. 1

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.