Opt.Gear Technical Report

Juneyoung Park; Youngwook Kwon

REVIEW 4 major objections 7 minor 52 references

Opt.Gear Technical Report

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces a hybrid architecture, Opt.Gear, that replaces most local-attention layers with a convolutional gate mixer, claiming up to 4.9x faster prefill and decode on NPUs and 20 tokens per second on a microcontroller.

desk verdict A legit engineering report with real released models and a credible MCU measurement, but the headline speedup and KV-cache claims rest on a missing layer split and no matched baseline. read the letter →

arxiv 2608.01034 v2 pith:ZCEMBEKO submitted 2026-08-02 cs.CL

Juneyoung Park , Youngwook Kwon This is my paper

classification cs.CL

keywords ConvKV-GatedMixerhybridattentionKVcachereductionon-deviceLLMmicrocontrollerinferencequantization-awaretrainingKorean-Englishbilingualdata-efficientpretraining

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The report presents Opt.Gear, a family of dense language models at 1M, 270M, and 1B parameters that targets phones, NPUs, and microcontrollers with a 64K context design. Its central claim is that a new operator, the ConvKV-Gated Mixer, can take over the work of most sliding-window attention layers while keeping output quality, replacing the sequence-length-dependent KV cache with a fixed-size convolutional state. If that holds, long-context inference on small devices needs far less memory traffic, and a 1M-parameter model can generate at 20 tokens per second on an ARM Cortex-M7 microcontroller. The paper also claims this capability comes from a 0.5T-token, non-distilled training recipe that beats or matches models trained on far more data.

What carries the argument

The ConvKV-Gated Mixer: a local sequence operator that applies causal depthwise convolution to the key and value streams, gates the convolved value by $g_t = \sigma(q_t \odot \tilde{k}_t)$, and projects the result. It replaces sliding-window attention layers with static linear, convolution, and element-wise operations, keeping a persistent state of size $O(2 L_{conv} d_{kv})$ that is independent of context length.

What would settle it

Train a matched 1B model on the same 0.5T tokens and compute budget, replacing every ConvKV-Gated Mixer layer with equal-capacity sliding-window attention. If the hybrid model does not show both a smaller KV cache at 64K context and higher prefill/decode tokens per second at equal MMLU and GPQA scores, the central claim fails.

Watch

Extended reading notes

Core claim

The central claim is that a hybrid sequence-mixing block can preserve the useful local behaviour of attention while shedding its cache. Each token is projected into query, key, and value streams; the key and value streams are passed through causal depthwise 1D convolutions; the convolved key modulates the current query through a sigmoid gate; and that gate selects from the convolved value before an output projection. This ConvKV-Gated Mixer uses no token-to-token score matrix and no softmax, and its only persistent state is a fixed convolution buffer of width $L_{conv}$ (3 for the 270M and 1B models, 4 for the 1M model). The paper argues that the remaining GQA layers still provide global rou

Load-bearing premise

The efficiency story assumes the ConvKV-Gated Mixer preserves output quality as well as the sliding-window attention layers it replaces, but the report never measures that substitution directly.

Editorial extensions

If this is right

For the 270M and 1B models, local-layer KV-cache memory stops scaling with sequence length, so 64K-context decoding spends less time moving state between memory and compute.
On the measured mobile NPU, the 1B model's prefill throughput of 7,042 tokens/sec exceeds the same-scale comparison models, suggesting long prompts can be processed at server-like speed on a phone.
The 1M model's 20 tokens/sec at W4A32 on a Cortex-M7 means an auto-regressive language model can support interactive, short-command use without any server or mobile accelerator.
The 0.5T-token, no-distillation training recipe places competitive or better scores on Korean and English benchmarks against models trained on 2T-36T tokens, implying data composition can substitute for raw token count.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An untested consequence of the design is that the number of remaining GQA layers is the new bottleneck; if the mixer is as lossless as claimed, replacing more attention layers should push the speedup beyond the reported 4.9x.
The MCU result was measured with a 160-token sequence and a 40-token generation cap, so the 20 TPS figure describes short command loops; longer interactions on the same chip would trade speed for state memory.
A direct test the report leaves open is a layer-by-layer ablation that swaps ConvKV layers back to sliding-window attention; that would separate the mixer's contribution from the general benefit of having fewer attention layers.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a legitimate engineering contribution — three trained models, open weights, deployment binaries, and an honest, well-instrumented MCU timing. But the headline claims outrun the data, and the missing per-layer architecture breakdown is the load-bearing issue, not a cosmetic omission.

What’s new: the ConvKV-Gated Mixer is a specific combination — causal depthwise convolution on K and V, query-key gating, GQA, separate RoPE scales — that keeps a fixed-size convolutional state instead of a sequence-length-dependent cache. The pieces are individually known, but this configuration is new and the released models are real. The 20 TPS on an ARM Cortex-M7 with W4A32 is concrete and credible: measured inside firmware, with build footprints reported. That is the most valuable part of the report. The QAT scale-initialization story is also clearly described and sensible.

Soft spots, in proportion: the biggest is that Table 1 gives layer counts (26/18/5) but not the split among global GQA, local GQA, and ConvKV layers. The paper itself states the mixer does not eliminate the KV cache from the entire model and that local attention layers keep a fixed-window cache. So the actual cache size, and hence the speedup attribution, depends entirely on that missing split. If even a few global layers remain, the 'fixed-size state' claim weakens a lot. Second, the abstract's 'up to 4.9x' does not match any ratio in Table 6; the closest is about 5x on Apple NPU decode vs EXAONE in Table 7, but it is never stated as such. That is sloppy. Third, there is no matched pure-attention model trained on the same 0.5T tokens, so the quality/efficiency trade-off cannot be cleanly attributed to the architecture. Fourth, 'most data-efficient foundation model' compares across different data mixes and compute budgets, not a controlled scaling study. Finally, the 1M model is trained only on a 50k GPT-generated board-control set; calling it a 'foundation model' is an overstatement.

None of this is fatal to the engineering value. The report is transparent about limitations, and the weights are public, so the open questions can be settled from the artifacts. Who it is for: practitioners building on-device small LLMs and anyone tracking hybrid attention alternatives. It deserves a serious referee, who should ask for the layer split, a matched ablation, and a precise statement of which comparison yields the 4.9x. I would engage with it and, after those corrections, take it.

Referee Report

4 major / 7 minor

Summary. The paper introduces Opt.Gear, a family of dense decoder-only models (1M, 270M, 1B) for on-device deployment, with a hybrid architecture that combines a convolutional key-value gated mixer (ConvKV-Gated Mixer) with local/global GQA layers. The authors claim that replacing many local attention layers with the mixer reduces the sequence-length-dependent KV cache to a fixed-size convolutional state, yielding up to 4.9x faster prefill and decoding on NPUs, and that Opt.Gear-1M reaches 20 tokens/s with W4A32 quantization on an ARM Cortex-M7 MCU. The report includes pretraining on a curated 0.5T-token subset (from a 2T corpus), long-context extension to 64K, QAT details, benchmark tables for English/Korean tasks, and device-side inference measurements on Qualcomm and Apple runtimes.

Significance. If the efficiency claims are substantiated, the paper would make a useful contribution to on-device LLM design: the ConvKV-Gated Mixer is a hardware-friendly local operator, the device-side timing is unusually concrete (HAL_GetTick instrumentation, ELF section sizes), and the released weights and runtimes would support reproducible follow-up work. The QAT section is also clearly described. However, the headline speedup and KV-cache savings are not derivable from the reported tables, and the lack of a per-layer architecture breakdown or a matched ablation means the attribution of the results to the new mixer is not yet established.

major comments (4)

[§2.1.3, Table 1 and §2.1.2] The central efficiency claim depends on the number of ConvKV, local GQA, and global GQA layers, but Table 1 reports only total layer counts (26/18/5). Equations in §2.1.2 give per-layer cache sizes (Cache_global=O(2 C d_kv), Cache_local=O(2 W d_kv), Cache_ConvKV=O(2 L_conv d_kv)) but not their multiplicities. At C=65,536 and d_kv=256, one global layer needs ~33.5M values (~67 MB fp16), while one local layer at W=512 needs 262K values; if even a few global layers remain, the total cache is not fixed-size and savings are much smaller than implied. Relatedly, the abstract's 'up to X4.9 faster' speedup is not derivable from Table 6: the largest prefill ratio is 7042/2466≈2.86 and the largest decode ratio is 86/41≈2.10. Please provide the layer split, a total-cache comparison at 64K, and the derivation of the 4.9x number.
[§4.2.1, Tables 3-4] No ablation compares the hybrid model with a matched pure-attention model trained on the same data, tokenizer, and compute budget. The benchmark results therefore cannot isolate the contribution of the ConvKV-Gated Mixer; differences could be due to data mixture, tokenizer, training budget, or post-training choices. Section 5 correctly notes that data scale, composition, and model capacity were not independently varied, but an architecture-level claim needs an ablation (e.g., replacing the mixer with local attention under identical training) or at least a per-layer sensitivity analysis.
[§4.1.1 vs Abstract/§3.1] The abstract and §3.1 state that Opt.Gear is trained 'without knowledge distillation,' but §4.1.1 describes Opt.Gear-1M post-training on 50,000 GPT-generated synthetic prompt-response pairs (stm32_cmd_synth). Using outputs of a proprietary teacher model to construct training targets is a form of teacher-derived supervision. Please clarify whether the no-distillation claim applies only to Opt.Gear-270M/1B, and discuss how GPT-generated data interacts with the data-efficiency and no-distillation statements.
[§3.2] The learning-rate schedule says the constant 1e-4 learning rate is maintained 'until the model consumes 2T training tokens,' but the stated pretraining budget is 0.5T tokens. This discrepancy makes the training recipe irreproducible. Please correct the number or explain the intended schedule.

minor comments (7)

[Table 2] The quantized format is listed as 'W4A32 (q4 weights, q16 embedding/LM head)'. Since the embedding and LM head are weights, this is inconsistent; clarify that W4A32 refers to transformer weights/activations while embedding/LM head are kept at q16.
[§2.1.4] The 20 tokens/s figure is measured with a 160-token sequence length and a 40-token generation limit. This configuration should be stated wherever the 20 TPS headline appears, or the claim should be labeled 'up to' under that specific setup.
[Eq. (1)] Notation typos: '(W)' and '(X)' should be plain W and X. Also, the equation writes W before the arg min but uses W inside the norm; make the symbols consistent.
[Figure 2] Figure 2 is referenced in the architecture section but no figure content is present in the provided text. If the figure is missing from the submission, include it.
[References] Reference [3] (ONNX) points to the Hugging Face nanotron repository URL; this citation should be corrected.
[Abstract and §4.2.1] The claim that Opt.Gear is 'the most data-efficient of the existing foundation models' is not supported by the evidence in Tables 3-4, which compare models with different parameter counts, training budgets, and distillation status. Suggest softening this to a comparative statement or providing a controlled efficiency-per-token analysis.
[Throughout] Several typos remain: 'real-tim inference', 'scalemodels', 'attentenion', and duplicate words in references. A proofreading pass is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: KV-cache savings are explicit complexity counts; benchmarks are external measurements; the missing layer split is a verification gap, not a self-referential derivation.

full rationale

The paper's load-bearing derivations are explicit complexity counts and external benchmark measurements, not fitted parameters renamed as predictions. The KV-cache formulas (Cache_global = O(2C d_kv), Cache_local = O(2W d_kv), Cache_ConvKV = O(2L_conv d_kv)) are algebraic identities from the cache definitions, and the purported savings follow directly from counting, not from any circular fit. The ConvKV-Gated Mixer is defined by equations (element-wise gate, causal depthwise convolution) and its benefit is argued from the cache-size formulas; no equation reduces the architecture's output to its input by construction. The QAT section follows GPTQ/LSQ methodology with calibration-data-based scale initialization, and its effectiveness is judged by comparison with RTN/AWQ/GPTQ baselines, not by renaming the optimized scales as predictions. The MCU throughput (20 TPS) is presented as an instrumented firmware measurement with concrete build-footprint numbers, not as a predicted outcome of a fitted model. The claims about data efficiency are empirical comparisons of benchmark scores and training token counts, which are independent of any fitted parameter inside the paper. The main weaknesses identified by the reader—the omitted per-layer split between ConvKV, local attention, and global attention, and the absence of a matched pure-attention ablation—are transparency and attribution gaps that could undermine the causal claim about ConvKV's contribution, but they are not circularity: the speedup numbers, if accurate, are measured, and the cache formulas do not assume the conclusion. The paper's own limitations section explicitly acknowledges that data scale, data composition, and model capacity were not independently scaled, which is an honest statement of confounds rather than an attempt to disguise an input as an output. No self-citation is load-bearing; KORMo, GPTQ, LSQ, and QK-LN are external prior work. There is no uniqueness theorem imported from the authors and no ansatz smuggled in via citation. Therefore no circular step can be exhibited, and the appropriate score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims rest on a handful of hand-chosen hyperparameters (kernel size, window size) and on unreported layer counts. There are no new physical entities; the architecture is composed of known operations. The data-efficiency claim relies on assumptions about the fairness of cross-model comparison.

free parameters (5)

Conv kernel size L_conv = 3 (270M/1B), 4 (1M)
Chosen by hand; directly sets the ConvKV cache size O(2 L_conv d_kv), central to the KV-cache reduction claim.
Sliding-window size W = 512
Chosen by hand; sets the baseline local attention cache size O(2 W d_kv) that the paper uses to claim savings.
Number of GQA layers among total layers = Not reported
The paper lists total layers (26/18/5) but not how many are global attention vs. local attention vs. ConvKV. The total KV-cache footprint and any speedup depend on this count, so the claim is unquantifiable as written.
Training token budget = 0.5T
Chosen resource; the data-efficiency claim ('most data-efficient') compares this number against baselines' 2T-36T budgets without controlling for data quality, architecture, or eval settings.
Data mixture ratios (English/Korean/math) = 92/6/2
Chosen by hand; the data-efficiency result depends on this mixture, but no ablation of the mixture is provided.

assumptions (4)

domain assumption Perplexity-based evaluation on multiple-choice benchmarks is a fair cross-model comparison.
Section 4.2.1 states all results come from LM Evaluation Harness, but does not report prompts, few-shot settings, or variance. Small differences (e.g., 21.2 vs 23.7 GPQA) may be within noise, so the comparisons rest on this assumption.
domain assumption The QAIRT NPU compilation is representative of the claimed speedup.
Section 4.2.2 relies on a single runtime (QAIRT) for the headline speedup; llama.cpp NPU results show Opt.Gear is 290 tokens/s, slower than LFM2.5 (2691), so the speedup depends heavily on this runtime and on undisclosed graph optimizations.
ad hoc to paper GPT-generated synthetic commands provide a valid training signal for the 1M model without constituting knowledge distillation.
Section 4.1.1 uses stm32_cmd_synth, a GPT-generated dataset, while the abstract claims training 'without knowledge distillation'. This is at odds unless synthetic-data generation is excluded from the definition.
domain assumption The benchmark tables (Tables 3-5) use comparable, correctly configured baselines.
The paper states results 'may differ from other reported scores', and no eval details are given for baselines. The 'Distilled ✓' column for baselines is taken from their reports, but the effect of distillation is not controlled.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Opt.Gear Technical Report." pith.science (2026). https://pith.science/paper/ZCEMBEKO

@misc{pith2026260801034,
  author       = {Pith},
  title        = {Pith review of: Opt.Gear Technical Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZCEMBEKO}},
  note         = {Machine review of arXiv:2608.01034}
}

read the original abstract

We introduce OptGear, a foundation model designed for efficient on-device deployment, real-tim inference, and strong task capability. It includes a dense model (1M, 270M, and 1B) with a context length of 64K. We designed a new hybrid architecture that combines a convolutional key-value gated mixer with local-global attention to reduce the KV-cache memory that tends to increase exponentially with long context. This architecture delivers up to X4.9 faster prefill and decoding speeds on the NPUs compared to models of a similar scale models. From a 2T tokens candidate corpus, OptGear is trained on a curated 0.5T tokens subset without knowledge distillation. This is the most data-efficient of the existing foundation models. All models are released with open weights and deployment binaries for ONNX, Qualcomm NPU, and Apple ANE making OptGear a practical base for edge applications that need fast, memory-efficient inference and strong task capabilities. Furthermore, to expand the ecosystem of on-device generative language models, we are introducing the OptGear-1M that can be deployed on Micro-Controller Units (MCUs), a Tiny Language Model (TLM). OptGear-1M is the first generative language model to achieve 20 TPS with W4A32 quantization on the ARM Cortex-M7 CPU of the STM32H747I-DISCO.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 14 canonical work pages

[1]

Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023. URLhttps: //arxiv.org/abs/2305.13245

arXiv 2023
[2]

Paech, Paul Pak, Rom N

Alexander Amini, Anna Banaszak, Harold Benoit, Arthur Böök, Tarek Dakhran, Song Duong, Alfred Eng, Fernando Fernandes, Marc Härkönen, Anne Harrington, Ramin Hasani, Saniya Karwa, Yuri Khrustalev, Maxime Labonne, Mathias Lechner, Valentine Lechner, Simon Lee, Zetian Li, Noel Loo, Jacob Marks, Edoardo Mosca, Samuel J. Paech, Paul Pak, Rom N. Parnichkun, Ale...

arXiv 2025
[3]

Chun-Wei Andreas and contributors. Onnx. https://github.com/huggingface/nanotron, 2024. Open Neural Network Exchange

work page 2024
[4]

Core ml tools.https://github.com/apple/coremltools

Apple. Core ml tools.https://github.com/apple/coremltools

work page
[5]

Exaone 4.0: Unified large language models integrating non-reasoning and reasoning modes, 2026

Kyunghoon Bae, Eunbi Choi, Kibong Choi, Stanley Jungkyu Choi, Yemuk Choi, Kyubeen Han, Seokhee Hong, Junwon Hwang, Taewan Hwang, Joonwon Jang, Hyojin Jeon, Kĳeong Jeon, Gerrard Jeongwon Jo, Hyunjik Jo, Jiyeon Jung, Euisoon Kim, Hyosang Kim, Jihoon Kim, Joonkee Kim, Seonghwan Kim, Soyeon Kim, Sunkyoung Kim, Yireun Kim, Yongil Kim, Youchul Kim, Edward Hwayo...

arXiv 2026
[6]

PIQA: reasoning about physical commonsense in natural language

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. InThe Thirty-FourthAAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in A...

doi:10.1609/aaai.v34i05.6239 2020
[7]

Generating long sequences with sparse transformers,

Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers,
[8]

Think you have solved question answering? try arc, the AI2 reasoning challenge.CoRR, abs/1803.05457, 2018

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge.CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457

arXiv 2018

Show all 52 references

[9]

Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, 2024

Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, 2024. URLhttps://arxiv.org/abs/2405.21060. 17

2024 arXiv
[10]

Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang, Carlos Riquelme, Matthias Mindere...

2023 arXiv
[11]

Esser, Jeffrey L

Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S. Modha. Learned step size quantization, 2020. URLhttps://arxiv.org/abs/1902.08153

2020 arXiv
[12]

Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023. URLhttps://arxiv.org/abs/2210.17323

2023 arXiv
[13]

The language model evaluation harness, 07 2024

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

2024
[14]

How to train long-context language models (effectively), 2025

Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. How to train long-context language models (effectively), 2025. URLhttps://arxiv.org/abs/2410.02660

2025
[15]

llama.cpp.https://github.com/ggml-org/llama.cpp, 2023

Georgi Gerganov and contributors. llama.cpp.https://github.com/ggml-org/llama.cpp, 2023. LLM inference in C/C++

2023
[16]

AaronGrattafiori, AbhimanyuDubey, AbhinavJauhri, AbhinavPandey, AbhishekKadian, AhmadAl-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hi...

2024 arXiv
[17]

Measuring massive multitask language understanding.arXivpreprint arXiv:2009.03300, 2020

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXivpreprint arXiv:2009.03300, 2020

2009 arXiv
[18]

Distilling the knowledge in a neural network, 2015

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015. URLhttps: //arxiv.org/abs/1503.02531

2015 arXiv
[19]

nanotron.https://github.com/huggingface/nanotron, 2024

HuggingFace. nanotron.https://github.com/huggingface/nanotron, 2024. Pretraining models made easy

2024
[20]

Quantization and training of neural networks for efficient integer-arithmetic-only inference,

Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference,
[21]

Kobest: Korean balanced evaluation of significant tasks, 2022

Dohyeong Kim, Myeongjun Jang, Deuk Sin Kwon, and Eric Davis. Kobest: Korean balanced evaluation of significant tasks, 2022. URLhttps://arxiv.org/abs/2204.04541

2022 arXiv
[22]

Click: A benchmark dataset of cultural and linguistic intelligence in korean, 2024

Eunsu Kim, Juyoung Suk, Philhoon Oh, Haneul Yoo, James Thorne, and Alice Oh. Click: A benchmark dataset of cultural and linguistic intelligence in korean, 2024. URLhttps://arxiv.org/abs/2403.06412

2024 arXiv
[23]

Kormo: Korean open reasoning model for everyone, 2025

Minjun Kim, Hyeonseok Lim, Hangyeol Yoo, Inho Won, Seungwoo Song, Minkyung Cho, Junhun Yuk, Changsu Choi, Dongjae Shin, Huige Lee, Hoyun Song, Alice Oh, and Kyungtae Lim. Kormo: Korean open reasoning model for everyone, 2025. URLhttps://arxiv.org/abs/2510.09426

2025
[24]

Gonzalez, Hao Zhang, and Ion Stoica

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023. URLhttps://arxiv.org/abs/2309.06180

2023 arXiv
[25]

Awq: Activation-aware weight quantization for llm compression and acceleration,

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for llm compression and acceleration,
[26]

Decoupled weight decay regularization.arXiv preprintarXiv:1711.05101, 2017

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprintarXiv:1711.05101, 2017

2017 arXiv
[27]

NVIDIAH200tensorcoreGPU,2024

NVIDIA. NVIDIAH200tensorcoreGPU,2024. URL https://www.nvidia.com/ko-kr/data-center/h200/. Accessed: 2024-11-19

2024
[28]

Nemotron-h: A family of accurate and efficient hybrid mamba-transformer models,

NVIDIA, :, Aaron Blakeman, Aarti Basant, Abhinav Khattar, Adithya Renduchintala, Akhiad Bercovich, Aleksander Ficek, Alexis Bjorlin, Ali Taghibakhshi, Amala Sanjay Deshmukh, Ameya Sunil Mahabaleshwarkar, Andrew Tao, AnnaShors,AshwathAithal,AshwinPoojary,AyushDattagupta,Balaram...
[29]

Qualcomm ai engine direct

Qualcomm. Qualcomm ai engine direct. https://www.qualcomm.com/developer/software/ qualcomm-ai-engine-direct-sdk
[30]

GPQA: A graduate-level google-proof q&a benchmark.arXivpreprintarXiv:2311.12022, 2023

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. GPQA: A graduate-level google-proof q&a benchmark.arXivpreprintarXiv:2311.12022, 2023. 20

2023 arXiv
[31]

Winogrande: An adversarial winograd schema challenge at scale, 2019

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019. URLhttps://arxiv.org/abs/1907.10641

2019 arXiv
[32]

Neural machine translation of rare words with subword units,

Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units,
[33]

Glu variants improve transformer, 2020

Noam Shazeer. Glu variants improve transformer, 2020. URLhttps://arxiv.org/abs/2002.05202

2020 arXiv
[34]

Kmmlu: Measuring massive multitask language understanding in korean,

Guĳin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. Kmmlu: Measuring massive multitask language understanding in korean,
[35]

Hae-rae bench: Evaluation of korean knowledge in language models, 2024

Guĳin Son, Hanwool Lee, Suwan Kim, Huiseo Kim, Jaecheol Lee, Je Won Yeom, Jihyu Jung, Jung Woo Kim, and Songseong Kim. Hae-rae bench: Evaluation of korean knowledge in language models, 2024. URLhttps: //arxiv.org/abs/2309.02706

2024 arXiv
[36]

Stm neural network deployment framework.https://stm32ai.st.com/stm32-cube-ai/

STMicroelectronics. Stm neural network deployment framework.https://stm32ai.st.com/stm32-cube-ai/
[37]

Roformer: Enhanced transformer with rotary position embedding, 2023

Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. URLhttps://arxiv.org/abs/2104.09864

2023 arXiv
[38]

GemmaTeam,AishwaryaKamath,JohanFerret,ShreyaPathak,NinoVieillard,RamonaMerhej,SarahPerrin,Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, SabelaRamos, EdouardYvinec, MichelleCasbon, EtiennePot, IvoPe...

2025 arXiv
[39]

Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, Jiaxi Hu, Fanqing Meng, Chengyin Liu, Xin Men, Songlin Yang, Zhiyuan Li, Wentao Li, Enzhe Lu, Weizhou Liu, Yanru Chen, Weixin Xu, Longhui Yu, Yejie Wang, Yu Fan, Longguang Zhong, Enming Yuan, Dehao Zhang, Yizhi Zhang, T. Y. Liu, ...

2025 arXiv
[40]

An empirical study of mamba-based language models, 2024

Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vĳay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, Garvit Kulshreshtha, Vartika Singh, Jared Casper, Jan Kautz, Mohammad Shoeybi, and Bryan Catanzaro. An empirical study of mamba-b...

2024 arXiv
[41]

Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.CoRR, abs/2406.01574,

Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language unders...

arXiv
[42]

Huggingface’s transformers: State-of-the-art natural language processing, 2020

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault,RémiLouf,MorganFuntowicz,JoeDavison,SamShleifer,PatrickvonPlaten,ClaraMa,YacineJernite,JulienPlu, CanwenXu,TevenLeScao,SylvainGugger,MariamaDrame,QuentinLhoest,a...

2020 arXiv
[43]

Qwen3 technical report, 2025

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

2025 arXiv
[44]

Gated delta networks: Improving mamba2 with delta rule, 2025

Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule, 2025. URLhttps://arxiv.org/abs/2412.06464

2025 arXiv
[45]

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish yoursentence? InAnnaKorhonen,DavidR.Traum,andLluísMàrquez,editors, Proceedingsofthe57thConference of the Association for Computational Linguistics, ACL 2019, Florence...

2019 doi
[48]

URLhttps://doi.org/10.48550/arXiv.2406.01574

doi arXiv
[2016]

URLhttps://arxiv.org/abs/1508.07909

arXiv
[2017]

URLhttps://arxiv.org/abs/1712.05877. 19

arXiv
[2019]

URLhttps://arxiv.org/abs/1904.10509

1904 arXiv
[2024]

URLhttps://arxiv.org/abs/2402.11548

arXiv
[2025]

URLhttps://arxiv.org/abs/2504.03624

arXiv
[2026]

URLhttps://arxiv.org/abs/2306.00978

arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023. URLhttps: //arxiv.org/abs/2305.13245

arXiv 2023

[2] [2]

Paech, Paul Pak, Rom N

Alexander Amini, Anna Banaszak, Harold Benoit, Arthur Böök, Tarek Dakhran, Song Duong, Alfred Eng, Fernando Fernandes, Marc Härkönen, Anne Harrington, Ramin Hasani, Saniya Karwa, Yuri Khrustalev, Maxime Labonne, Mathias Lechner, Valentine Lechner, Simon Lee, Zetian Li, Noel Loo, Jacob Marks, Edoardo Mosca, Samuel J. Paech, Paul Pak, Rom N. Parnichkun, Ale...

arXiv 2025

[3] [3]

Chun-Wei Andreas and contributors. Onnx. https://github.com/huggingface/nanotron, 2024. Open Neural Network Exchange

work page 2024

[4] [4]

Core ml tools.https://github.com/apple/coremltools

Apple. Core ml tools.https://github.com/apple/coremltools

work page

[5] [5]

Exaone 4.0: Unified large language models integrating non-reasoning and reasoning modes, 2026

Kyunghoon Bae, Eunbi Choi, Kibong Choi, Stanley Jungkyu Choi, Yemuk Choi, Kyubeen Han, Seokhee Hong, Junwon Hwang, Taewan Hwang, Joonwon Jang, Hyojin Jeon, Kĳeong Jeon, Gerrard Jeongwon Jo, Hyunjik Jo, Jiyeon Jung, Euisoon Kim, Hyosang Kim, Jihoon Kim, Joonkee Kim, Seonghwan Kim, Soyeon Kim, Sunkyoung Kim, Yireun Kim, Yongil Kim, Youchul Kim, Edward Hwayo...

arXiv 2026

[6] [6]

PIQA: reasoning about physical commonsense in natural language

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. InThe Thirty-FourthAAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in A...

doi:10.1609/aaai.v34i05.6239 2020

[7] [7]

Generating long sequences with sparse transformers,

Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers,

[8] [8]

Think you have solved question answering? try arc, the AI2 reasoning challenge.CoRR, abs/1803.05457, 2018

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge.CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457

arXiv 2018