Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantization

Jaewoong Sim; Jungi Lee; Wonbeom Lee

Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantization

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.12930 v1 pith:AH3LPLTD submitted 2024-06-16 cs.LG cs.AR

Jungi Lee , Wonbeom Lee , Jaewoong Sim This is my paper · ORCID

classification cs.LGcs.AR

keywords decomposedinferencetendercomputelanguagelargellmsmatrices

verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments

read the original abstract

Large language models (LLMs) demonstrate outstanding performance in various tasks in machine learning and have thus become one of the most important workloads in today's computing landscape. However, deploying LLM inference poses challenges due to the high compute and memory requirements stemming from the enormous model size and the difficulty of running it in the integer pipelines. In this paper, we present Tender, an algorithm-hardware co-design solution that enables efficient deployment of LLM inference at low precision. Based on our analysis of outlier values in LLMs, we propose a decomposed quantization technique in which the scale factors of decomposed matrices are powers of two apart. The proposed scheme allows us to avoid explicit requantization (i.e., dequantization/quantization) when accumulating the partial sums from the decomposed matrices, with a minimal extension to the commodity tensor compute hardware. Our evaluation shows that Tender achieves higher accuracy and inference performance compared to the state-of-the-art methods while also being significantly less intrusive to the existing accelerators.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

SeVeDo: A Heterogeneous Transformer Accelerator for Low-Bit Inference via Hierarchical Group Quantization and SVD-Guided Mixed Precision
cs.LG 2025-12 conditional novelty 6.0 of 10

SeVeDo achieves 12.7–13.8 TOPS/W in 28nm for ViT-Base and Llama2-7B by combining SVD-based low-rank decomposition, hierarchical group quantization, and SVD-guided mixed precision.
LightNobel: Improving Sequence Length Limitation in Protein Structure Prediction Model via Adaptive Activation Quantization
cs.AR 2025-05 conditional novelty 6.0 of 10

LightNobel combines token-wise adaptive activation quantization with a custom accelerator to run protein structure prediction on sequences up to about 10,000 amino acids within 80GB memory.
Transitive Array: An Efficient GEMM Accelerator with Result Reuse
cs.AR 2025-04 conditional novelty 6.0 of 10

A GEMM accelerator exploits subset-pattern reuse in bit-sliced weights to cut multiplications, reporting 7.46x and 3.97x speedups over Olive and BitVert at comparable LLaMA perplexity.
Reassessing Layer Pruning in LLMs: New Insights and Methods
cs.LG 2024-11 conditional novelty 6.0 of 10

Trimming the final 25% of layers and fine-tuning the head and last three layers outperforms sophisticated pruning metrics and LoRA-based recovery for LLM compression.
FineQ: Software-Hardware Co-Design for Low-Bit Fine-Grained Mixed-Precision Quantization of LLMs
cs.LG 2025-04 conditional novelty 5.0 of 10

FineQ quantizes LLM weights to 2.33 bits by protecting outliers inside 3-weight clusters and adds a temporal-coding accelerator that replaces multipliers with adders.

Pith tools