Pith. sign in

REVIEW 3 cited by

Tensorized Embedding Layers for Efficient Model Compression

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1901.10787 v2 pith:7I2SQGZ3 submitted 2019-01-30 cs.CL cs.LG

Oleksii Hrinchuk , Valentin Khrulkov , Leyla Mirvakhabova , Elena Orlova , Ivan Oseledets This is my paper · ORCID
classification cs.CLcs.LG
keywords embeddinglayerscompressionlanguagemodelnaturalperformanceprocessing
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The embedding layers transforming input words into real vectors are the key components of deep neural networks used in natural language processing. However, when the vocabulary is large, the corresponding weight matrices can be enormous, which precludes their deployment in a limited resource setting. We introduce a novel way of parametrizing embedding layers based on the Tensor Train (TT) decomposition, which allows compressing the model significantly at the cost of a negligible drop or even a slight gain in performance. We evaluate our method on a wide range of benchmarks in natural language processing and analyze the trade-off between performance and compression ratios for a wide range of architectures, from MLPs to LSTMs and Transformers.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. On the Downstream Performance of Compressed Word Embeddings

    cs.LG 2019-09 conditional novelty 7.0 of 10

    The eigenspace overlap score, a subspace-preservation measure between compressed and uncompressed embeddings, predicts downstream performance and selects better compressed embeddings more reliably than prior quality measures.

  2. Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems

    cs.LG 2019-09 conditional novelty 6.0 of 10

    Complementary partitions let each category be represented by composing entries from several small tables, reducing embedding memory from O(|S|D) to about O(k|S|^(1/k)D) while outperforming the hashing trick.

  3. Making deep neural networks work for medical audio: representation, compression and domain adaptation

    cs.SD 2025-05 conditional novelty 4.0 of 10

    A dissertation showing that transfer learning, tensor-compressed RNNs, and domain adaptation improve infant-cry models, while releasing the CryCeleb dataset for cry-based infant recognition.

Pith tools