Pith. sign in

REVIEW 2 major objections 5 minor 47 references

On the Downstream Performance of Compressed Word Embeddings

T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read One subspace score predicts which compressed word embeddings work

desk verdict A genuinely useful new score for ranking compressed embeddings, backed by a broad empirical study; the theory is narrower than claimed and Theorem 3 has a proof gap, but the paper is worth refereeing. read the letter →

arxiv 1909.01264 v2 pith:6R665PCJ submitted 2019-09-03 cs.LG stat.ML

Avner May , Jian Zhang , Tri Dao , Christopher Ré This is my paper · ORCID
classification cs.LGstat.ML MSC 68T0562J0568P30
keywords eigenspaceoverlapscorewordembeddingcompressiongeneralizationboundsuniformquantizationmodelselectionlinearregressionsubspacedownstreamNLPperformance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the downstream quality of a compressed word embedding is governed by one number: how much of the uncompressed embedding's left singular subspace survives in the compressed embedding. The authors call this the eigenspace overlap score, prove that for linear and logistic regression the expected generalization gap between compressed and uncompressed embeddings shrinks as this score grows, and show that uniform quantization achieves high scores with few bits. If the claim is right, a practitioner can choose between compressed embeddings by computing an SVD-based score instead of training a model for each task. The same connection explains why a simple uniform quantization method is competitive with more elaborate learned compression schemes.

What carries the argument

The load-bearing object is the eigenspace overlap score $E(X,\tilde X)$, the squared Frobenius norm of the projection of the uncompressed left-singular basis $U$ onto the compressed basis $\tilde U$, normalized by $\max(d,k)$. It matters because linear predictions $Xw$ depend on the data only through $U$, so two embeddings with overlapping left singular subspaces can emulate each other's linear models. Theorem 1 turns that geometric intuition into an exact risk gap, while the Davis-Kahan $\sin(\Theta)$ theorem connects the perturbation caused by quantization to a lower bound on the score, and the score itself is computable by SVD in $O(n\max(d,k)^2)$ time.

What would settle it

Take a fixed task and two compressed embeddings $\tilde X_A,\tilde X_B$ with $E(X,\tilde X_A)>E(X,\tilde X_B)$; if $\tilde X_A$ consistently performs worse on labels generated from directions orthogonal to its span, the selection claim fails. Concretely, synthesize labels $\bar y = U_\perp z$ (using left singular directions outside $\operatorname{span}(\tilde U)$) and check whether the exact gap in Theorem 1 still holds; it should not, because the theorem's premise is violated.

Watch

Extended reading notes

Core claim

The central claim is that the eigenspace overlap score $E(X,\tilde X) = \frac{1}{\max(d,k)}\|U^T\tilde U\|_F^2$ is the quantity that controls the expected downstream loss of a compressed embedding. For fixed-design linear regression with true labels drawn randomly from the span of the uncompressed embedding's left singular vectors, Theorem 1 gives an exact average-case gap $E_{\bar y}[R_{\bar y}(\tilde X)-R_{\bar y}(X)] = \frac{d}{n}(1-E(X,\tilde X)) - c^2\frac{d(d-k)}{n^2}$; for Lipschitz losses the same score appears in a matching upper bound. The paper also proves that a uniform quantizer with $b$ bits per entry attains expected eigenspace overlap at least $1-\epsilon$ once $b = O(\log(1/\epsilon))$, and that the score is a more accurate and robust selection criterion than existing compression-quality measures, with up to $2\times$ lower selection error rates on the tasks studied.

Load-bearing premise

The average-case analysis assumes the true label vector is random in the span of the uncompressed embedding's left singular subspace; if a real task's labels concentrate in directions that compression discards, the score can misrank embeddings and the guarantees do not apply.

Editorial extensions

If this is right

  • Compressed embeddings can be ranked without training a downstream model, so model selection over compression methods becomes a cheap linear-algebra computation.
  • Uniform quantization has a formal guarantee: at a logarithmic number of bits it keeps the expected eigenspace overlap close to 1, explaining why it matches or beats learned compression methods.
  • In the low-noise, low-dimensional regime the risk gap is approximately $\frac{d}{n}(1-E)$, so maximizing eigenspace overlap is approximately minimizing the expected loss gap.
  • The score carries the same information as a weighted variant of projected reconstruction error, so practitioners can also use a weighted reconstruction loss as a proxy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural testable extension is to use the gradient of the eigenspace overlap score as a training objective for learned compression, since the exact Theorem 1 expression suggests this directly minimizes expected downstream loss under the average-case label model; the paper does not take this step.
  • Because the proof relies only on the SVD of the feature matrix, the same score plausibly transfers to other linear-feature settings such as graph node embeddings or kernel approximations, even though the paper only demonstrates word embeddings.
  • The strong empirical correlation on neural models hints that the left singular subspace of input features is the main channel for downstream signal in these architectures; verifying this on a wider set of nonlinear models would be a separate claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes the eigenspace overlap score E(X, X̃) as a measure of compressed word embedding quality, defined as the normalized squared Frobenius inner product of the left singular subspaces of the original and compressed embedding Gram matrices. The authors derive average-case generalization bounds for fixed-design linear and logistic regression in which the expected risk difference between models trained on compressed and uncompressed embeddings is controlled by E. They also state a lower bound on E for uniformly quantized embeddings via the Davis-Kahan sin(Theta) theorem, and they validate the score empirically as a correlation measure and as a selection criterion on question answering, sentiment analysis, and GLUE tasks. The empirical study spans GloVe, fastText, and BERT WordPiece embeddings and multiple compression methods, with error bars across five random seeds.

Significance. If the claims hold, the eigenspace overlap score is a practically valuable, training-free measure for selecting compressed embeddings, and the paper provides a clean average-case analysis that goes beyond worst-case spectral approximation bounds. The strengths include an exact identity for the linear regression risk difference under an isotropic label model (Theorem 1), a Lipschitz-loss extension (Theorem 2), a broad and reproducible empirical evaluation with standard deviations and hyperparameter tables, and a public implementation of the quantization method. However, the proof of Theorem 3 contains a load-bearing technical flaw in the Davis-Kahan application, and the theoretical connection to downstream performance is established only under an identity-covariance label model that is not verified on the real tasks. These issues limit the current theoretical guarantees, although the empirical results are substantial and the Davis-Kahan argument appears repairable.

major comments (2)
  1. [Appendix C.1, Theorem 3 and Theorem 5] The Davis-Kahan sin(Theta) application in the proof of Theorem 3 is not valid as stated. The authors set a0 = sigma_min(K), a1 = infinity, and delta = sigma_min(K), so the excluded interval in Theorem 5 is (0, infinity). Since K̃ = X̃X̃ᵀ has rank at most k < n, the block R1 of the perturbed Gram matrix contains zero eigenvalues, and 0 is not excluded from the interval (0, infinity). Thus the condition that the eigenvalues of R1 be excluded from (a0 - delta, a1 + delta) is violated, and Theorem 5 cannot be applied. This is load-bearing because Theorem 3 is the main theoretical explanation for the strong performance of uniform quantization. The argument can likely be repaired by choosing a smaller delta, e.g., delta = sigma_min(K)/2, which makes the excluded interval (sigma_min(K)/2, infinity) and excludes the zero eigenvalues, at the cost of a larger constant in the bound; the authors should correct the proof or the statement accordingly.
  2. [Section 3.1.2 and Appendix B.1] The simplified form of the theoretical claim, in which the eigenspace overlap score E(X, X̃) controls the expected risk difference, depends critically on the label vector model ȳ = Uz with z having identity covariance. The generalized theorem in Appendix B.1 shows that the expected risk difference is governed by ||ŨᵀU Σ^{1/2}||_F², and E enters only through the lower bound lambda_min(Σ) d E. For non-isotropic label covariance, two compressed embeddings with the same E can have very different expected downstream performance, and the paper's own robustness example in Appendix B.3 concedes that deleting the largest singular direction changes E by only 1/d yet can dominate generalization when the label has a large component in that direction. The empirical correlations in Tables 1 and 2 are aggregated over many compression methods and rates and do not test whether the label covariance on SQuAD, SST-1, MNLI, or QQP is sufficiently isotropic for the identity-covariance result to apply. Please either add an empirical analysis of label alignment with the singular directions (or an evaluation of the weighted score ||ŨᵀU Σ^{1/2}||_F) or substantially qualify the theoretical claim that E alone controls downstream performance.
minor comments (5)
  1. [Appendix A.1] The displayed formulas for the stochastic rounding probabilities are garbled: both P[Q̃_b,r(x) = x] and P[Q̃_b,r(x) = x̄] appear to be written as (x - x̄)/(x - x̄). Please correct the typesetting so that the two probabilities are clearly distinguished.
  2. [Appendix E.8, Table 17] The text contains the typo 'language infernece'; it should read 'language inference'.
  3. [Appendix B.2, Theorem 2 (Generalized)] The notation 'lambda_min in R' is unclear; it should be 'lambda_min(Sigma)', the smallest eigenvalue of the label covariance matrix, matching the statement of Theorem 1 (Generalized).
  4. [Appendix C.2, Figure 4(b)] The caption for Figure 4(b) should state explicitly how the scalar a is computed for the synthetic matrices; the current description says the matrices are multiplied by diagonal matrices with logarithmically spaced entries, but it does not define a in terms of the smallest singular value.
  5. [Appendix B.4, Table 4] In the table caption, 'GloVe|ρ| / fastText|ρ|' is ambiguous because the entries are presented as four numbers without a visible separator; please format the entries as 'GloVe |ρ| / fastText |ρ|' or use two columns.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the eigenspace overlap score is derived from the embeddings alone, the generalization bound follows from an explicit average-case label model, and the empirical comparisons are against external baselines.

full rationale

The derivation chain is self-contained. The eigenspace overlap score E(X,X̃) = (1/max(d,k))||U^T Ũ||_F^2 is defined directly from the left singular subspaces (Definition 1) with no downstream labels and no fitted constants. Theorem 1, and its generalized form in Appendix B.1, derives the expected risk difference under an explicitly stated average-case label model ȳ = Uz (z with identity covariance for the exact equality, general covariance Σ for the inequality), and the proof is a closed-form calculation from the fixed-design risk formula in Proposition 4. The score enters the bound because E_ȳ[||Ũ^T ȳ||^2] = ||Ũ^T U Σ^{1/2}||_F^2, not because E was fit to downstream performance. The uniform quantization lower bound (Theorem 3) is an independent application of the Davis-Kahan sin(Θ) theorem to the Gram-matrix perturbation, and the paper itself notes the bound can be vacuous for quickly decaying spectra (Appendix C.2), which is a limitation rather than a circular step. The empirical validation against PIP loss, ∆, and ∆max (Tables 1-2) is an external comparison over embeddings produced by several compression methods; no parameter is fit to the downstream labels. The only self-citations are to prior work (notably [41], with overlapping authors, used to define the ∆max baseline and to motivate why worst-case bounds are loose), but the paper's central theorems do not depend on those citations. No step reduces a prediction to its input by construction.

Assumptions & free parameters 1 free parameters · 7 assumptions · 1 invented entities

The central theoretical results rely on standard regression assumptions plus a strong structural assumption about labels living in the embedding's subspace. Theorem 3 depends on a specific spectral-gap condition that is not properly established in the proof. The experimental method adds a clipping threshold as a hyperparameter but no downstream label information enters the score or the bounds.

free parameters (1)
  • clipping threshold r* = selected per embedding by minimizing reconstruction error (Algorithm 1)
    Used in the uniform quantization method's experimental implementation; not part of the theoretical bounds, but affects empirical performance of uniform quantization.
assumptions (7)
  • standard math Fixed-design linear regression with noise y = ȳ + ε, ε zero-mean with variance σ²
    Background model used to derive risk expressions in Section A.2 and Theorem 1.
  • domain assumption True labels are random in span(U): ȳ = Uz with zero-mean z, covariance Σ
    Central average-case assumption in Section 3.1.2 and Appendix B.1; makes the eigenspace overlap score relevant. Not empirically verified.
  • standard math For Theorem 2, loss is convex, non-negative, L-Lipschitz in both arguments, and satisfies arg min_{v'} l(v',v)=v
    Assumption for the Lipschitz generalization bound in Appendix B.2.
  • domain assumption For Theorem 3, embedding entries are bounded in [-1/sqrt(d), 1/sqrt(d)] and σ_min(X) = a sqrt(n/d) with a in (0,1]
    Assumptions for the uniform quantization lower bound; real pre-trained embeddings require clipping and may have small a, making the bound vacuous (acknowledged in Appendix C.2).
  • domain assumption Unbiased stochastic rounding for the uniform quantization analysis
    Theorem 3 analyzes stochastic quantization; experiments mostly use deterministic rounding, with a separate appendix showing similar performance.
  • standard math Davis-Kahan sin(Θ) theorem as used in Appendix C.1
    External theorem used to bound eigenvector perturbation; the specific parameter choice is the source of the proof gap.
  • domain assumption Empirical transfer from linear/logistic theory to neural downstream models (DrQA, CNN, BERT)
    The theory only covers linear and logistic regression; the paper validates the score empirically on nonlinear models.
invented entities (1)
  • eigenspace overlap score independent evidence
    purpose: Quality measure for compressed embeddings based on left singular subspace overlap
    It is a deterministic function of X and X̃; its predictive value is tested on multiple tasks against existing measures. Not a hidden entity: it is directly computable and falsifiable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Downstream Performance of Compressed Word Embeddings." pith.science (2026). https://pith.science/paper/6R665PCJ

@misc{pith2026190901264,
  author       = {Pith},
  title        = {Pith review of: On the Downstream Performance of Compressed Word Embeddings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6R665PCJ}},
  note         = {Machine review of arXiv:1909.01264}
}
abstract

Compressing word embeddings is important for deploying NLP models in memory-constrained settings. However, understanding what makes compressed embeddings perform well on downstream tasks is challenging---existing measures of compression quality often fail to distinguish between embeddings that perform well and those that do not. We thus propose the eigenspace overlap score as a new measure. We relate the eigenspace overlap score to downstream performance by developing generalization bounds for the compressed embeddings in terms of this score, in the context of linear and logistic regression. We then show that we can lower bound the eigenspace overlap score for a simple uniform quantization compression method, helping to explain the strong empirical performance of this method. Finally, we show that by using the eigenspace overlap score as a selection criterion between embeddings drawn from a representative set we compressed, we can efficiently identify the better performing embedding with up to $2\times$ lower selection error rates than the next best measure of compression quality, and avoid the cost of training a model for each task of interest.

Figures

Figures reproduced from arXiv: 1909.01264 by the authors.

Figure 1
Figure 1. The PIP loss does not satisfac￾torily explain the relative downstream performance of different compression methods. spectral measures of approximation error ∆ and ∆max only achieve Spearman correlation absolute values of 0.49, 0.46, and 0.62 with the question answering test F1 score, respectively ( [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Downstream performance vs. measures of compression quality. We plot the perfor￾mance of compressed fastText embeddings on the SQuAD question answering task as a function of different measures of compression quality. The eigenspace overlap score E demonstrates better alignment with downstream performance across compression methods than the other measures. We quantify the degree of alignment using the Spearman correla… view at source ↗
Figure 3
Figure 3. Eigenspace overlap and downstream per￾formance of uniform quantization. Uniform quantiza￾tion can attain high values for the eigenspace overlap E, and match the k-means and DCCL methods for fastText embeddings on the question answering (SQuAD) task. We show that across tasks and compres￾sion rates uniform quantization consis￾tently matches or outperforms the other compression methods, in terms of both the eigenspace… view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Empirical Validation of Theorem 3. We measure the eigenspace overlap score E of uniformly quantized embeddings with the uncompressed embedding for various precisions, values of a, vocabulary sizes n, and dimensions d. We observe that 1 − E decays as the precision b and…
Figure 5
Figure 5. Figure 5: The impact of clipping and deterministic vs. stochastic quantization on the eigenspace overlap score. (a) We plot the eigenspace overlap score as a function of the clipping threshold, for precisions b ∈ {1, 2, 4} and for both stochastic and deterministic quantization. …
Figure 6
Figure 6. Figure 6: Downstream performance vs. compression rate for compressed GloVe embeddings. We evaluate the downstream performance of the different compression methods on question answering and sentiment analysis tasks, across different compression rates. For question answering, we u…
Figure 7
Figure 7. Figure 7: Downstream performance vs. compression rate for compressed fastText embeddings. We evaluate the downstream performance of the different compression methods on question answering and sentiment analysis tasks, across different compression rates. For question answering, w…
Figure 8
Figure 8. Figure 8: Downstream performance vs. compression rate for compressed BERT WordPiece embeddings. We evaluate the downstream performance of the different compression methods on all GLUE tasks except WNLI (as discussed in Appendix D.1), across different compression rates. In these …
Figure 9
Figure 9. Figure 9: Downstream performance vs. compression rate: task-specific embeddings. We plot the average BLEU4 test performance for compressed task-specific embeddings on the IWSLT’14 German-to-English translation task across five random seeds (standard deviations indicated with err…
Figure 10
Figure 10. Figure 10: Dimension vs. precision trade-off. We plot the downstream performance on ques￾tion answering (SQuAD, left) and sentiment analysis (SST-2, right) of GloVe embeddings of dimensions d ∈ {25, 50, 100, 200, 400} compressed with uniform quantization with precisions b ∈ {1, …
Figure 11
Figure 11. Figure 11: Eigenspace overlap score vs. compression rate. We plot the eigenspace overlap scores attained by different compression methods at different compression rates, for Glove, fastText, and BERT WordPiece embeddings. Across embedding types and compression rates, we observe …
Figure 12
Figure 12. Figure 12: Downstream performance vs. measures of compression quality (GloVE embed￾dings). We plot the performance of compressed GloVe embeddings on question answering (SQuAD, left column) and sentiment analysis (SST-1, right column), in terms of the different measures of compre…
Figure 13
Figure 13. Figure 13: Downstream performance vs. measures of compression quality (fastText embed￾dings). We plot the performance of compressed fastText embeddings on question answering (SQuAD, left column) and sentiment analysis (SST-1, right column), in terms of the different measures of …
Figure 14
Figure 14. Figure 14: Downstream performance vs. measures of compression quality (BERT WordPiece embeddings). We plot the performance of compressed BERT WordPiece embeddings on the two largest GLUE datasets (MNLI, left column; QQP, right column) in terms of the different measures of compre…
Figure 15
Figure 15. Figure 15: Downstream performance vs. 1/(1 − ∆1) and ∆2 (GloVe embeddings). We plot the performance of compressed GloVe embeddings on question answering (SQuAD, left column) and sentiment analysis (SST-1, right column), in terms of the 1/(1 − ∆1) and ∆2 measures of compression q…
Figure 16
Figure 16. Figure 16: Downstream performance vs. compression rate for deterministic vs. stochastic uniform quantization. We can observe, using compressed GloVe embeddings on both the SQuAD question answering task and the SST-1 sentiment analysis task, that stochastic uniform quantization (…
Figure 17
Figure 17. Figure 17: Downstream performance vs. measures of compression quality for deterministic vs. stochastic uniform quantization. We can see that regardless of whether stochastic (left plots) or deterministic (right plots) quantization is used, the eigenspace overlap score correlates…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 34 canonical work pages

  1. [1]

    Globally normalized transition-based neural networks

    Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. Globally normalized transition-based neural networks. In ACL, 2016

    work page 2016

  2. [2]

    Compressing word embeddings

    Martin Andrews. Compressing word embeddings. In ICONIP, 2016

    work page 2016

  3. [3]

    Random Fourier features for kernel ridge regression: Approximation bounds and statistical guarantees

    Haim Avron, Michael Kapralov, Cameron Musco, Christopher Musco, Ameya Velingker, and Amir Zandieh. Random Fourier features for kernel ridge regression: Approximation bounds and statistical guarantees. In ICML, 2017

    work page 2017

  4. [4]

    FBK’s machine translation and speech translation systems for the IWSLT 2014 evaluation campaign

    Nicola Bertoldi, Prashant Mathur, Nicholas Ruiz, and Marcello Federico. FBK’s machine translation and speech translation systems for the IWSLT 2014 evaluation campaign. In IWSLT, 2014

    work page 2014

  5. [5]

    Reading Wikipedia to answer open-domain questions

    Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to answer open-domain questions. In ACL, 2017

    work page 2017

  6. [6]

    Learning k-way d-dimensional discrete codes for compact embedding representations

    Ting Chen, Martin Renqiang Min, and Yizhou Sun. Learning k-way d-dimensional discrete codes for compact embedding representations. In ICML, 2018

    work page 2018

  7. [7]

    Embeddings@Twitter,

    Dan Shiebler, Chris Green, Luca Belli, Abhishek Tayal. Embeddings@Twitter,

    work page

  8. [8]

    Davis and W

    C. Davis and W. Kahan. The rotation of eigenvectors by a perturbation. iii. SIAM Journal on Numerical Analysis, 7(1):1–46, 1970

    1970

Show all 47 references
  1. [9]

    High-accuracy low-precision training

    Christopher De Sa, Megan Leszczynski, Jian Zhang, Alana Marzoev, Christopher R Aberger, Kunle Olukotun, and Christopher Ré. High-accuracy low-precision training. arXiv preprint arXiv:1803.03383, 2018

    2018 arXiv

  2. [10]

    BERT: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  3. [11]

    A. Gersho. Quantization. IEEE Communications Society Magazine, 15(5):16–16, Sep. 1977

    1977

  4. [12]

    node2vec: Scalable feature learning for networks

    Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In KDD, 2016

    2016

  5. [13]

    Deep learning with limited numerical precision

    Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In ICML, 2015

    2015

  6. [14]

    Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and Huffman coding. In ICLR, 2016. 10

    2016

  7. [15]

    Analysis of a complex of statistical variables into principal components

    Harold Hotelling. Analysis of a complex of statistical variables into principal components. Journal of educational psychology, 24(6):417, 1933

    1933

  8. [16]

    Oseledets

    Valentin Khrulkov, Oleksii Hrinchuk, Leyla Mirvakhabova, and Ivan V . Oseledets. Tensorized embedding layers for efficient model compression. arXiv preprint arXiv:1901.10787, 2019

    1901 arXiv

  9. [17]

    J. Kiefer. Sequential minimax search for a maximum. Proceedings of the American Mathemati- cal Society, 4:502–506, 1953

    1953

  10. [18]

    Convolutional neural networks for sentence classification

    Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP, 2014

    2014

  11. [19]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  12. [20]

    Neural word embedding as implicit matrix factorization

    Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. In NeurIPS, 2014

    2014

  13. [21]

    Mixed precision training

    Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Frederick Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In ICLR, 2018

    2018

  14. [22]

    Efficient estimation of word representations in vector space

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013

    2013 arXiv

  15. [23]

    Advances in pre-training distributed word representations

    Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. Advances in pre-training distributed word representations. In LREC, 2018

    2018

  16. [24]

    Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization

    Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In ICML, 2019

    2019

  17. [25]

    fairseq: A fast, extensible toolkit for sequence modeling

    Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In NAACL-HLT: Demonstrations, 2019

    2019

  18. [26]

    Automatic differentiation in PyTorch

    Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NeurIPS Autodiff Workshop, 2017

    2017

  19. [27]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learni...

    2011

  20. [28]

    Jeffrey Pennington, Richard Socher, and Christopher D. Manning. GloVe: Global vectors for word representation. In EMNLP, 2014

    2014

  21. [29]

    Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer

    Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL-HLT, 2018

    2018

  22. [30]

    Sur les équations algébriques ayant toutes leurs racines réelles.Mathematica, 9:129–145, 1935

    Tiberiu Popoviciu. Sur les équations algébriques ayant toutes leurs racines réelles.Mathematica, 9:129–145, 1935

    1935

  23. [31]

    Random features for large-scale kernel machines

    Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In NeurIPS, 2007

    2007

  24. [32]

    SQuAD: 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP, 2016

    2016

  25. [33]

    Compressing word embeddings via deep compositional code learning

    Raphael Shu and Hideki Nakayama. Compressing word embeddings via deep compositional code learning. In ICLR, 2018

    2018

  26. [34]

    Manning, Andrew Y

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y . Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013

    2013

  27. [35]

    Low-memory neural network training: A technical report

    Nimit Sharad Sohoni, Christopher Richard Aberger, Megan Leszczynski, Jian Zhang, and Christopher Ré. Low-memory neural network training: A technical report. arXiv preprint arXiv:1904.10631, 2019

    1904 arXiv

  28. [36]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 11

    2017

  29. [37]

    GLUE: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019

    2019

  30. [38]

    Christopher K. I. Williams and Matthias W. Seeger. Using the Nyström method to speed up kernel machines. In NeurIPS, 2000

    2000

  31. [39]

    Google’s neural machine translation system: Bridging the gap between human and machine translation

    Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016

    2016 arXiv

  32. [40]

    On the dimensionality of word embedding

    Zi Yin and Yuanyuan Shen. On the dimensionality of word embedding. In NeurIPS, 2018

    2018

  33. [41]

    Low-precision random Fourier features for memory-constrained kernel approximation

    Jian Zhang, Avner May, Tri Dao, and Christopher Ré. Low-precision random Fourier features for memory-constrained kernel approximation. In AISTATS, 2019. A Background A.1 Uniform Quantization Ab-bit uniform quantization of a real numberx∈ [−r,r ] is computed as follows: First, ...

    2019

  34. [43]

    In Table 3, we show the impact of the above perturbation on the various measures of compression quality we have discussed

    If we assume thatz2 1≪∑d i=2z2 i (as would be expected in our average-case analysis), then ˜X would perform similarly toX. In Table 3, we show the impact of the above perturbation on the various measures of compression quality we have discussed. At a high-level, we observe tha...

  35. [44]

    show generalization bounds scale with 1 1−∆1 ). This makes sense, because ∆1 can be used to attain a worst-case generalization bound for the perturbed embeddings, and there exist cases where setting the largest singular value to 0 can significantly harm the generalization perfo...

  36. [45]

    GloVe|ρ|| fastText|ρ|

    Note that this second term is simply a version of the eigenspace overlap score 1 max(d,k)‖ ˜UTU‖2 F = 1 max(d,k) ∑d i=1‖ ˜UTUi‖2 2 which weights the projections of the different singular vectorsUi ofX onto ˜U according to the singular values ofX. In Section B.1 we show that in...

  37. [46]

    Case 1:i⁄=j E [ H2 ij ] = E [ (xT i cj +cT i xj +cT i cj)2] = E [ (xT i cj)2 + (cT i xj)2 + (cT i cj)2] = E [( d∑ k=1 xikcjk )2 ] + E [( d∑ k=1 cikxjk )2 ] + E [( d∑ k=1 cikcjk )2 ] = E [ d∑ k=1 x2 ikc2 jk ] + E [ d∑ k=1 c2 ikx2 jk ] + E [ d∑ k=1 c2 ikc2 jk ] = d∑ k=1 x2 ikE [...

  38. [47]

    uncompressed

    Case 2:i =j E [ H2 ii ] = E [ (xT i ci +cT i xi +cT i ci)2] = E [( 2 d∑ k=1 xikcik + d∑ l=1 c2 il )2 ] = E [ 4 ( d∑ k=1 xikcik )2 + 4 ( d∑ k=1 xikcik ) · ( d∑ l=1 c2 il ) + ( d∑ l=1 c2 il )2 ] = E  4 d∑ k=1 x2 ikc2 ik + 4 d∑ k,l=1 xikcikc2 il + d∑ k,l=1 c2 ilc2 ik   = 4 d∑...

    2014

  39. [2018]

    [Online; published 13-Sept-2018; accessed 20-May- 2019]

    URL https://blog.twitter.com/engineering/en_us/topics/insights/ 2018/embeddingsattwitter.html. [Online; published 13-Sept-2018; accessed 20-May- 2019]

    2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.