{"id":"2ae7fe36-ab81-43fd-86c2-a1d1635a7d47","arxiv_id":"1909.01264","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"The eigenspace overlap score, a subspace-preservation measure between compressed and uncompressed embeddings, predicts downstream performance and selects better compressed embeddings more reliably than prior quality measures.","lead":"Compressed word embeddings get a new quality score, the eigenspace overlap score, which measures how much of the original embedding's principal directions a compressed version keeps. The paper proves this score predicts linear-model performance and shows it can pick better compressed embeddings without training a model for every task.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's identity-covariance label model is the load-bearing assumption; the unweighted eigenspace overlap score can misrank embeddings when real task labels are concentrated in particular singular directions.","rationale":"The reader's weakest assumption matches the most load-bearing point. The theoretical bridge from the eigenspace overlap score to downstream performance is Theorem 1, and that bridge is exact only when the true label vector is isotropic in span(U). The paper explicitly frames this as an average-case choice, and the generalized theorem in Appendix B.1 shows that the correct quantity for non-isotropic covariance is the covariance-weighted overlap ||Ũ^T U Σ^{1/2}||_F^2; the unweighted E is not sufficient. Since the empirical tasks are classification and QA tasks whose labels are certainly not drawn from an identity-covariance distribution in U, the correlation and selection results in Tables 1-2 are doing the empirical work. The paper's own Appendix B.3 example, deleting the largest singular value, illustrates that E can be nearly unchanged while downstream performance changes a lot when a label direction is lost; this is precisely the failure mode the average-case model assumes away. I do not think this warrants rejection: the score is cheap, the experiments are extensive, and the paper is transparent about the assumption. But the central theoretical claim should be labeled conditional pending a check of label covariance on real tasks. The reader already reached CONDITIONAL, so the verdict is unchanged. The proposed covariance-weighted re-analysis would settle whether the concern lands.","tokens_in":39391,"tokens_out":14131,"duration_ms":145963,"concrete_test":"Run a covariance-weighted variant of the Table 2 selection test: for each task, train a linear (or logistic) probe on the uncompressed embedding X, take its predicted logits ŷ as the label proxy, set z = U^T ŷ, and estimate Σ̂ = (1/m) Σ z z^T over held-out points. For each pair of compressed embeddings, compute the exact Appendix B.1 quantity Δ = [tr(Σ̂) - ||Ũ^T U Σ̂^{1/2}||_F^2]/n - (d-k)σ̂²/n, and rank by Δ instead of by E. If the resulting selection error rates or Spearman correlations with actual downstream performance differ materially from Tables 1-2, the identity-covariance assumption is load-bearing and the unweighted eigenspace overlap score is not the right control for real tasks.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that E(X, X̃) controls downstream performance is established only under the average-case model ȳ = Uz with z having identity covariance (Section 3.1.2). The paper's own generalized theorem (Appendix B.1) shows the expected risk difference is governed by ||Ũ^T U Σ^{1/2}||_F^2, and E enters only as a lower bound via λ_min(Σ)E when Σ is not isotropic. Thus, two compressed embeddings with the same E can have very different expected downstream performance if one preserves the singular directions where real labels are concentrated and the other does not. This is not a worst-case technicality: word embeddings have highly skewed spectra, and the paper's own robustness example (Appendix B.3) concedes that deleting the largest singular direction changes E by only 1/d while it can dominate generalization when the label has large component in that direction. The empirical correlations in Tables 1-2 are aggregate over many compression methods and rates, and they do not test whether the label covariance on SQuAD, SST-1, MNLI, or QQP is sufficiently isotropic for the identity-covariance result to apply. Without such a check, the theoretical explanation of downstream performance is conditional on an unverified generative model for labels.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the eigenspace overlap score E(X, X̃) as a measure of compressed word embedding quality, defined as the normalized squared Frobenius inner product of the left singular subspaces of the original and compressed embedding Gram matrices. The authors derive average-case generalization bounds for fixed-design linear and logistic regression in which the expected risk difference between models trained on compressed and uncompressed embeddings is controlled by E. They also state a lower bound on E for uniformly quantized embeddings via the Davis-Kahan sin(Theta) theorem, and they validate the score empirically as a correlation measure and as a selection criterion on question answering, sentiment analysis, and GLUE tasks. The empirical study spans GloVe, fastText, and BERT WordPiece embeddings and multiple compression methods, with error bars across five random seeds.","tokens_in":39649,"tokens_out":6384,"duration_ms":65286,"significance":"If the claims hold, the eigenspace overlap score is a practically valuable, training-free measure for selecting compressed embeddings, and the paper provides a clean average-case analysis that goes beyond worst-case spectral approximation bounds. The strengths include an exact identity for the linear regression risk difference under an isotropic label model (Theorem 1), a Lipschitz-loss extension (Theorem 2), a broad and reproducible empirical evaluation with standard deviations and hyperparameter tables, and a public implementation of the quantization method. However, the proof of Theorem 3 contains a load-bearing technical flaw in the Davis-Kahan application, and the theoretical connection to downstream performance is established only under an identity-covariance label model that is not verified on the real tasks. These issues limit the current theoretical guarantees, although the empirical results are substantial and the Davis-Kahan argument appears repairable.","major_comments":[{"comment":"The Davis-Kahan sin(Theta) application in the proof of Theorem 3 is not valid as stated. The authors set a0 = sigma_min(K), a1 = infinity, and delta = sigma_min(K), so the excluded interval in Theorem 5 is (0, infinity). Since K̃ = X̃X̃ᵀ has rank at most k < n, the block R1 of the perturbed Gram matrix contains zero eigenvalues, and 0 is not excluded from the interval (0, infinity). Thus the condition that the eigenvalues of R1 be excluded from (a0 - delta, a1 + delta) is violated, and Theorem 5 cannot be applied. This is load-bearing because Theorem 3 is the main theoretical explanation for the strong performance of uniform quantization. The argument can likely be repaired by choosing a smaller delta, e.g., delta = sigma_min(K)/2, which makes the excluded interval (sigma_min(K)/2, infinity) and excludes the zero eigenvalues, at the cost of a larger constant in the bound; the authors should correct the proof or the statement accordingly.","section":"Appendix C.1, Theorem 3 and Theorem 5"},{"comment":"The simplified form of the theoretical claim, in which the eigenspace overlap score E(X, X̃) controls the expected risk difference, depends critically on the label vector model ȳ = Uz with z having identity covariance. The generalized theorem in Appendix B.1 shows that the expected risk difference is governed by ||ŨᵀU Σ^{1/2}||_F², and E enters only through the lower bound lambda_min(Σ) d E. For non-isotropic label covariance, two compressed embeddings with the same E can have very different expected downstream performance, and the paper's own robustness example in Appendix B.3 concedes that deleting the largest singular direction changes E by only 1/d yet can dominate generalization when the label has a large component in that direction. The empirical correlations in Tables 1 and 2 are aggregated over many compression methods and rates and do not test whether the label covariance on SQuAD, SST-1, MNLI, or QQP is sufficiently isotropic for the identity-covariance result to apply. Please either add an empirical analysis of label alignment with the singular directions (or an evaluation of the weighted score ||ŨᵀU Σ^{1/2}||_F) or substantially qualify the theoretical claim that E alone controls downstream performance.","section":"Section 3.1.2 and Appendix B.1"}],"minor_comments":[{"comment":"The displayed formulas for the stochastic rounding probabilities are garbled: both P[Q̃_b,r(x) = x] and P[Q̃_b,r(x) = x̄] appear to be written as (x - x̄)/(x - x̄). Please correct the typesetting so that the two probabilities are clearly distinguished.","section":"Appendix A.1"},{"comment":"The text contains the typo 'language infernece'; it should read 'language inference'.","section":"Appendix E.8, Table 17"},{"comment":"The notation 'lambda_min in R' is unclear; it should be 'lambda_min(Sigma)', the smallest eigenvalue of the label covariance matrix, matching the statement of Theorem 1 (Generalized).","section":"Appendix B.2, Theorem 2 (Generalized)"},{"comment":"The caption for Figure 4(b) should state explicitly how the scalar a is computed for the synthetic matrices; the current description says the matrices are multiplied by diagonal matrices with logarithmically spaced entries, but it does not define a in terms of the smallest singular value.","section":"Appendix C.2, Figure 4(b)"},{"comment":"In the table caption, 'GloVe|ρ| / fastText|ρ|' is ambiguous because the entries are presented as four numbers without a visible separator; please format the entries as 'GloVe |ρ| / fastText |ρ|' or use two columns.","section":"Appendix B.4, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The empirical contribution is solid and the paper is within scope; I would not reject it. The main technical issue is the invalid Davis-Kahan parameter choice in Theorem 3, which is likely repairable by choosing a smaller delta. The label-covariance caveat is explicitly acknowledged in the paper, but it is central enough to the theoretical narrative that it should be addressed with either an empirical check or a clearer statement of the scope of the theoretical claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this one. First, the empirical core is solid: the eigenspace overlap score E(X, X̃) = (1/max(d,k))||U^T Ũ||_F^2 correlates with downstream performance more strongly than PIP loss, Δ, and Δmax across GloVe, fastText, and BERT on SQuAD, sentiment, and GLUE, and it selects better compressed embeddings with substantially lower error rates. Second, the theory is real but narrower than the abstract suggests: the clean result (Theorem 1) holds only when the true label vector is Uz with z having identity covariance, and Theorem 3's Davis-Kahan application has a gap.\n\nWhat's genuinely new: the score itself, the average-case bounds connecting it to risk for linear and Lipschitz losses, and the selection-criterion experiments. The empirical study is careful — five seeds, error bars, multiple tasks and embedding types, and uniform quantization is shown to match or beat learned methods. That last result is a useful data point on its own.\n\nSoft spots, in order of importance. The theoretical guarantee for uniform quantization (Theorem 3) as written does not follow from the cited Davis-Kahan theorem. They set a0 = σ_min(K), a1 = ∞, δ = σ_min(K), which excludes the interval (0,∞); the eigenvalues of the quantized Gram matrix in the 'rest' subspace are positive, so they are not excluded. The bound might be salvageable with a different gap argument, but as written it's a proof gap. Also, the main theory assumes y = Uz with z isotropic. The generalized theorem in Appendix B.1 shows the risk difference depends on ||Ũ^T U Σ^{1/2}||_F^2, and E only enters as a lower bound via λ_min(Σ). Real task labels are likely not isotropic in the embedding subspace; the paper never checks that on SQuAD or GLUE. So the theoretical explanation of downstream performance is conditional on an unverified generative model. The empirical correlations are still useful, but they don't validate the theory's core assumption.\n\nThat said, the paper is honest about the average-case assumption and the limitation of worst-case bounds, and the empirical claims stand independently. The citation pattern is fine, and the comparison to existing measures is fair.\n\nWho should read it: anyone working on embedding compression, quantization, or feature approximation. It deserves a serious referee — if it were submitted today, I'd send it out rather than desk reject, with a request to fix or reframe Theorem 3 and to temper the theory claims. I'd probably cite it if I were writing about compression quality measures.","headline":"A genuinely useful new score for ranking compressed embeddings, backed by a broad empirical study; the theory is narrower than claimed and Theorem 3 has a proof gap, but the paper is worth refereeing.","tokens_in":40156,"tokens_out":2917,"would_cite":true,"duration_ms":29655,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T05","62J05","68P30"],"pacs":[],"model":"deepseek-v4-flash","headline":"One subspace score predicts which compressed word embeddings work","keywords":["eigenspace overlap score","word embedding compression","generalization bounds","uniform quantization","model selection","linear regression","subspace overlap","downstream NLP performance"],"falsifier":"Take a fixed task and two compressed embeddings $\\tilde X_A,\\tilde X_B$ with $E(X,\\tilde X_A)>E(X,\\tilde X_B)$; if $\\tilde X_A$ consistently performs worse on labels generated from directions orthogonal to its span, the selection claim fails. Concretely, synthesize labels $\\bar y = U_\\perp z$ (using left singular directions outside $\\operatorname{span}(\\tilde U)$) and check whether the exact gap in Theorem 1 still holds; it should not, because the theorem's premise is violated.","tokens_in":39205,"feed_emoji":"📐","tokens_out":6257,"duration_ms":56601,"temperature":0.7,"pith_summary":"The paper claims that the downstream quality of a compressed word embedding is governed by one number: how much of the uncompressed embedding's left singular subspace survives in the compressed embedding. The authors call this the eigenspace overlap score, prove that for linear and logistic regression the expected generalization gap between compressed and uncompressed embeddings shrinks as this score grows, and show that uniform quantization achieves high scores with few bits. If the claim is right, a practitioner can choose between compressed embeddings by computing an SVD-based score instead of training a model for each task. The same connection explains why a simple uniform quantization method is competitive with more elaborate learned compression schemes.","feed_headline":"One subspace score predicts which compressed word embeddings work","feed_subtitle":"A single overlap number ranks compressed embeddings without training a model, and explains why uniform quantization is competitive.","key_machinery":"The load-bearing object is the eigenspace overlap score $E(X,\\tilde X)$, the squared Frobenius norm of the projection of the uncompressed left-singular basis $U$ onto the compressed basis $\\tilde U$, normalized by $\\max(d,k)$. It matters because linear predictions $Xw$ depend on the data only through $U$, so two embeddings with overlapping left singular subspaces can emulate each other's linear models. Theorem 1 turns that geometric intuition into an exact risk gap, while the Davis-Kahan $\\sin(\\Theta)$ theorem connects the perturbation caused by quantization to a lower bound on the score, and the score itself is computable by SVD in $O(n\\max(d,k)^2)$ time.","core_discovery":"The central claim is that the eigenspace overlap score $E(X,\\tilde X) = \\frac{1}{\\max(d,k)}\\|U^T\\tilde U\\|_F^2$ is the quantity that controls the expected downstream loss of a compressed embedding. For fixed-design linear regression with true labels drawn randomly from the span of the uncompressed embedding's left singular vectors, Theorem 1 gives an exact average-case gap $E_{\\bar y}[R_{\\bar y}(\\tilde X)-R_{\\bar y}(X)] = \\frac{d}{n}(1-E(X,\\tilde X)) - c^2\\frac{d(d-k)}{n^2}$; for Lipschitz losses the same score appears in a matching upper bound. The paper also proves that a uniform quantizer with $b$ bits per entry attains expected eigenspace overlap at least $1-\\epsilon$ once $b = O(\\log(1/\\epsilon))$, and that the score is a more accurate and robust selection criterion than existing compression-quality measures, with up to $2\\times$ lower selection error rates on the tasks studied.","pith_inferences":["A natural testable extension is to use the gradient of the eigenspace overlap score as a training objective for learned compression, since the exact Theorem 1 expression suggests this directly minimizes expected downstream loss under the average-case label model; the paper does not take this step.","Because the proof relies only on the SVD of the feature matrix, the same score plausibly transfers to other linear-feature settings such as graph node embeddings or kernel approximations, even though the paper only demonstrates word embeddings.","The strong empirical correlation on neural models hints that the left singular subspace of input features is the main channel for downstream signal in these architectures; verifying this on a wider set of nonlinear models would be a separate claim."],"forward_implications":["Compressed embeddings can be ranked without training a downstream model, so model selection over compression methods becomes a cheap linear-algebra computation.","Uniform quantization has a formal guarantee: at a logarithmic number of bits it keeps the expected eigenspace overlap close to 1, explaining why it matches or beats learned compression methods.","In the low-noise, low-dimensional regime the risk gap is approximately $\\frac{d}{n}(1-E)$, so maximizing eigenspace overlap is approximately minimizing the expected loss gap.","The score carries the same information as a weighted variant of projected reconstruction error, so practitioners can also use a weighted reconstruction loss as a proxy."],"supporting_citations":[{"why":"Supplies the Davis-Kahan sin(Theta) theorem used to bound the eigenspace overlap score of uniformly quantized embeddings.","marker":"[8]"},{"why":"Provides the worst-case spectral-approximation framework and the Delta and Delta_max baselines that the eigenspace overlap score is compared against.","marker":"[41]"},{"why":"Defines the PIP loss baseline used as an existing measure of compression quality.","marker":"[40]"},{"why":"Defines the unbiased stochastic rounding used in the uniform quantization analysis of Theorem 3.","marker":"[13]"},{"why":"Supplies the DCCL compression baseline whose downstream performance is compared to uniform quantization.","marker":"[33]"},{"why":"Supplies the k-means compression baseline used in the empirical comparisons.","marker":"[2]"},{"why":"Provides the GloVe embeddings used in the downstream experiments.","marker":"[28]"},{"why":"Provides the DrQA model and SQuAD task used as a downstream evaluation.","marker":"[5]"},{"why":"Provides the GLUE benchmark tasks used for evaluating compressed BERT WordPiece embeddings.","marker":"[37]"}],"fun_headline_variants":["Eigenspace overlap score predicts compressed embedding performance","One score, no training: pick better compressed word embeddings","A single overlap number eliminates per-task model training","Eigenspace overlap explains uniform quantization's edge","Pick better embeddings: eigenspace overlap score cuts errors in half"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The average-case analysis assumes the true label vector is random in the span of the uncompressed embedding's left singular subspace; if a real task's labels concentrate in directions that compression discards, the score can misrank embeddings and the guarantees do not apply.","fun_headline_variants_meta":{"raw":{"variants":["Eigenspace overlap score predicts compressed embedding performance","One score, no training: pick better compressed word embeddings","A single overlap number eliminates per-task model training","Eigenspace overlap explains uniform quantization's edge","Pick better embeddings: eigenspace overlap score cuts errors in half"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00042,"raw_usage":{"total_tokens":2153,"prompt_tokens":933,"completion_tokens":1220,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":1143}},"tokens_in":549,"tokens_out":1220,"duration_ms":9006,"temperature":1.0,"reasoning_tokens":1143,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:23:29.270252+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed task and two compressed embeddings $\\tilde X_A,\\tilde X_B$ with $E(X,\\tilde X_A)>E(X,\\tilde X_B)$; if $\\tilde X_A$ consistently performs worse on labels generated from directions orthogonal to its span, the selection claim fails. Concretely, synthesize labels $\\bar y = U_\\perp z$ (using left singular directions outside $\\operatorname{span}(\\tilde U)$) and check whether the exact gap in Theorem 1 still holds; it should not, because the theorem's premise is violated.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the GloVe embeddings used in the downstream experiments."},{"cited_title":"Reading Wikipedia to answer open-domain questions","cited_arxiv_id":null,"evidence_quote":"Provides the DrQA model and SQuAD task used as a downstream evaluation."},{"cited_title":"GLUE: A multi-task benchmark and analysis platform for natural language understanding","cited_arxiv_id":null,"evidence_quote":"Provides the GLUE benchmark tasks used for evaluating compressed BERT WordPiece embeddings."},{"cited_title":"Low-precision random Fourier features for memory-constrained kernel approximation","cited_arxiv_id":null,"evidence_quote":"Provides the worst-case spectral-approximation framework and the Delta and Delta_max baselines that the eigenspace overlap score is compared against."},{"cited_title":"On the dimensionality of word embedding","cited_arxiv_id":null,"evidence_quote":"Defines the PIP loss baseline used as an existing measure of compression quality."},{"cited_title":"Compressing word embeddings via deep compositional code learning","cited_arxiv_id":null,"evidence_quote":"Supplies the DCCL compression baseline whose downstream performance is compared to uniform quantization."},{"cited_title":"Compressing word embeddings","cited_arxiv_id":null,"evidence_quote":"Supplies the k-means compression baseline used in the empirical comparisons."}],"review_version":1}