{"id":"bd49d6af-96fd-432e-aa2d-5f074439f2a1","arxiv_id":"2501.16650","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper defines a weight-matrix similarity index based on maximum absolute cosine values and Gumbel fitting, then uses it to show that neighboring transformer layers in open LLMs have similar weights and form clusters.","lead":"This paper introduces DOCS, an index that measures how similar the weight matrices of two transformer layers are by looking at how well each direction in one matrix aligns with a direction in the other. The index reveals that neighboring layers in large open-source language models often have similar weights and cluster together, which could inform compression, fine-tuning, and distillation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The proof of Theorem 1 computes SDOCS from constant max-cosine vectors, but the Gumbel MLE for constant data is degenerate, so Algorithm 1 does not produce the stated value.","rationale":"The central claim has two parts: a mathematical guarantee that DOCS is discriminative on orthogonal matrices, and an empirical demonstration of structural patterns. The mathematical guarantee is the novel differentiator claimed over CCA, SVCCA, and linear CKA, so I looked for the point where that guarantee is least secure. That point is the step in Appendix C where all maximum absolute cosine similarities are equal to 1/sqrt(m). A Gumbel maximum-likelihood fit to constant data is degenerate, so the exact SDOCS value asserted in Theorem 1 is not actually produced by Algorithm 1. This is an internal consistency problem in the proof rather than a disagreement with external consensus. The same degeneracy appears in the reflexivity lemma, where the argument is easier to repair by taking the obvious limit, but the theorem's constructive claim is not so easily patched without changing the algorithm. The reader's weakest assumption identified this same issue, and I agree with that identification. The empirical claims about clusters and expert specialization are also limited by the absence of null distributions and significance tests, but those claims are secondary because the index definition itself is shared by all experiments and Appendix E.3 provides a relative sanity check against randomly initialized weights. The proposed concrete test directly checks whether Algorithm 1 returns the advertised value on the constructed matrices. If it does, the concern is resolved. If it does not, Theorem 1 needs revision, though the paper may still retain the weaker discriminative claim via Appendix A.5. The reader's CONDITIONAL verdict remains appropriate: the flaw is real and load-bearing, but it is addressable and does not force rejection of the entire framework.","tokens_in":20924,"tokens_out":8503,"duration_ms":91736,"concrete_test":"Implement Algorithm 1 exactly as written on the Appendix C construction for n = 8, 16, and 32, using a standard Gumbel MLE fitter such as scipy.stats.gumbel_r.fit; record whether it returns a finite scale and whether (uX + uY)/2 equals 1/sqrt(m). Separately recompute the Appendix A.5 orthogonal pairs with the same routine; if the values 0.88 and 0.76 reproduce, the weaker discriminative claim survives even though Theorem 1 needs a corrected proof or a well-defined estimator for degenerate input.","verdict_should_be":"UNCHANGED","load_bearing_attack":"For the matrices constructed in Appendix C (X = [e1,...,em], Y = (1/sqrt(m))[H_m; 0]), every absolute cosine |cos(theta_jk)| equals 1/sqrt(m), so both MAXCOSSIM vectors sX and sY are constant vectors with all entries c = 1/sqrt(m). Algorithm 1 then specifies fitting a Gumbel distribution by maximum likelihood and taking the location parameter. For m identical observations c, the Gumbel log-likelihood has no finite maximizer: setting the location parameter to c, the derivative with respect to the positive scale parameter is -m/beta, which is never zero for finite beta, and the likelihood is maximized only in the boundary limit beta -> 0. Standard Gumbel MLE routines therefore fail to converge or return an arbitrary scale, and the claimed identity SDOCS(X,Y) = (uX + uY)/2 = 1/sqrt(m) is not a consequence of Algorithm 1 as written. Because Theorem 1 is the main theoretical evidence for the headline advantage that DOCS is effective on orthogonal matrices, this is a load-bearing gap in the proof. The weaker Definition 3 discriminative claim is separately supported by the explicit orthogonal pairs in Appendix A.5, so the issue is repairable rather than fatal; however, as stated the theorem is not established. The empirical cluster and expert-specialization claims also lack null baselines, but the Gumbel degeneracy is the more precise and fundamental defect.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DOCS, a weight-matrix similarity index that computes per-column maximum absolute cosine similarities between two matrices, fits a Gumbel distribution to each of the two resulting vectors, and averages the fitted location parameters. The authors claim that DOCS is discriminative on orthogonal matrices, where CCA, SVCCA, and linear CKA are constant or dimension-dependent, and they present a theorem (Theorem 1) intended to show this discriminative power. Empirically, the paper applies DOCS to weight matrices of several open-source LLMs and reports that adjacent layers are highly similar, that clusters of similar layers exist, that base and instruction-tuned models share high DOCS values, and that Mixture-of-Experts models often contain a distinctive expert. The paper also proves several invariance properties (permutation, symmetry, isotropic scaling, reflexivity) and provides a comparison of DOCS with other indices on a Llama model.","tokens_in":21191,"tokens_out":7646,"duration_ms":74485,"significance":"If the DOCS index is valid, it offers a simple and computationally lightweight tool for analyzing weight-space structure in LLMs, potentially informing layer-level pruning, parameter-efficient fine-tuning, and knowledge distillation. The paper is commendable for giving explicit definitions, a clear algorithm, and concrete numerical examples, and for identifying a real limitation of existing representation-similarity indices on orthogonal matrices. The discriminative example in Appendix A.5 supports the weaker but meaningful claim that DOCS can distinguish some orthogonal pairs. However, the central theoretical result, Theorem 1, rests on a degenerate maximum-likelihood fit, and the main empirical findings are supported only by visual inspection without statistical baselines. The contribution is therefore promising but not yet rigorously established; the issues are local and repairable.","major_comments":[{"comment":"The proof of Theorem 1 constructs matrices X and Y for which every absolute cosine similarity equals 1/sqrt(m), so the vectors sX and sY are constant. Algorithm 1 then specifies fitting a Gumbel distribution by maximum likelihood. For a constant vector, the Gumbel log-likelihood has no finite maximizer: the likelihood increases without bound as the scale parameter tends to 0, and the location parameter is unidentified. Consequently, Algorithm 1 as written does not produce the stated value SDOCS(X,Y) = 1/sqrt(m), and the theorem is not established by the given proof. The same degeneracy appears in the reflexivity proof in Appendix A.4, where sX is a vector of all ones and the claim that the Gumbel MLE yields uX = 1 is not valid. The weaker Definition 3 claim is separately supported by the explicit orthogonal pair in Appendix A.5, so the issue is repairable; for example, the index could be redefined using a well-defined functional of the max-cosine vectors (e.g., the empirical mean or median), or the algorithm could specify a regularized or limiting Gumbel estimator whose value on constant data is explicitly derived and proven.","section":"Section 3.1, Theorem 1, Appendix C, and Algorithm 1"},{"comment":"The central empirical claims that adjacent layers exhibit similar weights, that clusters of similar layers exist, and that some MoE experts are uniquely specialized are supported only by visual inspection of heatmaps and scatter plots. The Gini coefficient in Table 2 is a descriptive statistic without confidence intervals or a null baseline. Without a null model (e.g., random orthogonal matrices matched for dimensionality, column-permuted weights, or layer-shuffled matrices), the observed diagonal and block patterns could arise from the index's intrinsic dependence on column norms, dimensionality, or shared input statistics rather than from functional specialization. Please provide quantitative validation, such as permutation tests, bootstrap confidence intervals, or cluster validity indices, and report effect sizes.","section":"Sections 4.2-4.5 and Table 2"},{"comment":"The choice of the Gumbel distribution is justified solely by histograms of the max-cosine vectors from one pair of layers (layers 4 and 8) of a single model, and no goodness-of-fit test or theoretical extreme-value argument is given. This is a post-hoc model selection on the same data that the method then fits, and it does not address the degenerate constant-data case that arises in Theorem 1 and Appendix A.4. Since the interpretation of the DOCS value depends on the location parameter of a well-specified Gumbel model, please either supply a theoretical justification (e.g., from extreme-value theory under explicit independence assumptions) or demonstrate robustness to distributional misspecification by comparing DOCS with nonparametric summaries of the max-cosine vectors.","section":"Appendix G"}],"minor_comments":[{"comment":"The transposition convention is stated for Wv, Wk, Wq, and MLP-UP, and for W1 and W3 in the MoE experiment, but not for Wo; since Wo heatmaps appear in Appendix I, please clarify whether Wo is also transposed and why.","section":"Section 4, first paragraph"},{"comment":"The proof states that Cjk = Ckj, but the cosine similarity matrix is not symmetric in general; Cjk uses X_j and Y_k, while C_kj uses X_k and Y_j. The symmetry of SDOCS actually holds because the index averages the fits from both directions, so the proof should be revised to derive symmetry from that averaging rather than from a false symmetry of C.","section":"Appendix A.2, proof of Lemma 2"},{"comment":"The statement 'Fitting a Gumbel distribution to sX consisting of all ones yields a location parameter uX = 1' is not justified; as noted in the major comments, the Gumbel MLE is degenerate for constant data. A separate argument is needed for the reflexivity property if the algorithm is changed.","section":"Appendix A.4, proof of Lemma 4"},{"comment":"The notation 'm = Ω( n)' appears with a stray space; it should read 'm = Omega(n)'. The proof uses m = 2^{floor(log2 n)}, which indeed gives m = Omega(n), but the construction should be stated more cleanly and the constant in the Omega notation should be explicit.","section":"Theorem 1 statement"},{"comment":"The discriminative example reports numerical values SDOCS(X,Y) = 0.88 and SDOCS(X',Y') = 0.76, but the intermediate max-cosine vectors and Gumbel fits are not shown. Providing these intermediate computations, or the code used to generate them, would improve reproducibility and allow readers to verify the claim.","section":"Appendix A.5"}],"recommendation":"major_revision","confidential_remarks":"The paper is already published as an ICLR 2025 conference paper, so a journal submission would need to offer substantial additional validation. The core idea is interesting, and the explicit discriminative example is a useful sanity check, but the Gumbel degeneracy in the theorem's proof and the lack of null baselines for the empirical claims are genuine obstacles. If the authors address the major comments with a redefined, well-posed estimator and add statistical tests, the manuscript could become a solid journal contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: DOCS is a simple and sensible weight-similarity index, and the paper has an interesting empirical story about adjacent-layer similarity and cluster structure across several modern LLMs. But the headline theoretical result, Theorem 1, is not proven as stated: the constructed pair gives constant max-cosine vectors, and the Gumbel MLE for constant data is degenerate, so Algorithm 1 does not return the claimed value.\n\nWhat is genuinely new and useful: the index itself—per-column maximum absolute cosine similarity summarized by a Gumbel location parameter—is cheap, easy to reproduce, and applied at scale to Llama, Gemma, Yi, and Mixtral. The empirical finding that adjacent layers share high weight similarity is consistent across many matrices and model families, and the sanity check against a randomly reinitialized model (Appendix F) is a good control. The observation that CCA, SVCCA, and linear CKA are constant or dimension-dependent on orthogonal matrices is correct and worth knowing. The paper also does a decent job of situating itself in the representation-similarity literature.\n\nThe soft spots are real. The Gumbel MLE degeneracy is load-bearing: for the X and Y constructed in Appendix C, every absolute cosine is 1/sqrt(m), so both sX and sY are constant vectors. For constant data, the Gumbel likelihood has no finite maximizer—the location is pinned to the constant value but the scale is driven to zero. Thus the stated SDOCS value is not produced by Algorithm 1. The same problem appears in the reflexivity proof (A.4), where all ones are fed into the same degenerate fit. The weaker discriminative claim (Definition 3) does hold via the explicit orthogonal pairs in A.5, so the overall discriminative idea is repairable, but the theorem as written is not established. On the empirical side, cluster and expert-specialization claims rely on visual inspection and lack null baselines, significance tests, and error bars on the index itself. The interpretation that high DOCS reflects functional specialization is plausible but not directly tested.\n\nThis paper is for researchers who want a quick weight-space diagnostic for model compression, SFT, or MoE routing. It deserves to be read, but with the theorem treated as open. If I were handling peer review, I would send it out—the method is cheap, the experiments are broad, and the flaw is fixable—but I would ask the authors to correct the theoretical claim, add a robust location estimator or modify the construction, and provide at least a permutation-based null for the cluster claims.","headline":"A cheap and reasonable weight-similarity index with a broad empirical survey, but the central theorem is unproven because the Gumbel MLE degenerates on the constructed example.","tokens_in":21734,"tokens_out":2998,"would_cite":false,"duration_ms":31577,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces the DOCS index for comparing transformer weight matrices, proves it can tell orthogonal matrices apart where CKA, CCA, and SVCCA cannot, and uses it to show that neighboring layers of large language models tend to…","keywords":["weight matrix similarity","large language models","cosine similarity","orthogonal matrices","Gumbel distribution","layer clustering","transformer interpretability","mixture of experts"],"falsifier":"Run the algorithm described in the paper on the X and Y constructed in Theorem 1; every per-column maximum absolute cosine similarity is 1/sqrt(m), a constant vector, and maximum-likelihood fitting of a Gumbel distribution to a constant sample is degenerate, so Algorithm 1 will not output the claimed 1/sqrt(m) without an added tie-breaking or smoothing step. A reproduction that yields the claimed value would therefore have to use a different estimation procedure than the one defined.","tokens_in":20684,"feed_emoji":"🧠","tokens_out":8209,"duration_ms":76802,"temperature":0.7,"pith_summary":"The paper introduces DOCS (Distribution of Cosine Similarity), an index for comparing the weight matrices of transformer layers by looking at how well each neuron-weight column aligns with some column in the other matrix. Its central claim is that DOCS is genuinely discriminative where standard indices are blind: for orthogonal weight matrices, CCA, SVCCA, and linear CKA all return constant scores, while DOCS provably returns different values for genuinely different matrices. The paper further claims that applying DOCS to open LLMs reveals consistent structure: adjacent layers have unusually similar weights, and groups of nearby layers form clusters, which supports the idea that transformer layers specialize by depth. If these claims hold, DOCS is a cheap, weight-only way to inspect trained models and could guide layer sharing, pruning, distillation, and fine-tuning strategies.","feed_headline":"DOCS finds layer clusters that older weight-similarity tools miss","feed_subtitle":"A cosine-based score distinguishes orthogonal matrices and shows adjacent LLM layers form functional clusters.","key_machinery":"The object carrying the argument is the per-column maximum absolute cosine similarity. For two weight matrices X and Y, DOCS computes the absolute cosine between every column of X and every column of Y, records, for each column of X, the largest value it attains against columns of Y, and does the same in the reverse direction; it then fits a Gumbel distribution to each of the two lists of maxima by maximum likelihood and takes the average of the two fitted location parameters as the similarity score. The location parameter acts as a summary of how closely the best-matching columns align. The theoretical example that proves discriminative power pairs a matrix of standard basis vectors with a normalized Hadamard matrix, so every absolute cosine is exactly 1/sqrt(m), giving the stated small DOCS value despite the large Frobenius difference.","core_discovery":"The paper's central claim is that DOCS (Distribution of Cosine Similarity) is a valid similarity index for weight matrices, and one that is discriminative where established indices are blind. Existing measures such as CCA, SVCCA, and linear CKA are shown, with proofs, to return a constant score for any pair of orthogonal matrices; DOCS instead assigns different scores to different orthogonal pairs. Theorem 1 constructs column-orthogonal matrices X and Y with Frobenius difference $\\Omega$($\\sqrt$(m)) and DOCS score 1/$\\sqrt$(m), demonstrating that the index can register large structural differences that standard indices collapse to a constant. On real models, DOCS heatmaps show strong similarity between adjacent layers, decreasing similarity with layer distance, and clusters of mutually similar layers, which the paper reads as evidence of depth-wise functional specialization.","pith_inferences":["The paper leaves no null distribution for DOCS, so a natural next step is a permutation test that shuffles columns within layers and asks whether adjacent-layer scores exceed chance; this would calibrate the reported clusters as statistically significant or not.","If the clusters genuinely mark redundant computation, sharing or dropping weights inside a detected cluster should degrade downstream performance less than the same intervention across clusters, a prediction the paper motivates but does not test.","Because DOCS is orientation-sensitive, rerunning the pipeline on row vectors instead of the transposed column vectors used here would show how much of the reported layer structure depends on the chosen neuron ordering convention.","The same analysis applied to vision transformers or diffusion models offers a direct test of the paper's implicit claim that adjacent-layer similarity is a generic outcome of training residual stacks."],"forward_implications":["Adjacent transformer layers in the studied decoder-only LLMs share unusually similar weights, and weight similarity decreases as layer distance grows, with a secondary rise at the first and last layers.","Groups of mutually similar consecutive layers exist at consistent depths across model families and sizes, suggesting a universal structural pattern from training.","Base and instruction-tuned versions of the same model keep DOCS scores above 0.7 for every matrix, and the matrices split into three families: MLP up/down, query/key, and value/output.","In the MoE model studied, most layers have a single expert whose weights stand apart from the others, consistent with expert specialization or routing imbalance.","DOCS has a higher Gini coefficient on layer-similarity heatmaps than the seven other indices tested, meaning its similarity signal is concentrated in fewer, stronger layer pairs."],"supporting_citations":[{"why":"Defines linear CKA, the primary baseline that DOCS is shown to beat on orthogonal matrices and on heatmap clarity.","marker":"Kornblith et al., 2019"},{"why":"Defines SVCCA, one of the indices proved constant on orthogonal matrices.","marker":"Raghu et al., 2017"},{"why":"Defines CCA-based similarity indices and motivates representation-similarity analysis, the baseline family DOCS extends to weights.","marker":"Morcos et al., 2018"},{"why":"Provides the claim that orthogonal matrices occur in LLM training, motivating the discriminative-on-orthogonal requirement.","marker":"Tian et al., 2023"},{"why":"Explains the residual connection that motivates weight-level rather than representation-level analysis.","marker":"He et al., 2016"},{"why":"Reports adjacent-layer redundancy and stages of inference, used to interpret DOCS adjacency clusters.","marker":"Lad et al., 2024"},{"why":"Demonstrates block-level redundancy in LLMs, supporting the cluster interpretation.","marker":"Song et al., 2024"},{"why":"Shows analogous attention patterns in nearby layers, corroborating the adjacent-layer weight similarity finding.","marker":"Mu et al., 2024"}],"fun_headline_variants":["DOCS reveals hidden layer clusters in LLMs","DOCS distinguishes orthogonal matrices that CKA cannot","New index DOCS spots functional grouping in LLM layers","Weight similarity tool DOCS uncovers depth-wise specialization","Cosine-based DOCS proves orthogonal similarity and sees clusters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The index assumes that fitting a Gumbel curve to the largest per-column cosine similarities is always well-defined and that the fitted location parameter reflects genuine functional overlap, rather than shared input statistics or chance alignments.","fun_headline_variants_meta":{"raw":{"variants":["DOCS reveals hidden layer clusters in LLMs","DOCS distinguishes orthogonal matrices that CKA cannot","New index DOCS spots functional grouping in LLM layers","Weight similarity tool DOCS uncovers depth-wise specialization","Cosine-based DOCS proves orthogonal similarity and sees clusters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000688,"raw_usage":{"total_tokens":3056,"prompt_tokens":818,"completion_tokens":2238,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":434,"completion_tokens_details":{"reasoning_tokens":2163}},"tokens_in":434,"tokens_out":2238,"duration_ms":19933,"temperature":1.0,"reasoning_tokens":2163,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T11:41:23.896779+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the algorithm described in the paper on the X and Y constructed in Theorem 1; every per-column maximum absolute cosine similarity is 1/sqrt(m), a constant vector, and maximum-likelihood fitting of a Gumbel distribution to a constant sample is degenerate, so Algorithm 1 will not output the claimed 1/sqrt(m) without an added tie-breaking or smoothing step. A reproduction that yields the claimed value would therefore have to use a different estimation procedure than the one defined.","supporting_citations":[],"review_version":1}