{"id":"35b1a33d-4af2-47d3-9529-964a1ff3fa7c","arxiv_id":"2412.00359","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Shared-weight self-attention, using one matrix and three diagonal scalers for Q, K, and V, cuts BERT's attention parameters by two-thirds while giving roughly comparable GLUE accuracy, though claimed accuracy gains are not supported.","lead":"A BERT variant replaces the three separate projection matrices for keys, queries, and values with one shared matrix plus three per-dimension scaling vectors, cutting attention-block parameters by about 66 percent. Tests on GLUE and SQuAD show roughly comparable accuracy to the standard model, but the paper's headline accuracy and speedup claims are inconsistent with its own tables.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The accuracy-parity claim is not yet supported: the baselines are not pretrained under the shared model's recipe, and the abstract's stated 0.38% gain over standard BERT contradicts Table 1's own averages.","rationale":"The stress-test pass agrees with the reader's weakest assumption. The parameter-reduction claim can be verified directly (d^2+3d vs 3d^2 gives 66.53% for d=768), so it is not the point of contention. The load-bearing part of the central claim is the empirical parity. The paper's own Section 2.5 gives a detailed pretraining setup for the shared model, but no comparable setup for the baselines; the baselines are introduced only by citation. Without matched pretraining, the small reported differences are well within the range of known BERT pretraining/fine-tuning variance, so no conclusion about the architecture follows. The internal inconsistency between the abstract (+0.38% over standard) and Table 1/§2.6 (-0.05% vs standard) further weakens the advertised result. A single controlled rerun of the standard baseline with the same recipe would settle the question. If the matched standard model lands at 79.97 and the shared model at 79.92, the 'maintaining accuracy' claim survives; if not, the central claim fails. The recommendation is to keep the reader's REJECT verdict: the paper's architectural idea may be worth investigating, but the evidence presented is not sufficient to support the central claim. No change to the reader's verdict is needed.","tokens_in":9607,"tokens_out":11656,"duration_ms":105362,"concrete_test":"Retrain the standard BERT-base baseline from scratch with the exact pretraining recipe in §2.5 (BooksCorpus + English Wikipedia, 20 epochs, batch size 132, four H100 GPUs, 0.15 mask ratio, Adam with the stated betas and weight decay) and fine-tune it on the eight GLUE tasks with the hyperparameters in §A.4, using at least three seeds. Also run the symmetric and pairwise baselines under the same protocol if their checkpoints are not available. If the matched standard baseline's average is not within a bootstrap 95% interval of the reported 79.97, or if the shared model is not statistically indistinguishable from the matched baselines, the accuracy-parity claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the shared-weight factorization (Wq=WsDq, Wk=WsDk, Wv=WsDv) cuts attention parameters by 66.53% and total BERT parameters by 12.94% while maintaining or improving GLUE accuracy. The parameter arithmetic is correct, but the empirical half of the claim rests on Table 1. Section 2.5 describes pretraining only the shared model: BooksCorpus+Wikipedia, 20 epochs, batch size 132, four H100 GPUs, masked LM ratio 0.15. Section 2.4 lists the baselines by citation (Devlin et al., 2018; Courtois et al., 2024) and never states that those baselines were pretrained and fine-tuned under the same data, steps, batch size, and hyperparameters. If the standard, symmetric, and pairwise numbers are external, the observed deltas (-0.05% vs standard, +3.55% vs symmetric, +0.89% vs pairwise) may reflect pretraining budget or fine-tuning choices rather than the architecture. The abstract's '0.38% improvement over the standard' is also inconsistent with Table 1, where the shared average (79.92) is 0.05% below the standard average (79.97); Section 2.6 correctly reports -0.05%. Thus the headline accuracy claim is internally contradicted and externally uncontrolled, so the central 'maintaining accuracy' assertion is not established as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes replacing the three separate projection matrices Wq, Wk, Wv in BERT self-attention with a single shared matrix Ws followed by three diagonal scaling matrices Dq, Dk, Dv. The authors claim this reduces attention-block parameters from 3d^2 to d^2+3d (a 66.53% reduction) and total BERT-base parameters by 12.94%, while maintaining or improving GLUE accuracy, improving robustness to noisy inputs, and reducing training time. The parameter arithmetic is straightforward and largely correct, but the empirical claims are not yet supported as written because of internal contradictions, unmatched baselines, and apparent arithmetic errors in the reported averages.","tokens_in":9986,"tokens_out":6790,"duration_ms":58937,"significance":"If the accuracy-parity and efficiency claims could be substantiated under controlled conditions, the shared-weight factorization would be a simple, useful drop-in modification for parameter-efficient Transformer encoders. The parameter counting in Section 3 and Table 4 is a strength: the reduction is derived directly from the equations and is machine-checkable. The public release of model and code is also a positive feature. However, the empirical half of the central claim currently rests on comparisons whose validity is undermined by the issues detailed below; the headline numbers in the abstract and abstract-adjacent text do not match the paper's own tables.","major_comments":[{"comment":"The abstract's accuracy claims contradict the body: the abstract reports improvements of 0.38%, 5.81%, and 1.06% over standard, symmetric, and pairwise BERT, but Section 2.6 and Table 1 show -0.05%, +3.55%, and +0.89%. The Table 1 averages (79.92 vs 79.97) actually show the shared model 0.05% below standard. These are load-bearing discrepancies in the central claim and must be resolved before the paper can be evaluated.","section":"Abstract and Section 2.6"},{"comment":"The abstract's claim of training time reduction by 'around one-tenth' is inconsistent with Figure 3, which reports reductions of 11% to 43%, and with Table 7, which says '15-20% faster'. A 90% reduction is very different from an 11-43% reduction, and the paper does not provide any experiment supporting a tenfold speedup.","section":"Abstract, Figure 3, and Table 7"},{"comment":"The baselines are not matched. Section 2.5 describes pretraining only for the shared model, while Section 2.4 lists standard, symmetric, and pairwise BERT by citation. The text never states that these baselines were pretrained and fine-tuned with the same data, number of steps/epochs, batch size, sequence length, and hyperparameters as the shared model. Without such control, the deltas in Table 1 (and in Tables 2 and 5) cannot be attributed to the attention architecture rather than to training budget or hyperparameter differences.","section":"Sections 2.4, 2.5, and 2.6"},{"comment":"The SQuAD averages are arithmetically wrong for the standard model: the average EM should be (82.18+79.35)/2 = 80.77, not 80.10, and the average F1 should be (90.01+83.65)/2 = 86.83, not 81.47. With corrected numbers, the shared model is worse than standard in both EM (80.20 vs 80.77) and F1 (86.30 vs 86.83), reversing the favorable comparison implied by the table. The column header 'SQuAD v1.2' also does not match the text's SQuAD v2.0.","section":"Table 2"},{"comment":"No error bars, confidence intervals, or multiple-seed results are reported. Differences of the magnitude claimed in the GLUE table (e.g., 0.05% overall and task-level differences below 1 point) are within the typical run-to-run variance of BERT fine-tuning, so the accuracy-parity claim is not statistically supported as presented. At minimum, the authors should report the mean and variance over several fine-tuning seeds for every configuration.","section":"Tables 1, 2, and 5"},{"comment":"The cross-domain experiment is not described. Table 6 purports to show performance of standard and shared models on MNLI, QNLI, QQP, and MRPC, but neither Section 3 nor Appendix A explains whether each row corresponds to a model fine-tuned on that row's domain and evaluated on the other domains, what the fine-tuning protocol was, or how the off-diagonal entries were obtained. Without this protocol, the claimed 'cross-domain performance' advantage is not verifiable.","section":"Section 3, Table 6"}],"minor_comments":[{"comment":"The parameter reduction is reported as 66.53% in the abstract and Table 3, but as 66.67% in Table 7 and Section 5. The exact percentage depends on d (for d=768 it is about 66.54%), so the paper should use one consistent rounded value and state the formula.","section":"Sections 3 and 5, Table 7"},{"comment":"The dataset is called 'SQuAD v1.2' in Table 2 but 'SQuAD v2.0' in the text and Appendix A.2; please standardize to the correct version name.","section":"Section 2.7 and Table 2"},{"comment":"The row label 'QNL' appears to be a typo for 'QNLI'.","section":"Table 6"},{"comment":"The pretraining description says '20 epochs' but the loss curves are described as stabilizing at 'approximately 200,000 steps.' The relationship between these numbers is unclear; please report the total number of training steps and the effective token budget.","section":"Section 2.5"},{"comment":"The limitation statement says 'Our models were benchmarked with GLUE and the newer SuperGLUE,' but no SuperGLUE experiments appear anywhere in the paper; this sentence should be removed or the experiments added.","section":"Appendix A.1"},{"comment":"The GLUE average appears to treat MNLI matched and mismatched as two separate scores (nine scores total). Please state this averaging convention explicitly, since it affects the reported averages.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The shared-weight factorization itself is simple and the parameter-reduction calculation is sound, but the empirical case is not currently credible because of the abstract/body contradictions, the statistically uncontrolled baselines, and the incorrect SQuAD averages. I believe these are fixable with a controlled rerun of the baselines, proper multi-seed reporting, and correction of the internal inconsistencies, so I am not recommending rejection. However, if the authors cannot provide matched baselines, the paper should be reframed as a parameter-efficiency study with no accuracy-parity claims against external numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: the proposed factorization — Wq=WsDq, Wk=WsDk, Wv=WsDv with a full shared matrix and three diagonal scaling matrices — is a genuine, natural variant of the symmetric/pairwise attention baselines, and the parameter-count arithmetic is correct. d²+3d vs 3d² gives a 66.53% reduction for d=768, which translates to ~12.9% total BERT parameters. That part is solid.\n\nThe paper does not, however, establish that the shared model maintains accuracy. The abstract claims a 0.38% improvement over standard BERT on GLUE, but Table 1 shows the shared model averaging 79.92 vs standard's 79.97, i.e., -0.05%. The abstract also claims training time 'around one-tenth,' while Figure 3 and the text report 11–43% reductions. Those are internal contradictions, not minor typos, because the accuracy-maintenance claim is the load-bearing empirical result. Adding to that, the baselines (Devlin et al.; Courtois et al.) are cited but never shown to be pretrained and fine-tuned under the same recipe. Section 2.5 describes pretraining only the shared model. If the baseline numbers are imported from prior work, the small deltas are uninterpretable.\n\nThe robustness and cross-domain sections are suggestive but also under-controlled. The noise-injection procedure is described cursorily, and the cross-domain table contains at least one obvious typo ('QNL') and some odd off-diagonal values. No error bars or repeated runs appear anywhere.\n\nOn balance: the core idea is real, simple, and worth a controlled follow-up. Whoever picks this up should pretrain all variants under identical data, steps, and hardware, report variance, and reconcile the abstract with the tables. The current version is not ready to be taken at face value, and the internal contradictions alone justify a major revision.\n\nI'd send this to a serious referee — the parameter-reduction claim is checkable and the architecture is novel enough to warrant scrutiny — but the referee should be told to focus on empirical comparability first. For a reading group, it's a useful example of how an efficiency claim can outrun its evaluation.","headline":"A genuinely simpler attention factorization with correct parameter arithmetic, but the empirical claims are internally contradictory and uncontrolled.","tokens_in":10507,"tokens_out":2677,"would_cite":false,"duration_ms":23264,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single shared weight matrix can replace the three Query, Key, and Value matrices in a transformer's self-attention, cutting attention-block parameters by two-thirds while keeping BERT-level accuracy.","keywords":["shared weight self-attention","BERT","parameter reduction","GLUE benchmark","transformer efficiency","noise robustness","attention factorization"],"falsifier":"Pretrain standard BERT and the shared-weight BERT from the same corpora with identical hyperparameters and step counts, fine-tune both on the same GLUE splits, and compare average accuracy; if the shared model falls more than roughly one point behind standard BERT, the claim that two-thirds fewer attention parameters preserve accuracy would be refuted.","tokens_in":9439,"feed_emoji":"⚡","tokens_out":8615,"duration_ms":68932,"temperature":0.7,"pith_summary":"This paper asks whether the three separate weight matrices that transformers use to compute Queries, Keys, and Values are actually necessary. It argues they are not: a single learned matrix Ws followed by three diagonal scaling matrices Dq, Dk, and Dv can play the same role. In a BERT-base model, this factorization cuts self-attention parameters by 66.53% and total model parameters by 12.94%, while keeping GLUE accuracy within 0.05 points of the standard BERT baseline and improving robustness to noisy and out-of-domain inputs. The reason a curious reader should care is that it points to a simple drop-in change that makes transformer encoders cheaper to train, store, and deploy without sacrificing language-understanding performance.","feed_headline":"One matrix instead of three: shared attention slashes BERT weights","feed_subtitle":"A merged Query/Key/Value weight cuts attention-block parameters by 66.5% and keeps GLUE accuracy near the BERT baseline.","key_machinery":"The load-bearing object is the shared transformation S(X) = XWs with diagonal role matrices Dq, Dk, and Dv; the parameter count falls from $3d^{2}$ to $d^{2}$ + 3d. Each attention head computes softmax((SDq)(SDk)^T / $\\sqrt$(d))(SDv), so queries and keys are scaled versions of the same representation rather than independent projections. The diagonal matrices are what let one matrix act as all three while preserving the asymmetry between the query and key roles.","core_discovery":"The paper's central claim is that standard self-attention's Wq, Wk, Wv can be factored as Wq = WsDq, Wk = WsDk, and Wv = WsDv, with a single learned d x d matrix Ws and three diagonal d x d matrices Dq, Dk, Dv. This keeps the roles of query, key, and value distinct through element-wise scaling while replacing $3d^{2}$ attention parameters with $d^{2}$ + 3d, a 66.53 percent reduction. Pretrained on the same BooksCorpus and Wikipedia data as BERT-base and fine-tuned on GLUE, the shared-weight model is reported to stay essentially at parity (average GLUE 79.92 versus 79.97 for standard BERT) while using 95.3 million total parameters instead of 109.5 million, and to degrade more slowly than standard attention when Gaussian noise is added to input embeddings. The authors take this as evidence that the expressive burden carried by separate projection matrices is largely redundant.","pith_inferences":["If the factorization is as interchangeable as claimed, the same Ws plus Dq, Dk, Dv decomposition should transfer to decoder self-attention and cross-attention; the paper explicitly leaves that untested, and a small translation or generation experiment would separate the architecture's benefit from encoder-specific effects.","The diagonal scales in the paper are single matrices rather than per-head or per-layer parameters; a natural variant that learns separate diagonal scales per head could recover any lost expressiveness on harder NLU tasks while keeping most of the parameter savings.","The noise-robustness results suggest the shared matrix behaves like parameter-sharing regularization; if that is the mechanism, tying the shared matrix across layers should preserve or amplify the gains, which is a testable extension the paper does not run.","The paper itself flags that its efficiency findings were observed on a roughly 100-million-parameter BERT, so a matched-budget replication on a model an order of magnitude larger would determine whether the 12.94 percent total-parameter saving survives scaling."],"forward_implications":["Attention-block parameters fall by about two-thirds, and total BERT-base parameters fall from 109.5 million to 95.3 million, making the encoder cheaper to store and fine-tune.","Reported GLUE fine-tuning times drop by 11 to 30 percent per task at one epoch, so training budgets shrink under the same hardware.","Average GLUE accuracy stays within 0.05 points of standard BERT, with the largest relative gains on MRPC, CoLA, and STS-B against the symmetric and pairwise baselines.","Under 0 to 40 percent added input noise, the shared model loses accuracy more slowly than standard attention on MNLI, QQP, and SST-2, suggesting more robust representations.","Cross-domain transfer improves in several settings, such as an MNLI-trained model scoring higher on QNLI and MRPC, consistent with the shared matrix acting as a regularizer."],"supporting_citations":[{"why":"Supplies the BERT architecture and baseline whose attention block is replaced, and the pretraining configuration the shared model adopts.","marker":"Devlin et al., 2018"},{"why":"Defines the standard self-attention mechanism with separate Wq, Wk, Wv weight matrices that the shared factorization is compared against.","marker":"Vaswani et al., 2017"},{"why":"Provides the symmetric and pairwise self-attention BERT models used as accuracy and parameter-count baselines.","marker":"Courtois et al., 2024"},{"why":"Supplies the GLUE benchmark and its per-task evaluation metrics for the downstream accuracy comparisons.","marker":"Wang et al., 2018"},{"why":"Supplies the SQuAD v1.1 question-answering dataset used in the fine-tuning evaluation.","marker":"Rajpurkar et al., 2016"},{"why":"Supplies the SQuAD v2.0 question-answering dataset used in the fine-tuning evaluation.","marker":"Rajpurkar et al., 2018"},{"why":"Supplies the BooksCorpus pretraining corpus whose 800M words contribute to the shared model's training data.","marker":"Zhu et al., 2015"}],"fun_headline_variants":["Shared attention weight: one matrix instead of three","Single QKV weight cuts BERT attention params by 66.5%","Fewer attention weights, same GLUE accuracy","One weight matrix for attention speeds training","Self-attention simplified: shared weight matrix"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy comparisons assume the standard, symmetric, and pairwise BERT baselines were pretrained and fine-tuned under the same data, steps, and hyperparameters as the shared-weight model, and the paper never explicitly states that these conditions were matched.","fun_headline_variants_meta":{"raw":{"variants":["Shared attention weight: one matrix instead of three","Single QKV weight cuts BERT attention params by 66.5%","Fewer attention weights, same GLUE accuracy","One weight matrix for attention speeds training","Self-attention simplified: shared weight matrix"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1445,"prompt_tokens":960,"completion_tokens":485,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":411}},"tokens_in":576,"tokens_out":485,"duration_ms":5058,"temperature":1.0,"reasoning_tokens":411,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:29:04.622335+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Pretrain standard BERT and the shared-weight BERT from the same corpora with identical hyperparameters and step counts, fine-tune both on the same GLUE splits, and compare average accuracy; if the shared model falls more than roughly one point behind standard BERT, the claim that two-thirds fewer attention parameters preserve accuracy would be refuted.","supporting_citations":[],"review_version":1}