{"id":"a677fd48-a33c-4bfa-a820-4bd148e55a3e","arxiv_id":"2412.00884","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Fixing layers beyond the effective depth, and transformer feedforward layers, to simplex ETFs gives nearly unchanged Fashion-MNIST accuracy with fewer parameters, though the method is underspecified.","lead":"Two training schemes are proposed: Adaptive-ETF freezes any MLP layer to a simplex equiangular tight frame once its neural collapse measure drops below a threshold, and ETF-Transformer does the same for all feedforward layers in a vision transformer. The reported Fashion-MNIST accuracy is close to baseline with large parameter savings, but the construction of intermediate ETF layers is not specified and no code or error bars are provided.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No construction given for applying the k×C simplex ETF of §2.2 to hidden layers whose width is not the class count; reported experiments may rely on an unstated square-ETF variant.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the simplex ETF defined in §2.2 has shape k×C and only matches the final classifier, not hidden layers or transformer feedforward layers. My analysis of the parameter counts in Table 1 confirms that the reported savings correspond to fixing full square weight matrices, which would require a separate, unstated construction. This is a genuine gap, not a matter of tuning or convention: the paper's own definition of a simplex ETF cannot be applied to the architectures it claims to constrain. The internal inconsistency in the accuracy claims (the 'does not impact' statement versus Table 1's drops) is secondary but reinforces the need for a careful re-evaluation. No independent evidence (code, error bars, ablations) is provided. Therefore, the reader's REJECT verdict is appropriate, and no change to the verdict is warranted. A constructive path forward would be to release code and either extend the ETF definition to square (or non-class) matrices with proper theoretical grounding, or restrict the claims to layers whose input/output dimensions match a class-count ETF.","tokens_in":5127,"tokens_out":4112,"duration_ms":40325,"concrete_test":"Obtain or re-implement the 'Last three layers ETF' MLP. Attempt to instantiate the third-from-last layer with the §2.2 simplex ETF using k=128 and C=10 (the number of Fashion-MNIST classes). This matrix is 128×10, so it cannot serve as a 128→128 linear layer; the forward pass would fail at the next layer. If the authors' code instead uses a 128×128 matrix built from the ETF formula with C=128, that is a different object and should be documented as such. Run the same classification experiment with the formula as written; if the architecture cannot be built or deviates from the stated definition, the central claim is unsubstantiated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that intermediate layers and transformer feedforward layers can be fixed to simplex ETFs without accuracy loss, but §2.2 defines the simplex ETF as a k×C matrix, where C is the number of classes (10 for Fashion-MNIST). In the MLP experiments, hidden layers map 128→128, so their weight matrices are 128×128. The parameter savings in Table 1 (16,384 per hidden layer) correspond exactly to fixing a full 128×128 weight matrix, which would require an ETF with C=128, not the C=10 simplex ETF defined in the paper. No construction, formula, or justification is given for such a square ETF, nor for the transformer FC layers (d_model↔2048). Consequently, the experiments as described cannot be reproducing the stated ETF constraint. If square ETFs were used, the link to neural collapse—which concerns class means forming a simplex ETF—is unsupported and the theoretical motivation collapses. Additionally, Table 1 itself shows test accuracy drops of 0.38% (last two layers) and 1.08% (last three layers) relative to baseline, contradicting the §4.1 assertion that setting layers past L0 to ETFs 'does not impact the train or test accuracy.' The primary load-bearing defect, however, is the missing shape-compatible ETF construction; without it, the reported parameter reductions and accuracies are not attributable to the proposed method.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes to reduce trainable parameters by fixing network layers beyond an empirically measured effective depth to simplex equiangular tight frames (ETFs). Two training schemes are introduced: Adaptive-ETF, which freezes a layer once its NCC error falls below a threshold epsilon, and ETF-Transformer, which applies ETF constraints to the feedforward layers of a vision transformer. Experiments on Fashion-MNIST report accuracies close to baseline (e.g., 89.78% vs. 89.80% for the ETF-Transformer) while cutting parameters from 9.49M to 3.20M. The central assertion is that intermediate layers and transformer FC layers can be constrained to ETFs with negligible accuracy loss.","tokens_in":5389,"tokens_out":9127,"duration_ms":84845,"significance":"If the construction were fully specified and reproducible, the empirical findings would be a useful step toward parameter-efficient training: the ViT results in Table 2 are striking, and the idea of freezing layers once they reach NCC separability is a concrete, testable heuristic. The paper is also candid about the absence of neural collapse in transformer FC layers. However, the current manuscript does not deliver on this: the ETF constraint for non-final layers is undefined, the adaptive rule is underspecified, and the 'no impact' claim is contradicted by the paper's own table. As submitted, the experiments cannot be attributed to the proposed method, so the reported efficiencies are not established. No code, pseudocode, or architecture-level construction is provided.","major_comments":[{"comment":"Section 2.2 defines the simplex ETF as a k×C matrix whose columns are indexed by the C classes. The MLP hidden layers in Table 1 are 128×128, and fixing one hidden layer removes exactly 16,384 parameters, i.e., the full weight matrix. No construction is given for an ETF of shape 128×128; the §2.2 construction with C=10 would produce a 128×10 matrix and save only 1,280 parameters per layer. The transformer FC layers (d_model×2048) have the same problem. If the authors instead used a square ETF, the number of frame vectors would equal the hidden width (128), not the number of classes, so the claimed link to neural collapse—where the C class means form a simplex ETF—does not apply. This missing construction is load-bearing: without it, the parameter reductions and accuracies in Tables 1 and 2 cannot be attributed to the proposed method.","section":"§2.2, §4.1, Table 1"},{"comment":"The text states that 'setting all layers past L0 to simplex ETFs does not impact the train or test accuracy of the network,' but Table 1 shows drops of 0.38 and 1.08 percentage points in test accuracy for the last-two and last-three-layer ETF models, together with larger drops in training accuracy. These are small effects, but the claim of 'no impact' is inaccurate. The same section later describes these settings as having 'lowered' accuracy, so the qualitative summary should be corrected or the experiments should be repeated with confidence intervals.","section":"§4.1"},{"comment":"The description of Adaptive-ETF is incomplete: it does not specify when the NCC error is evaluated (at initialization, after some number of epochs, or continuously), whether the freeze decision is made once or repeatedly, whether gradients flow through frozen layers, or whether biases and normalization parameters remain trainable. The parameter counts in Table 1 suggest that biases remain trainable, but this is never stated. A precise algorithm or pseudocode is needed to make the method reproducible.","section":"§4.1, Adaptive-ETF"},{"comment":"The paper reports that neural collapse is not directly observed in the FC layers of transformer blocks, yet the ETF-Transformer fixes all such layers to ETFs. Since the theoretical motivation of the paper is the neural-collapse/ETF equivalence (NC2, NC3), this is a nontrivial gap. The authors should either provide a theoretical or empirical justification for applying the ETF constraint outside the collapsed regime, or explicitly reframe ETF-Transformer as an empirically motivated parameter-sharing scheme that is only inspired by neural collapse.","section":"§4.2"},{"comment":"The training accuracy of the 'Last layer + Last Block FC ETF' model is reported as 98.89%, almost five points above the baseline's 93.91%, with no explanation. Since the paper uses the same optimizer and training schedule for all models, this large discrepancy is surprising and should be clarified. A plausible explanation, such as a difference in checkpoint selection or a metric definition issue, needs to be provided before the parameter-efficiency claims can be fully trusted.","section":"Table 2"}],"minor_comments":[{"comment":"The text 'tscope of neural collapse has been expanded' should read 'the scope of neural collapse has been expanded'.","section":"§1"},{"comment":"'the featureshgo to zero' is missing a space and should read 'the features h go to zero'.","section":"§2.1"},{"comment":"'Simplfication' should be 'Simplification'.","section":"§2.4"},{"comment":"'Figure 1 shows the compares' should be 'Figure 1 shows the comparison'.","section":"§4.1"},{"comment":"'replaying layers beyond a fully-connected network’s effective depth' should be 'replacing layers beyond a fully-connected network’s effective depth'.","section":"§4.1"},{"comment":"'worthwile' should be 'worthwhile'.","section":"§5"},{"comment":"Figures 1–4 are referenced but not embedded in the text; the captions refer to 'Training progress' and NCC results, but the actual plots are missing, which makes the empirical claims difficult to verify.","section":"Figures"}],"recommendation":"reject","confidential_remarks":"The core problem is reproducibility: the central operation of the method, setting intermediate weight matrices to simplex ETFs, is not defined for the shapes used in the experiments. A resubmission could be considered if the authors provide an explicit construction for arbitrary-width ETF constraints, pseudocode for Adaptive-ETF, and a corrected interpretation of the empirical results; however, as submitted, the paper's main claims are not supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a workshop submission applying simplex ETF constraints beyond the final layer to intermediate MLP layers and transformer feedforward layers. The direction is reasonable: Galanti et al. showed NCC separability holds past effective depth, so freezing those layers to a fixed frame is a natural efficiency idea. The Adaptive-ETF thresholding rule is also sensible. Credit where due: the paper cites the right prior work and the empirical scope (MLP and ViT) is broader than the usual final-layer-only studies.\n\nThe problem is that the paper never specifies the central operation. §2.2 defines the simplex ETF as a k×C matrix, C the number of classes. Hidden MLP layers are 128×128 and ViT FC layers are d_model×2048. There is no construction for turning those weight matrices into a simplex ETF. The parameter counts in Table 1 reveal the issue: fixing one extra hidden layer saves exactly 16,384 parameters, which is the full 128×128 weight matrix. That would require an ETF with C=128, not the C=10 ETF defined in the paper. So the experiments as described cannot be reproducing the stated constraint. If the author used a square ETF, the link to neural collapse—which concerns class means—is unsupported.\n\nThere is also an internal contradiction. §4.1 says 'setting all layers past L0 to simplex ETFs does not impact the train or test accuracy,' but Table 1 shows 89.20% and 88.50% test accuracy for last-two and last-three ETF versus 89.58% baseline—drops of 0.38 and 1.08 points. The Adaptive-ETF row also shows a drop. So the headline claim is contradicted by the paper's own table.\n\nEverything else is secondary: no code, no error bars, single dataset, one run. The ViT results are interesting but rest on the same unspecified construction.\n\nIf the author provided a clear definition of a dimension-matched ETF, justified its connection to neural collapse, and showed the experiments actually use that construction, the paper could be a useful workshop contribution. As stands, the load-bearing implementation is unverifiable and likely not what was described. I would not send this to a full review cycle; desk reject with an invitation to resubmit after fixing the construction and reconciling the text with Table 1.","headline":"Plausible idea, but the central construction is missing: the paper never explains how a k×C simplex ETF can be applied to 128×128 hidden layers or transformer FC layers, and its own Table 1 contradicts the claim that fixing layers past effective depth is accuracy-neutral.","tokens_in":5878,"tokens_out":3381,"would_cite":false,"duration_ms":30680,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A simplex equiangular tight frame can replace trained layers beyond a network's effective depth, and transformer feedforward layers, without significant loss of accuracy.","keywords":["neural collapse","simplex equiangular tight frame","effective depth","adaptive parameter freezing","transformer feedforward layers","Fashion-MNIST","parameter efficiency"],"falsifier":"Count the parameters in Tables 1 and 2 against the stated ETF construction: the simplex ETF defined in Section 2.2 has one column per class, so a 128-by-128 MLP hidden layer or a transformer feedforward layer whose output width is four times its input width cannot be represented by it without an additional construction. Checking whether the reported parameter savings match the number of weights in those specific layers, and whether the forward pass actually uses a matrix of ETF shape, would settle whether the constraint was applied as claimed.","tokens_in":4917,"feed_emoji":"📉","tokens_out":8131,"duration_ms":69230,"temperature":0.7,"pith_summary":"This paper tries to establish that neural collapse is not just an observed end state but a usable training constraint: once a layer's representations have converged to a simplex equiangular tight frame (ETF), the layer can be pinned to that fixed frame and left untrained. It proposes Adaptive-ETF, which freezes every layer past a network's effective depth once its nearest-class-center error drops below a threshold, and ETF-Transformer, which applies the same idea to the feedforward layers inside transformer blocks. On Fashion-MNIST, the paper reports that a five-layer MLP with layers past the effective depth replaced by ETFs trains to 89.38% test accuracy versus 89.58% for baseline, and that a vision transformer with all feedforward layers outside multi-head attention fixed to ETFs reaches 89.78% versus 89.80% while using 3.20M parameters instead of 9.49M. The payoff would be a cheap way to cut trainable parameters and memory in overparameterized networks without sacrificing accuracy.","feed_headline":"ETFs cut transformer parameters to one-third at equal accuracy","feed_subtitle":"A vision transformer keeps 89.78% test accuracy with 3.20M parameters instead of 9.49M.","key_machinery":"The central object is the simplex equiangular tight frame (ETF), a set of $C$ vectors in $\\mathbb{R}^k$ that are equal-norm, mutually equidistant, and maximally separated; the paper writes it as $\\dot{M} = \\sqrt{C/(C-1)}\\,P\\,(I_C - (1/C)\\mathbf{1}\\mathbf{1}^\\top)$. It works together with the effective depth $L_0$, the smallest layer index at which a network's nearest-class-center (NCC) accuracy reaches a threshold. The mechanism is identification: because neural collapse drives class means and classifier weights toward an ETF, a layer whose NCC error is below $\\epsilon$ has already converged to that geometry, so replacing its weights with the fixed ETF and excluding them from training should preserve the network's behavior while removing parameters. NCC accuracy serves as the empirical collapse meter throughout the experiments.","core_discovery":"The paper's central claim is that the simplex ETF structure associated with neural collapse propagates to intermediate layers, and that those layers can be replaced by a fixed ETF once they reach a measurable collapse threshold. The paper argues that because neural collapse drives class means and classifier weights to the same ETF, enforcing that ETF at an intermediate layer beyond the effective depth does not change the network's function enough to hurt accuracy. It supports this with two demonstrations: an MLP whose layers past effective depth are replaced by ETFs, and a vision transformer whose feedforward layers outside multi-head attention are replaced by ETFs. In the strongest case, the transformer keeps 89.78% test accuracy against a baseline of 89.80% while reducing parameters from 9.49M to 3.20M. The paper concludes that NC2 and NC3, originally stated for the final classifier, also apply to intermediate layers and can be leveraged for efficient training.","pith_inferences":["A natural extension the paper does not run is making the Adaptive-ETF threshold an online rule: if a layer is frozen as soon as its NCC error passes epsilon during training, the method becomes a dynamic parameter-freezing schedule rather than a fixed architecture chosen after the fact.","Because the transformer's parameter savings are concentrated in the 2048-unit feedforward layers, a testable variant would freeze only those layers and leave attention projections trainable to isolate exactly where the ETF constraint produces the reported gain.","The early-layer finding (lower train accuracy, unchanged test accuracy) points to a structured regularizer; measuring it on a larger, more diverse dataset than Fashion-MNIST would show whether the effect is robust or an artifact of the simple benchmark.","If the ETF shape were generalized beyond one column per class, the same idea could apply to layers whose width is unrelated to the class count, which would greatly widen the range of usable architectures."],"forward_implications":["Adaptive-ETF can be applied to any overparameterized fully connected network where layers beyond the effective depth satisfy the NCC threshold, producing most of the accuracy of the unconstrained network with a fraction of the trainable weights.","ETF-Transformer shows that the feedforward layers of transformer blocks, despite not exhibiting clean neural collapse, can be pinned to ETFs without changing prediction accuracy, which should extend to other transformer-based architectures beyond vision.","The paper's result that NC2 and NC3 hold in intermediate layers suggests that NCC-based generalization bounds could be rederived using ETF geometry rather than nearest-class-center error alone.","Fixing layers before the effective depth to ETFs lowers training accuracy while leaving test accuracy intact, so the same constraint may act as a regularizer in under-trained regimes."],"supporting_citations":[{"why":"Defines the four neural collapse conditions, including convergence to a simplex ETF, which is the geometric target the paper reuses.","marker":"[4]"},{"why":"Introduces effective depth and shows NCC separability in intermediate layers, the basis for extending ETF constraints past the final layer.","marker":"[3]"},{"why":"Shows that fixing the final classifier to a simplex ETF can reduce parameters without harming accuracy, the starting point this paper generalizes.","marker":"[7]"},{"why":"Supplies the vision transformer architecture modified in the ETF-Transformer experiments.","marker":"[2]"},{"why":"Provides the Fashion-MNIST dataset used in all reported training runs.","marker":"[6]"}],"fun_headline_variants":["Simplex ETF layers shrink transformers to one-third the parameters","Neural collapse's simplex ETF cuts parameters to a third at same accuracy","Simplex ETF on hidden layers: 66% fewer parameters, same test accuracy","Transformers cut to one-third size with simplex ETF intermediate layers","Intermediate neural collapse guides fixed layers, cuts parameters, matches accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a weight matrix of any shape—for example, a hidden MLP layer mapping 128 units to 128 units or a transformer feedforward layer whose output width is four times its input width—can be set to the same simplex ETF used for the final classifier, even though that ETF is defined to have exactly one column per class.","fun_headline_variants_meta":{"raw":{"variants":["Simplex ETF layers shrink transformers to one-third the parameters","Neural collapse's simplex ETF cuts parameters to a third at same accuracy","Simplex ETF on hidden layers: 66% fewer parameters, same test accuracy","Transformers cut to one-third size with simplex ETF intermediate layers","Intermediate neural collapse guides fixed layers, cuts parameters, matches accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001314,"raw_usage":{"total_tokens":5345,"prompt_tokens":930,"completion_tokens":4415,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":4324}},"tokens_in":546,"tokens_out":4415,"duration_ms":28278,"temperature":1.0,"reasoning_tokens":4324,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:53:49.790424+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Count the parameters in Tables 1 and 2 against the stated ETF construction: the simplex ETF defined in Section 2.2 has one column per class, so a 128-by-128 MLP hidden layer or a transformer feedforward layer whose output width is four times its input width cannot be represented by it without an additional construction. Checking whether the reported parameter savings match the number of weights in those specific layers, and whether the forward pass actually uses a matrix of ETF shape, would settle whether the constraint was applied as claimed.","supporting_citations":[{"cited_title":"A Geometric Analysis of Neural Collapse with Unconstrained Features","cited_arxiv_id":null,"evidence_quote":"Shows that fixing the final classifier to a simplex ETF can reduce parameters without harming accuracy, the starting point this paper generalizes."},{"cited_title":"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale","cited_arxiv_id":null,"evidence_quote":"Supplies the vision transformer architecture modified in the ETF-Transformer experiments."}],"review_version":1}