{"id":"acf9d314-c72c-4937-a36a-11564b3c7870","arxiv_id":"2506.14704","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"On structured SNOMED-derived memorization tasks, small transformers memorize most when embedding size is large and depth is kept low.","lead":"This paper trains small text-predicting models on sentences built from a medical knowledge graph and measures how many facts they can memorize. It finds that embedding size matters more than depth and that Softmax is more stable, offering a practical recipe for tiny on-device assistants.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The depth-vs-width parameter-equalization formula in Section 2.3.1 likely fails to hold total parameter count constant, undermining the central claim that embedding size, not depth, drives capacity.","rationale":"The reader's verdict (CONDITIONAL, moderate confidence) and weakest_assumption identify the parameter-count equalization formula as the key vulnerability. My stress-test confirms this is the single most load-bearing concern: the central claim about embedding size versus depth is only interpretable if total parameter count is actually held constant. The paper's own description ('model sizes ranged from 2.9 to 44.5 million parameters, primarily varying with embedding size and layer count, but also influenced by vocabulary size') and the inconsistent use of '16 parameters' versus 'embedding size 16 per layer with 32 total parameters' indicate the formula is not a true parameter budget. There is independent support for the paper's general direction: the theoretical bounds cited (Kim et al. 2023; Kajitsuka and Sato 2024) do make embedding dimension central in memorization capacity, and the paper is honest about its scope and limitations. However, the depth-versus-width experiment is the only place where the paper directly tests depth, and its confound prevents a clean conclusion. I do not see a reason to REJECT: the finding that embedding size affects learning speed is plausible and likely robust, but the stronger statement that 'adding layers without increasing embedding size neither accelerates convergence nor improves final capacity' needs a corrected parameter-matched experiment. Hence UNCHANGED would understate the need for this correction; CONDITIONAL (with the condition being a proper parameter-count check and/or rerun) is the right verdict. I agree with the reader that this is the weakest assumption.","tokens_in":11858,"tokens_out":1907,"duration_ms":16622,"concrete_test":"Recompute the true total parameter counts for the third-setup configurations (16/32/64/128 base, 1 vs 2 layers) using the actual vocabulary size from the SNOMED tokenizer and the stated architecture: embedding_matrix(vocab_size × d_model) + per-layer(attention + FFN + norms) + output_head. If the 2-layer configurations have substantially more total parameters than the corresponding 1-layer configurations (e.g., 2-layer '16-total' has roughly 2× the embedding parameters plus extra layer weights), then the fair-comparison premise fails. A cleaner follow-up: re-run the depth comparison while fixing total parameter count exactly via a proper budget (e.g., compute d_model for each depth from the true total-parameter equation) and check whether the embedding-size advantage survives.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim (Section 3.3) is that 'embedding size is the key factor influencing learning speed, while adding layers without increasing embedding size neither accelerates convergence nor improves final capacity.' This claim rests on the depth-versus-width comparison in the third setup of Section 2.3.1, which asserts that total parameter count is kept constant via embedding_size = floor(base_number_of_parameters / n_layers), with base values 16/32/64/128. This formula is not a valid parameter-count equalization for a transformer with a fixed vocabulary. The embedding matrix alone has vocab_size × d_model parameters; with a fixed vocabulary, reducing d_model by half when doubling layers reduces embedding parameters by half, while adding a layer adds attention/FFN parameters in all layers. Since the reported model sizes are 'primarily varying with embedding size and layer count, but also influenced by vocabulary size,' the premise that parameter count is held constant is explicitly not satisfied. Note the text itself describes '16, 1' as a model with embedding size 16 and '32, 2' as embedding size 16 per layer and total parameters 32; if 'number of parameters' is actually the per-layer base used in the formula rather than the true total, then total parameters differ hugely between the 1-layer and 2-layer configurations (2× embedding parameters for 2 layers plus additional layer weights). The observed difference in learning speed could then be due to total parameter count or raw embedding width, not an architectural law that depth is irrelevant. The reader flagged exactly this as the weakest assumption; it is load-bearing because every depth-related conclusion (layers add no capacity, wider embeddings are better per parameter, shallow wide models are recommended for edge devices) inherits it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a proof-of-concept study of transformer memorization on structured data derived from the SNOMED medical ontology, using synthetic triplet and sequence datasets with controlled tokenization. The authors train small decoder-only transformers and report that embedding size is the primary determinant of learning speed and memorization capacity, that adding layers does not help (and sometimes hurts), that Softmax activation is more stable than ReLU/GELU/RReLU, and that larger datasets improve final memorization. The paper introduces a pipeline for converting ontologies into memorization benchmarks and applies the Maximum Attainable Capacity (MAC) metric, claimed to correlate with Maximum Library Size from prior work. The main conclusion is that for simple structured memorization, wider embeddings are preferable to deeper architectures.","tokens_in":12131,"tokens_out":4803,"duration_ms":53404,"significance":"If the central claim holds, the paper provides actionable guidance for designing small on-device transformers for domain-specific memorization, a relevant issue for medical and edge applications. The strengths are the reproducible code release, the use of a real-world ontology with clear ground-truth memorization tasks, repeated seeds with reported variability, and a systematic sweep over depth, width, and activation functions. The paper is also honest about its scope as a proof-of-concept and lists limitations. However, the quantitative claim that embedding size, not depth or total parameter count, drives capacity rests on a parameter-equalization procedure that does not actually equalize total parameters, so the main architectural conclusion is not yet established. The capacity metric also relies on a self-cited prior validation rather than independent evidence.","major_comments":[{"comment":"The parameter-equalization formula does not actually keep total model parameters constant. The paper states embedding_size = floor(base_number_of_parameters / n_layers), with base values 16/32/64/128, and claims this ensures fair comparisons. But with a fixed vocabulary, the embedding matrix contributes vocab_size × d_model parameters, so halving d_model when doubling the layer count roughly halves the embedding parameters, and the additional layer adds only its own attention and feed-forward parameters. The paper itself notes that model sizes are 'primarily varying with embedding size and layer count, but also influenced by vocabulary size.' Consequently, the configurations compared in Section 3.3 differ in total parameter count, and the observed advantage of larger embeddings over deeper models could be driven by total parameter count or raw embedding width rather than an architectural law. This is load-bearing for the central claim that 'embedding size is the key factor influencing learning speed' and that depth does not matter. I request the authors report the actual total parameter counts for each configuration and either redesign the comparison to truly equalize total parameters or restrict the claim to what the current design supports, e.g., 'adding layers while keeping embedding size fixed does not improve memorization.'","section":"2.3.1 and 3.3"},{"comment":"The 'final capacity' claims, especially the capacity barrier at 100,000 samples for the two-layer model with embedding size 8, are not supported by evidence that all configurations have converged. The training is stopped at a fixed 500 epochs, and no plateau diagnostics (e.g., flat accuracy or loss curves, or longer training runs) are shown for the smallest embedding configurations. The lower capacity of the 16-parameter two-layer model (85,935 vs. ~88,200 for other configurations) may reflect slower convergence rather than a true capacity limit. Since the paper distinguishes 'learning speed' from 'final capacity,' the final-capacity statements need either longer training for the small-embedding configurations or an explicit demonstration of saturation.","section":"3.3 and Table 3"},{"comment":"The capacity metric MAC is justified solely by a self-cited prior paper (Härmä et al., 2024) asserting a strong correlation between MLS and MAC; no independent benchmark or in-paper validation is provided. Given that 'capacity' is central to the title and conclusions, the authors should either provide a direct validation of MAC on at least a few configurations (e.g., compare MAC with an MLS-style measurement on the same data) or clearly state that MAC is used as a proxy for memorization accuracy on a large training set, not as an independently validated capacity bound.","section":"2.2 and 3.2"}],"minor_comments":[{"comment":"The variable name 'base_number_of_parameters' is misleading, since the formula sets the embedding dimension, not a true parameter count. The text should be rephrased to avoid the impression that total parameters are held constant.","section":"2.3.1"},{"comment":"The caption states that 'number of parameters is a total number for all layers: green –16, blue –32', but these values are the base numbers used in the formula, not the actual total parameter count of the models. This should be corrected to avoid confusion.","section":"Figure 4 caption"},{"comment":"There is a typo: 'trainded' should be 'trained'.","section":"2.2"},{"comment":"The sentence 'a one-layer model with 16 parameters (embedding size is 16, light green)' is incorrect: the model has far more parameters than 16 because the embedding matrix size is vocab_size × 16. The terminology should distinguish the base value from the actual parameter count.","section":"3.3"}],"recommendation":"major_revision","confidential_remarks":"The self-citation to Härmä et al. (2024) for the MAC metric is a potential circularity concern that the editor may want to monitor. The paper is otherwise appropriate for a workshop-level venue, but the parameter-equalization flaw currently weakens the central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful proof-of-concept with a reproducible SNOMED-based memorization benchmark and a solid empirical observation that embedding width, not depth, drives learning speed on these structured memorization tasks. It is not a breakthrough, and it has one load-bearing flaw in how parameter counts are reported, but the core depth finding survives that flaw.\n\nWhat's genuinely new: the pipeline converting SNOMED to tokenized triplet and sequence datasets, the systematic comparison of activation functions on a structured memorization task, and the scale (546 models, 3,100 hours) with code released. The finding that Softmax is more stable than ReLU on this task is a useful counterpoint to Shen et al. 2023.\n\nThe soft spots, in proportion. First, the parameter-equalization formula in Section 2.3.1 (embedding_size = floor(base/n_layers)) does not equalize total parameters when vocabulary is fixed. The embedding matrix is vocab_size * d_model, so halving d_model when doubling layers roughly halves embedding parameters; the added layer does not compensate. The paper even admits model sizes are 'influenced by vocabulary size.' This invalidates the second setup's claim to constant parameter count, and it undercuts the Discussion's 'better memorization per parameter' recommendation.\n\nSecond, the central third-setup comparison -- same embedding size, different layer counts -- is actually not contaminated by the formula issue. Comparing d=16 one-layer to d=16 two-layer shows extra depth/params don't help, which is exactly the claim that 'adding layers without increasing embedding size' is useless. So the main architectural conclusion holds.\n\nThird, the MAC metric rests on a self-cited prior paper at the same lab. The correlation with MLS is asserted, not reproduced here. For a proof-of-concept that's acceptable, but it means the capacity numbers are not independently anchored.\n\nThe Limitations section is honest about missing error-pattern analysis, untested longer sequences, and no sparsity/quantization. Good.\n\nVerdict: worth a serious referee. I would require fixing the parameter accounting and either validating MAC independently or softening the per-parameter claims. The paper is most useful to people building tiny local memorizers for structured domains and to anyone who wants a realistic benchmark for memorization studies.\n\nRecommendation: engage with it in review; don't desk reject.","headline":"A worthwhile proof-of-concept: the SNOMED memorization pipeline and the 'width beats depth' observation are real, but the parameter-equalization formula is wrong and the per-parameter recommendation overreaches.","tokens_in":12714,"tokens_out":4552,"would_cite":true,"duration_ms":43613,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Embedding width drives transformer memorization, not depth.","keywords":["transformer memorization","embedding size","model depth","activation functions","knowledge graph data","Maximum Attainable Capacity","medical ontology","structured sequences"],"falsifier":"Count the total parameters, including the vocabulary embedding matrix, for a one-layer and a two-layer configuration that the paper's formula treats as equal in size. If the totals differ and memorization capacity tracks the total rather than the embedding size, the claim that embedding size is the key factor is not established.","tokens_in":11636,"feed_emoji":"🧠","tokens_out":2445,"duration_ms":26976,"temperature":0.7,"pith_summary":"This paper argues that, for small generative transformers trained to memorize structured medical knowledge, the size of the embedding layer is the main determinant of how fast and how much the model learns. Adding transformer layers without widening the embedding neither speeds up convergence nor improves final memorization, and on simple datasets extra depth can even hurt. A softer activation function, Softmax, is reported to give more stable training and higher memorization capacity than ReLU-style activations. The result matters because it suggests practical design rules for small on-device models that must reliably store domain facts, such as medical assistants on wearables.","feed_headline":"Embedding width drives transformer memorization, not depth","feed_subtitle":"Shallow, wide models memorize medical knowledge better than deep, narrow ones, a new proof-of-concept finds.","key_machinery":"Maximum Attainable Capacity (MAC), a metric that counts the maximum number of training samples a model can memorize when given a large library. It is used to compare architectures by training on datasets derived from a medical ontology, either as static triplets or as longer node-edge sequences, and reporting the number of correctly predicted concepts. The comparison across depths is carried out by a parameter-equalization formula, embedding_size = floor(base_number_of_parameters / n_layers), intended to keep total parameters constant while varying depth and width.","core_discovery":"Using synthetic text datasets built from a medical knowledge graph, the authors trained hundreds of small decoder-only transformers and measured memorization with the Maximum Attainable Capacity (MAC) method. Their central claim is that embedding size is the key factor influencing learning speed and final capacity, while increasing the number of layers without increasing embedding size neither accelerates convergence nor improves capacity. They also report that Softmax activation is more stable and achieves higher capacity than ReLU, GELU, or RReLU, and that models trained on longer graph-traversal sequences reach near-perfect memorization faster than models trained on isolated triplets, despite showing more fluctuation during training.","pith_inferences":["My inference: if embedding width is the true limiting factor, then capacity per parameter could be improved further by compressing or sharing the embedding matrix, since the embedding dominates parameter count with a fixed vocabulary.","My inference: the finding suggests a testable design rule for edge deployment: hold embedding size fixed, prune or remove layers, and measure memorization; the paper's results predict little or no capacity loss.","My inference: the contrast with studies reporting ReLU superiority indicates that activation-function effects are task-dependent; a useful next experiment would vary dataset complexity continuously and trace where the advantage flips.","My inference: the reported capacity barrier at large dataset sizes with very small embeddings suggests an interaction between vocabulary coverage and embedding rank, which could be probed by measuring the effective rank of the embedding matrix during training."],"forward_implications":["If embedding size is the binding factor, then for a fixed parameter budget, designers of small memorization-focused transformers should prefer shallow models with wide embeddings over deep models with narrow embeddings.","Adding layers to a small transformer may slow training and reduce final memorization on simple structured data, so depth should not be assumed beneficial for memory-bound tasks.","Softmax activations appear to provide more stable training and higher capacity than ReLU-style activations on this type of structured memorization task, contrary to some earlier findings on other tasks.","Longer structured sequences, such as graph traversal paths, can be memorized faster and more completely than isolated triplets, suggesting that data structure itself is a lever for capacity.","Larger datasets can lead to higher final capacity but require longer warm-up, so there is a trade-off between dataset size and training time that depends on the target capacity."],"supporting_citations":[{"why":"Introduces the MAC method and demonstrates its correlation with Maximum Library Size, providing the capacity metric used throughout the paper.","marker":"Härmä et al. (2024)"},{"why":"Supplies theoretical memorization capacity bounds for transformers, which the paper's empirical observations on embedding size extend.","marker":"Kim et al. (2023)"},{"why":"Provides optimal memorization capacity results that motivate measuring memorization as a standalone objective.","marker":"Kajitsuka and Sato (2024)"},{"why":"The contrasting claim that ReLU outperforms Softmax in transformers, which the paper's results directly challenge for structured memorization tasks.","marker":"Shen et al. (2023)"},{"why":"Cited as evidence that ReLU-based activations suffer instability in deeper networks, supporting the paper's observed variability.","marker":"Paik and Choi (2023)"},{"why":"Cited alongside Paik and Choi to account for decreased capacity and increased variability when depth grows with ReLU activations.","marker":"Chen and Ge (2024)"},{"why":"Cited to support the claim that many transformer layers are redundant and can be pruned without loss, which the paper's depth findings reinforce.","marker":"He et al. (2024)"}],"fun_headline_variants":["Width beats depth for transformer memorization","Embedding width, not depth, drives memory capacity","Softmax stability aids wide transformers in memory tasks","Wide embeddings trump deep layers in transformer recall"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that the formula used to shrink the embedding size as layers are added actually keeps total model size equal; if it does not, the observed advantage of wider embeddings could just be a total-parameter effect.","fun_headline_variants_meta":{"raw":{"variants":["Width beats depth for transformer memorization","Embedding width, not depth, drives memory capacity","Softmax stability aids wide transformers in memory tasks","Wide embeddings trump deep layers in transformer recall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000175,"raw_usage":{"total_tokens":1208,"prompt_tokens":789,"completion_tokens":419,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":405,"completion_tokens_details":{"reasoning_tokens":361}},"tokens_in":405,"tokens_out":419,"duration_ms":4133,"temperature":1.0,"reasoning_tokens":361,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:48:07.459124+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Count the total parameters, including the vocabulary embedding matrix, for a one-layer and a two-layer configuration that the paper's formula treats as equal in size. If the totals differ and memorization capacity tracks the total rather than the embedding size, the claim that embedding size is the key factor is not established.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies theoretical memorization capacity bounds for transformers, which the paper's empirical observations on embedding size extend."},{"cited_title":"The Disharmony between BN and ReLU Causes Gradient Explosion, but is Offset by the Correlation between Activations","cited_arxiv_id":"2304.11692","evidence_quote":"Cited as evidence that ReLU-based activations suffer instability in deeper networks, supporting the paper's observed variability."}],"review_version":2}