Pith. sign in

REVIEW 4 major objections 3 minor 65 references

Learning Mechanism Underlying NLP Pre-Training and Fine-Tuning

T0 review · 4 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that masked-language pre-training organizes the token vocabulary into small clusters of strong semantic matches, visible when the model makes wrong token predictions, and that this organization, rather than embedding geome

desk verdict A plausible but unproven claim that pre-training groups tokens into semantic clusters; the diagnostic is new, but the lack of a null model and sensitivity analysis makes the central result provisional. read the letter →

arxiv 2509.03407 v1 pith:W4QSCBXB submitted 2025-09-03 cs.CL

Yarden Tzach , Ronit D. Gross , Ella Koresh , Shalom Rosner , Or Shpringer , Tal Halevi , Ido Kanter This is my paper
classification cs.CL
keywords maskedlanguagemodelingtokenconfusionmatrixclusteringpre-trainingfine-tuningBERTaccuracypersingle-nodalperformancesemanticsimilarity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to explain what masked-language pre-training actually learns and why it helps downstream classification. Its central proposal is an order parameter, the average accuracy per token (), and a structural discovery: when a pre-trained BERT-6 model mispredicts a masked token, the wrong answer is usually a strong semantic match, so the token confusion matrix decomposes into many small meaning clusters. This clustering sharpens through the transformer blocks toward the output layer, and fine-tuning accuracy rises along the same blocks. The paper also reports that classification confidence is independent of how well the input tokens were individually learned, because errors preserve meaning. The authors infer that this token-cluster mechanism is a universal learning principle shared with image classifiers.

What carries the argument

The token confusion matrix: a T-number by T-number table counting how often a modified token i is predicted as token j. Each row is normalized by its diagonal, entries above a threshold T_h = 0.05 are set to one, the matrix is symmetrized by keeping only mutual connections, and a percolation algorithm extracts connected components as 'strong match token clusters.' The paper treats these clusters as the observable signature of higher-order language structure. Two supporting measures carry the rest of the argument: the average accuracy per token (<APT>), used as an order parameter for pre-training quality along the transformer blocks, and single-nodal/single-head signal-to-noise analyses used

What would settle it

Run the identical confusion-matrix construction on the same BERT-6 architecture with randomly initialized or untrained embeddings. If percolating the thresholded, symmetrized matrix gives a similar distribution of small semantic clusters (months, directions, ordinals), the symmetry-breaking claim is not specific to pre-training. Also sweep T_h from 0.01 to 0.2 on the trained model: if the small semantic clusters exist only in a narrow threshold window or merge into one giant cluster at neighboring values, the cluster picture is threshold-selected rather than emergent.

Watch

Extended reading notes

Core claim

Pre-training breaks the symmetry among tokens and groups them into finite, small, strong-match token clusters, as inferred from the token confusion matrix. When the model is asked to recover a masked, replaced, or unchanged token and gets it wrong, it picks a token that is usually a close semantic or functional relative: directions cluster together, months cluster together, ordinal numbers chain together, and antonyms pair up. This higher-order language structure emerges even though the training loss only asks for a single correct token. The structure is weak and noisy in the embedding layer's cosine-similarity graph, but becomes sharp and coherent through the transformer blocks, and the sam

Load-bearing premise

Everything rests on the assumption that the small meaning-clusters are learned structure rather than a by-product of how the confusion matrix was thresholded and which rows were kept; random-embedding baselines and threshold sweeps would expose this.

Editorial extensions

If this is right

  • Average accuracy per token can serve as a layer-wise order parameter for pre-training quality, allowing model comparisons without running downstream tasks.
  • Confusion-matrix clusters give a direct way to extract synonym-like word sets from a masked language model, usable for augmentation or robustness tests.
  • Classification confidence can remain high even for inputs made of poorly learned tokens, because the model's errors tend to preserve meaning.
  • Fine-tuning accuracy along transformer blocks is tied to signal-to-noise of individual nodes and attention heads, suggesting that pruning or early stopping could be guided by those statistics.
  • Small pre-training corpora, on the order of tens of thousands of paragraphs, reproduce the qualitative cluster structure, making mechanism studies feasible without large-scale compute.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural test the paper does not run is a threshold sweep plus a random-embedding null model: if small semantic clusters also appear for an untrained model or across a wide range of T_h values, the claim that pre-training creates them would be weakened.
  • The same confusion-matrix clustering could be applied to class labels in image classifiers; if confusable classes form small semantic clusters there, the paper's hinted universality would gain direct evidence rather than analogy.
  • The independence of confidence from average input APT suggests a testable robustness prediction: replacing tokens with same-cluster strong matches should change classifications less than random token substitutions.
  • The paper raises but does not define an entropy of strong-match tokens; one could compare cluster-size distributions across models and corpora as a quantitative measure of language structure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. This paper studies BERT-6 pre-trained on Wikipedia (full and tiny subsets W_S=40k/90k) and fine-tuned on FewRel/DBpedia. It reports (i) per-token accuracy (APT) increases with token frequency and with transformer-block depth; (ii) a confusion-matrix analysis showing that mispredictions are concentrated in small semantic groups, interpreted as symmetry-breaking into 'strong match token clusters'; (iii) that output-layer clusters are more coherent than embedding-layer cosine-similarity clusters; (iv) that fine-tuning accuracy improves along transformer blocks, with single-node/single-head SNR measures increasing; and (v) that FewRel classification confidence is practically independent of average input APT. The paper argues that these findings reveal the mechanism of pre-training and hint at universality with image classification.

Significance. The potential contribution is the use of the MLM confusion matrix as a direct observational probe of token organization. If the cluster claim survives statistical controls, it is a falsifiable and surprising result: the MLM objective, which only asks for the correct token, appears to produce semantic family structure. The paper is also commendable for proposing a concrete order parameter (<APT>), for making block-wise comparisons at a modest computational scale, and for presenting its observations in a transparent, reproducible way. However, the central cluster claim currently rests on post hoc thresholding, row exclusion, and selected examples; the absence of a null model leaves a real risk that the clusters reflect sampling noise rather than learned token semantics. With added controls this could be an interesting paper, but in its present form the main claim is not yet supported at the standard expected for a journal.

major comments (4)
  1. [Section 2.2, Eq. (1), Appendix 4] The cluster analysis is the evidential core of the abstract's 'strong match token clusters' claim, and it depends on three post hoc choices: (i) rows whose diagonal is not maximal are excluded; (ii) each row is normalized by its diagonal; (iii) a fixed T_h=0.05 turns normalized off-diagonals into ones. The paper states that results are 'insensitive' near T_h=0.05 but gives no sweep. More importantly, with W_test=90,000 and 30 repetitions, low-frequency tokens have very small counts, so a single off-diagonal co-occurrence can exceed T_h after normalization (e.g., 1/5=0.2). Sampling noise can therefore produce mutual above-threshold edges and finite percolation clusters like those in Tables 2a-2c. Without a null model in which predicted tokens are shuffled or random embeddings are substituted, and without reporting how many rows were excluded, the conclusion that these clusters reflect lea
  2. [Section 2.2, Figs. 4-5] The semantic interpretation of the clusters is anecdotal. A handful of hand-picked clusters are shown as evidence of 'strong match' tokens, but no quantitative semantic-coherence measure is computed over the full cluster distribution. To support the global claim that pre-training groups tokens into meaning-based families, the authors should compare cluster coherence against random token groupings or use an external lexical resource. The selected examples alone cannot validate the cluster claim.
  3. [Section 2.4, Tables 2 and 3] The comparison between output-block and embedding-layer clusters is confounded by different graph constructions. Output-layer clusters are formed from a thresholded confusion matrix (T_h=0.05), whereas embedding-layer clusters are formed from Top-K (q=2,3,4) on cosine similarity. Different sparsity rules, rather than layer quality, may explain the difference in coherence. The same-method comparison along transformer blocks (Table 2b vs. 2c) is valid, but the embedding-layer comparison should use an analogous pipeline before claiming that the output layer 'considerably' outperforms the embedding layer.
  4. [Section 2.4, Fig. 16] The claim that FineRel confidence is 'practically independent' of APT_Ave is based on visual inspection of the plotted points, with no error bars, no correlation coefficient, and no significance test. Because the bins contain different numbers of inputs, sampling noise may be substantial. Please report confidence intervals and a formal test for a trend before asserting confidence-independence as a finding.
minor comments (3)
  1. [Section 2.4] There are two sections numbered '2.4' (Embedding layer and Confidence versus higher-order language structures). The second should be renumbered, likely as 2.5.
  2. [Appendix 6] 'standard division' should be 'standard deviation'.
  3. [Appendix 4] The sentence 'Rows which had maximal off-diagonal elements were then diluted' is ambiguous and appears inconsistent with the main text's statement that rows whose diagonal was not maximal were excluded. Please clarify.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the paper's main claims are observational analyses of measured confusion matrices and accuracies, not derivations that reduce to their own inputs.

full rationale

The central results—APT increasing with frequency, cluster-size distributions from the thresholded confusion matrix, confidence independence from APT, and block-wise accuracy/SNR trends—are empirical measurements. The clustering pipeline in Section 2.2 defines clusters operationally from the symmetrized, thresholded confusion matrix; while the threshold and row-exclusion choices are post-hoc and could be sensitive to noise, the reported cluster content and sizes are data-dependent outputs rather than fitted parameters used to predict the same quantity. The paper does not fit a parameter to a subset and then claim a closely related prediction; no equation is equivalent to the conclusion by construction. The SNP/SHP and SNR machinery in Section 2.5 is imported from the authors' prior work, but the method is described in the text, the reported tables are new measurements on new tasks, and the trend that SNR increases with block is a descriptive summary of the same data, not a self-citation acting as an unverified load-bearing premise. The confidence-independence explanation appeals to the same clusters, but those clusters are independently exemplified as semantically coherent (months, directions, etc.), so the explanation is not a tautological reduction. The lack of a null model and the qualitative threshold-insensitivity statement are correctness/robustness concerns, not circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claims rest on several hand-chosen thresholds and on domain assumptions about what the confusion matrix and probe layers represent. The free parameters are not derived from theory and are not tested for sensitivity. No new physical or mechanistic entities are postulated; the 'strong match tokens' and the order parameter <APT> are derived measurements from the model's outputs, not independent entities with falsifiable handles.

free parameters (4)
  • Confusion matrix threshold T_h = 0.05 (relative to diagonal)
    Off-diagonal elements above 0.05 of the diagonal are set to 1 in the binary confusion matrix; chosen from the distribution in Fig. 3b, not from theory. Cluster sizes depend on this value. Section 2.2, Fig. 3b.
  • Top-K for cosine-similarity graph = q = 4, 3, 2 (primary results use q=2,3)
    The cosine-similarity matrix is diluted by keeping only the q largest elements per row; q is hand-chosen because the similarity distribution is continuous. Section 2.4, Tables 3a-b.
  • SNP/SHP threshold = 0.6
    Booleans the single-nodal and single-head value matrices at threshold 0.6 before computing clusters and noise; not optimized or justified. Section 2.5, Tables 4-7.
  • Per-layer fine-tuning hyperparameters = Tables 8-10 list different learning rates and L2 for each layer
    Each block's probe accuracy is measured with separately tuned hyperparameters, so block-to-block accuracy differences may reflect optimization rather than representation strength. Appendix 3, Tables 8-10.
assumptions (5)
  • domain assumption The masked-language-model confusion matrix is a valid proxy for token semantic similarity.
    The paper interprets large off-diagonal confusion entries as 'strong match' tokens and builds semantic clusters from them (Section 2.2). This assumes prediction errors carry semantic signal rather than systematic token-frequency or position artifacts.
  • domain assumption The average accuracy per token (<APT>) is a sufficient order parameter for pre-training success.
    Section 2.1 defines <APT> as the order parameter without showing it uniquely determines pre-training quality or correlates with downstream task performance beyond the trends in Tables 4-5.
  • domain assumption DistilBERT (or the custom 6-layer BERT) pre-trained on English Wikipedia is representative of NLP pre-training generally.
    The paper generalizes all conclusions to NLP pre-training and fine-tuning from one small English model (Section 1, 3).
  • ad hoc to paper Symmetric mutual confusion (Eq. 1 product with transpose) is a meaningful basis for token clusters.
    Eq. 1 forces symmetry by keeping only mutual nearest-neighbor confusions; the paper later relaxes this with Top-K but the cluster analysis relies on the symmetric construction.
  • domain assumption A fully connected layer trained on frozen block outputs gives an unbiased measure of block quality.
    Blocks are probed by training a new output layer with per-layer hyperparameters; this assumes any accuracy differences reflect the block's learned representation rather than the probe's optimization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning Mechanism Underlying NLP Pre-Training and Fine-Tuning." pith.science (2026). https://pith.science/paper/W4QSCBXB

@misc{pith2026250903407,
  author       = {Pith},
  title        = {Pith review of: Learning Mechanism Underlying NLP Pre-Training and Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W4QSCBXB}},
  note         = {Machine review of arXiv:2509.03407}
}
read the original abstract

Natural language processing (NLP) enables the understanding and generation of meaningful human language, typically using a pre-trained complex architecture on a large dataset to learn the language and next fine-tune its weights to implement a specific task. Twofold goals are examined; to understand the mechanism underlying successful pre-training and to determine the interplay between the pre-training accuracy and the fine-tuning of classification tasks. The following main results were obtained; the accuracy per token (APT) increased with its appearance frequency in the dataset, and its average over all tokens served as an order parameter to quantify pre-training success, which increased along the transformer blocks. Pre-training broke the symmetry among tokens and grouped them into finite, small, strong match token clusters, as inferred from the presented token confusion matrix. This feature was sharpened along the transformer blocks toward the output layer, enhancing its performance considerably compared with that of the embedding layer. Consequently, higher-order language structures were generated by pre-training, even though the learning cost function was directed solely at identifying a single token. These pre-training findings were reflected by the improved fine-tuning accuracy along the transformer blocks. Additionally, the output label prediction confidence was found to be independent of the average input APT, as the input meaning was preserved since the tokens are replaced primarily by strong match tokens. Finally, although pre-training is commonly absent in image classification tasks, its underlying mechanism is similar to that used in fine-tuning NLP classification tasks, hinting at its universality. The results were based on the BERT-6 architecture pre-trained on the Wikipedia dataset and fine-tuned on the FewRel and DBpedia classification tasks.

Figures

Figures reproduced from arXiv: 2509.03407 by the authors.

Figure 1
Figure 1. APT of each individual token ordered from most popular token to least popular (Grey) pre-trained on the entire Wikipedia dataset on BERT-6 and <APT> (Horizontal dashed-red line). Average APT over 200 non-overlapping groups of consecutive tokens (Orange) [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Average APT over 200 non-overlapping groups of consecutive tokens ordered from most popular tokens to least popular, pre-trained on the entire Wikipedia dataset and tested on 𝑊𝑡𝑒𝑠𝑡 = 90,000 (Blue), and <APT> (Horizontal blue-dashed line). Same as Blue but pre-trained on 𝑊𝑆 = 90,000 (Orange), and <APT> (Horizontal, orange-dashed line). 2.2. Confusion matrix and high-order language structures An insight into the quali… view at source ↗
Figure 3
Figure 3. (a) Distribution of the confusion matrix, 𝑀𝐶𝑜𝑛𝑓𝑢𝑠, non-zero off-diagonal elements. (b) Zoom-in on panel a in the range [0, 0.2], where 𝑇ℎ = 0.05 represents the threshold where the distribution of off-diagonal elements become notably sparser. The product of all pairs of elements with the same indices (𝑖,𝑗) in 𝑀𝐵𝑖𝑛𝑎𝑟𝑦 𝐶𝑜𝑛𝑓𝑢𝑠 and its transposed matrix 𝑀𝑎𝑑𝑗𝑎𝑐𝑒𝑛𝑐𝑦(𝑖,𝑗) = 𝑀𝐵𝑖𝑛𝑎𝑟𝑦 𝐶𝑜𝑛𝑓𝑢𝑠(𝑖,𝑗) × 𝑀𝐵𝑖𝑛𝑎𝑟𝑦 𝐶𝑜𝑛𝑓𝑢𝑠 + (𝑖,𝑗) (1) r… view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Exemplified notable small clusters generated by the 6th block of BERT￾6 architecture pre-trained on the entire Wikipedia and tested on 𝑊𝑡𝑒𝑠𝑡 = 90,000 [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Exemplified notable large clusters generated by the 6th block of BERT￾6 architecture pre-trained on the entire Wikipedia and tested on 𝑊𝑡𝑒𝑠𝑡 = 90,000 [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Distribution of APT for tokens appearing in clusters of size 1 (light blue) with average 0.614 (Vertical dashed-blue line) and for cluster sizes greater than 1 (Orange) with average 0.448 (Vertical dashed-orange line). The cluster sizes were obtained for BERT-6 pre-tra…
Figure 7
Figure 7. Figure 7: Exemplified notable small clusters generated by the 6th block of BERT￾6 architecture pre-trained and tested on 𝑊𝑆 = 𝑊𝑡𝑒𝑠𝑡 = 90,000 [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Exemplified notable large clusters generated by the 6th block of BERT￾6 architecture pre-trained and tested on 𝑊𝑆 = 𝑊𝑡𝑒𝑠𝑡 = 90,000 [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Exemplified notable small clusters generated by the 2nd block of BERT￾6 architecture pre-trained and tested on 𝑊𝑆 = 𝑊𝑡𝑒𝑠𝑡 = 90,000 [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Exemplified notable large clusters generated by the 2nd block of BERT￾6 architecture pre-trained and tested on 𝑊𝑆 = 𝑊𝑡𝑒𝑠𝑡 = 90,000. 2.3 Asymmetric nearest-neighbor connections between tokens The formation of token clusters assumes symmetric connections between tokens.…
Figure 11
Figure 11. Figure 11: Selected tokens and their Top-20 tokens obtained from the confusion matrix, normalized each row by its maximal value (diagonal), using the 6th block of BERT-6 pre-trained on the entire Wikipedia dataset. The number below each token represents its normalized appearance…
Figure 12
Figure 12. Figure 12: Distribution of token–vector cosine-similarity pairs in the embedding layer of pre-trained BERT-6 over the entire Wikipedia dataset. The small rise at 1.0 is attributed to the diagonal elements of the vectors multiplied by themselves. The cluster size distribution for…
Figure 13
Figure 13. Figure 13: Exemplified notable clusters generated by the cosine-similarity matrix of the embedding layer matrix of pre-trained BERT-6 over the entire Wikipedia dataset by applying Top-3 connections for each token [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]
Figure 14
Figure 14. Figure 14: Exemplified notable clusters generated by the cosine-similarity of the embedding layer matrix of pre-trained BERT-6 over the entire Wikipedia dataset by applying Top-2 connections for each token [PITH_FULL_IMAGE:figures/full_fig_p027_14.png]
Figure 15
Figure 15. Figure 15: Selected tokens and their Top-K tokens formed through the cosine￾similarity matrix, using the embedding layer of BERT-6 pre-trained on the entire Wikipedia dataset. The number below each token represents its cosine￾similarity values. The proposed clustering method is …
Figure 16
Figure 16. Figure 16: (a) Number of FewRel inputs within a bin of 𝐴𝑃𝑇𝐴𝑣𝑒, which are correctly (green) and incorrectly (red) classified, and the confidence per bin (blue dots connected by blue-dashed line) as defined in Eq. (3). (b) Similar to panel a but the bins are now grouped FewRel inp…
Figure 17
Figure 17. Figure 17: (a) Distribution of label appearance in the diagonal clusters of the 12 SHP of the 6th transformer block of BERT-6 pre-trained on 𝑊𝑆 = 90,000. (b) Same as in panel a but pre-trained on 𝑊𝑆 = 40,000 [PITH_FULL_IMAGE:figures/full_fig_p036_17.png]
Figure 18
Figure 18. Figure 18: (a) Accuracy (Acc.) for each of the 64 FewRel labels (blue circles) versus its number of appearances in the diagonal clusters of the 12 SHP of the 6 th transformer block of BERT-6 pre-trained on 𝑊𝑆 = 90,000. (b) Same as panel a but pre-trained on 𝑊𝑆 = 40,000. 3. Discu…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

65 extracted references · 50 canonical work pages

  1. [1]

    Jiang, F.F

    Z. Jiang, F.F. Xu, J. Araki, G. Neubig, How can we know what language models know?, Transactions of the Association for Computational Linguistics, 8 (2020) 423-438

    work page 2020

  2. [2]

    Petroni, T

    F. Petroni, T. Rocktäschel, P. Lewis, A. Bakhtin, Y. Wu, A.H. Miller, S. Riedel, Language models as knowledge bases?, arXiv preprint arXiv:1909.01066, (2019)

    arXiv 1909

  3. [4]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, Advances in neural information processing systems, 30 (2017)

    work page 2017

  4. [5]

    Gross, Y

    R.D. Gross, Y. Tzach, T. Halevi, E. Koresh, I. Kanter, Tiny language models, arXiv preprint arXiv:2507.14871, (2025)

    work page arXiv 2025

  5. [6]

    Lu, Full-text federated search in peer-to-peer networks, 2007

    J. Lu, Full-text federated search in peer-to-peer networks, 2007

    work page 2007

  6. [7]

    Chang, X

    Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang, A survey on evaluation of large language models, ACM transactions on intelligent systems and technology, 15 (2024) 1-45

    work page 2024

  7. [8]

    Mielke, Z

    S.J. Mielke, Z. Alyafeai, E. Salesky, C. Raffel, M. Dey, M. Gallé, A. Raja, C. Si, W.Y. Lee, B. Sagot, Between words and characters: A brief history of open-vocabulary modeling and tokenization in NLP, arXiv preprint arXiv:2112.10508, (2021)

    arXiv 2021

  8. [9]

    S. Choo, W. Kim, A study on the evaluation of tokenizer performance in natural language processing, Applied Artificial Intelligence, 37 (2023) 2175112

    work page 2023

Show all 65 references
  1. [10]

    T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, Transformers: State-of-the-art natural language processing, in: Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstratio...

    2020

  2. [11]

    Mesnil, Y

    G. Mesnil, Y. Dauphin, X. Glorot, S. Rifai, Y. Bengio, I. Goodfellow, E. Lavoie, X. Muller, G. Desjardins, D. Warde-Farley, Unsupervised and transfer learning challenge: a deep learning approach, in: Proceedings of ICML Workshop on Unsupervised and Transfer Learning, JMLR Work...

    2012

  3. [12]

    Yosinski, J

    J. Yosinski, J. Clune, Y. Bengio, H. Lipson, How transferable are features in deep neural networks?, Advances in neural information processing systems, 27 (2014)

    2014

  4. [13]

    Lozano-Diez, O

    A. Lozano-Diez, O. Plchot, P. Matejka, J. Gonzalez-Rodriguez, DNN based embeddings for language recognition, in: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2018, pp. 5184-5188

    2018

  5. [14]

    W. Luo, Y. Li, R. Urtasun, R. Zemel, Understanding the effective receptive field in deep convolutional neural networks, Advances in neural information processing systems, 29 (2016)

    2016

  6. [15]

    LeCun, K

    Y. LeCun, K. Kavukcuoglu, C. Farabet, Convolutional networks and applications in vision, in: Proceedings of 2010 IEEE international symposium on circuits and systems, IEEE, 2010, pp. 253-256

    2010

  7. [16]

    Britz, Understanding convolutional neural networks for NLP, Denny’s Blog, (2015)

    D. Britz, Understanding convolutional neural networks for NLP, Denny’s Blog, (2015)

    2015

  8. [17]

    Hassani, S

    A. Hassani, S. Walton, N. Shah, A. Abuduweili, J. Li, H. Shi, Escaping the big data paradigm with compact transformers, arXiv preprint arXiv:2104.05704, (2021)

    2021 arXiv

  9. [18]

    Koresh, R.D

    E. Koresh, R.D. Gross, Y. Meir, Y. Tzach, T. Halevi, I. Kanter, Unified CNNs and transformers underlying learning mechanism reveals multi-head attention modus vivendi, arXiv preprint arXiv:2501.12900, (2025)

    2025 arXiv

  10. [19]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint arXiv:2010.11929, (2020)

    2010 arXiv

  11. [20]

    Gross, T

    R.D. Gross, T. Halevi, E. Koresh, Y. Tzach, I. Kanter, Low-latency vision transformers via large-scale multi-head attention, arXiv preprint arXiv:2506.23832, (2025)

    2025 arXiv

  12. [21]

    Jawahar, B

    G. Jawahar, B. Sagot, D. Seddah, What does BERT learn about the structure of language?, in: ACL 2019-57th Annual Meeting of the Association for Computational Linguistics, 2019

    2019

  13. [22]

    Clark, U

    K. Clark, U. Khandelwal, O. Levy, C.D. Manning, What does bert look at? an analysis of bert's attention, arXiv preprint arXiv:1906.04341, (2019)

    1906 arXiv

  14. [23]

    Rogers, O

    A. Rogers, O. Kovaleva, A. Rumshisky, A primer in BERTology: What we know about how BERT works, Transactions of the association for computational linguistics, 8 (2021) 842-866

    2021

  15. [24]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, v...

    2019

  16. [25]

    Hoshen, R

    J. Hoshen, R. Kopelman, Percolation and cluster distribution. I. Cluster multiple labeling technique and critical concentration algorithm, Phys Rev B, 14 (1976) 3438

    1976

  17. [26]

    Havlin, R

    S. Havlin, R. Nossal, Topological properties of percolation clusters, Journal of Physics A: Mathematical and General, 17 (1984) L427

    1984

  18. [27]

    Z. Yin, Y. Shen, On the dimensionality of word embedding, Advances in neural information processing systems, 31 (2018)

    2018

  19. [28]

    S. Wang, W. Zhou, C. Jiang, A survey of word embeddings based on deep learning, Computing, 102 (2020) 717-740

    2020

  20. [29]

    Ethayarajh, How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings, arXiv preprint arXiv:1909.00512, (2019)

    K. Ethayarajh, How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings, arXiv preprint arXiv:1909.00512, (2019)

    1909 arXiv

  21. [30]

    Mikolov, K

    T. Mikolov, K. Chen, G. Corrado, J. Dean, Efficient estimation of word representations in vector space, arXiv preprint arXiv:1301.3781, (2013)

    2013 arXiv

  22. [31]

    Radovanovic, A

    M. Radovanovic, A. Nanopoulos, M. Ivanovic, Hubs in space: Popular nearest neighbors in high-dimensional data, Journal of Machine Learning Research, 11 (2010) 2487-2531

    2010

  23. [32]

    J. Mu, S. Bhat, P. Viswanath, All-but-the-top: Simple and effective postprocessing for word representations, arXiv preprint arXiv:1702.01417, (2017)

    2017 arXiv

  24. [33]

    Conneau, G

    A. Conneau, G. Lample, M.A. Ranzato, L. Denoyer, H. Jégou, Word translation without parallel data, arXiv preprint arXiv:1710.04087, (2017)

    2017 arXiv

  25. [34]

    Dhillon, D.S

    I.S. Dhillon, D.S. Modha, Concept decompositions for large sparse text data using clustering, Machine learning, 42 (2001) 143-175

    2001

  26. [35]

    Banerjee, I.S

    A. Banerjee, I.S. Dhillon, J. Ghosh, S. Sra, G. Ridgeway, Clustering on the Unit Hypersphere using von Mises-Fisher Distributions, Journal of Machine Learning Research, 6 (2005)

    2005

  27. [36]

    McInnes, J

    L. McInnes, J. Healy, S. Astels, hdbscan: Hierarchical density based clustering, J. Open Source Softw., 2 (2017) 205

    2017

  28. [37]

    Pratap, A

    R. Pratap, A. Deshmukh, P. Nair, T. Dutt, A Faster Sampling Algorithm for Spherical $k$- means, in: Z. Jun, T. Ichiro (Eds.) Proceedings of The 10th Asian Conference on Machine Learning, PMLR, Proceedings of Machine Learning Research, 2018, pp. 343--358

    2018

  29. [38]

    Faraki, X

    M. Faraki, X. Yu, Y.-H. Tsai, Y. Suh, M. Chandraker, Cross-domain similarity learning for face recognition in unseen domains, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 15292-15301

    2021

  30. [39]

    Tzach, Y

    Y. Tzach, Y. Meir, R.D. Gross, O. Tevet, E. Koresh, I. Kanter, Advanced deep architecture pruning using single-filter performance, Phys Rev E, 111 (2025) 065307

    2025

  31. [40]

    Y. Meir, Y. Tzach, S. Hodassman, O. Tevet, I. Kanter, Towards a universal mechanism for successful deep learning, Sci Rep-Uk, 14 (2024) 5881

    2024

  32. [41]

    Tzach, Y

    Y. Tzach, Y. Meir, O. Tevet, R.D. Gross, S. Hodassman, R. Vardi, I. Kanter, The mechanism underlying successful deep learning, arXiv preprint arXiv:2305.18078, (2023)

    2023 arXiv

  33. [42]

    Conneau, H

    A. Conneau, H. Schwenk, L. Barrault, Y. Lecun, Very deep convolutional networks for text classification, arXiv preprint arXiv:1606.01781, (2016)

    2016 arXiv

  34. [43]

    Krizhevsky, G

    A. Krizhevsky, G. Hinton, Learning multiple layers of features from tiny images, (2009)

    2009

  35. [44]

    Y. Meir, O. Tevet, Y. Tzach, S. Hodassman, R.D. Gross, I. Kanter, Efficient shallow learning as an alternative to deep learning, Sci Rep-Uk, 13 (2023) 5423

    2023

  36. [45]

    Koresh, T

    E. Koresh, T. Halevi, Y. Meir, D. Dilmoney, T. Dror, R. Gross, O. Tevet, S. Hodassman, I. Kanter, Scaling in Deep and Shallow Learning Architectures, Physica A: Statistical Mechanics and its Applications, 646 (2024) 129909

    2024

  37. [46]

    Tevet, R.D

    O. Tevet, R.D. Gross, S. Hodassman, T. Rogachevsky, Y. Tzach, Y. Meir, I. Kanter, Efficient shallow learning mechanism as an alternative to deep learning, Physica A: Statistical Mechanics and its Applications, 635 (2024) 129513

    2024

  38. [47]

    Y. Meir, Y. Tzach, S. Hodassman, O. Tevet, I. Kanter, Universality of underlying mechanism for successful deep learning, arXiv e-prints, (2023) arXiv: 2309.07537

    2023 arXiv

  39. [48]

    Y. Meir, S. Sardi, S. Hodassman, K. Kisos, I. Ben-Noam, A. Goldental, I. Kanter, Power-law scaling to assist with key challenges in artificial intelligence, Scientific reports, 10 (2020) 19628

    2020

  40. [49]

    Y. Meir, S. Sardi, S. Hodassman, K. Kisos, I. Ben-Noam, A. Goldental, I. Kanter, Power-law scaling to assist with key challenges in artificial intelligence, Scientific Reports, 10 (2020) 1-7

    2020

  41. [50]

    Clark, K.A

    J.W. Clark, K.A. Gernoth, S. Dittmar, M. Ristig, Higher-order probabilistic perceptrons as Bayesian inference engines, Phys Rev E, 59 (1999) 6161

    1999

  42. [51]

    Ghosh, Y

    J. Ghosh, Y. Shin, Efficient higher-order neural networks for classification and function approximation, International Journal of Neural Systems, 3 (1992) 323-350

    1992

  43. [52]

    Durbin, D.E

    R. Durbin, D.E. Rumelhart, Product units: A computationally powerful and biologically plausible extension to backpropagation networks, Neural Comput, 1 (1989) 133-142

    1989

  44. [53]

    Hodassman, R

    S. Hodassman, R. Vardi, Y. Tugendhaft, A. Goldental, I. Kanter, Efficient dendritic learning as an alternative to synaptic plasticity hypothesis, Scientific Reports, 12 (2022) 6571

    2022

  45. [54]

    Vardi, Y

    R. Vardi, Y. Tugendhaft, S. Sardi, I. Kanter, Significant anisotropic neuronal refractory period plasticity, Europhysics Letters, 134 (2021) 60007

    2021

  46. [55]

    Sardi, R

    S. Sardi, R. Vardi, A. Goldental, A. Sheinin, H. Uzan, I. Kanter, Adaptive nodes enrich nonlinear cooperative learning beyond traditional adaptation by links, Scientific reports, 8 (2018) 5100

    2018

  47. [56]

    Sardi, R

    S. Sardi, R. Vardi, A. Goldental, Y. Tugendhaft, H. Uzan, I. Kanter, Dendritic learning as a paradigm shift in brain learning, ACS chemical neuroscience, 9 (2018) 1230-1232

    2018

  48. [57]

    Lehmann, R

    J. Lehmann, R. Isele, M. Jakob, A. Jentzsch, D. Kontokostas, P.N. Mendes, S. Hellmann, M. Morsey, P. Van Kleef, S. Auer, Dbpedia–a large-scale, multilingual knowledge base extracted from wikipedia, Semantic web, 6 (2015) 167-195

    2015

  49. [58]

    X. Han, H. Zhu, P. Yu, Z. Wang, Y. Yao, Z. Liu, M. Sun, FewRel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation, arXiv preprint arXiv:1810.10147, (2018)

    2018 arXiv

  50. [59]

    Paszke, Pytorch: An imperative style, high-performance deep learning library, arXiv preprint arXiv:1912.01703, (2019)

    A. Paszke, Pytorch: An imperative style, high-performance deep learning library, arXiv preprint arXiv:1912.01703, (2019)

    1912 arXiv

  51. [60]

    Schmidhuber, Deep learning in neural networks: An overview, Neural networks, 61 (2015) 85-117

    J. Schmidhuber, Deep learning in neural networks: An overview, Neural networks, 61 (2015) 85-117

    2015

  52. [61]

    K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778

    2016

  53. [62]

    Loshchilov, F

    I. Loshchilov, F. Hutter, Decoupled weight decay regularization, arXiv preprint arXiv:1711.05101, (2017)

    2017 arXiv

  54. [63]

    Cortes, M

    C. Cortes, M. Mohri, A. Rostamizadeh, L2 regularization for learning kernels, arXiv preprint arXiv:1205.2653, (2012)

    2012 arXiv

  55. [64]

    Goyal, P

    P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, K. He, Accurate, large minibatch sgd: Training imagenet in 1 hour, arXiv preprint arXiv:1706.02677, (2017)

    2017 arXiv

  56. [65]

    V. Sanh, L. Debut, J. Chaumond, T. Wolf, DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, arXiv preprint arXiv:1910.01108, (2019)

    1910 arXiv

  57. [66]

    Howard, S

    J. Howard, S. Ruder, Universal language model fine-tuning for text classification, arXiv preprint arXiv:1801.06146, (2018)

    2018 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.