Shrink the longest: improving latent space isotropy with symplicial geometry

Eduard Klyshinsky; Olesya Karpik; Sergei Kudriashov

REVIEW 5 major objections 5 minor 25 references

Shrink the longest: improving latent space isotropy with symplicial geometry

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A persistent-entropy regularizer derived from the Vietoris-Rips filtration of [CLS] embeddings reduces latent-space anisotropy and can improve GLUE scores during fine-tuning of transformer models, with no retraining or inference overhead.

desk verdict The entropy loss ignores directions, so the central claim that it improves isotropy is mathematically unfounded, and the empirical evidence is too thin to rescue it. read the letter →

arxiv 2501.05502 v1 pith:76MGTBAH submitted 2025-01-09 cs.LG

Sergei Kudriashov , Olesya Karpik , Eduard Klyshinsky This is my paper · ORCID

classification cs.LG

keywords latentspaceisotropyrepresentationdegenerationpersistenthomologyentropytopologicaldataanalysisBERTfine-tuningregularizationVietoris-Ripsfiltration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Transformer embeddings tend to collapse into a narrow cone, a problem known as representation degeneration. This paper proposes a regularization loss that counteracts that collapse during fine-tuning: it maximizes the persistent entropy of selected barcodes obtained from a Vietoris-Rips filtration of the [CLS] embeddings. The authors show that, paired with a topological feature-selection step, the loss lowers the anisotropy of the embedding matrix on BERT and RoBERTa fine-tuned on MRPC and CoLA, and it improves or maintains downstream accuracy. The significance is that the method works on the existing geometric structure of the latent space, so it requires no reparametrization, no retraining, and no extra cost at inference time. If the claim holds, it offers a cheap, model-agnostic way to improve representation quality in low-data fine-tuning scenarios.

What carries the argument

The central object is the persistent entropy of a 0-dimensional Vietoris-Rips barcode, defined as the Shannon entropy of the normalized edge lengths of the minimum spanning tree built on the embedding points. The machinery works in three steps: (1) compute the barcode from a batch of [CLS] embeddings, (2) apply the feature-selection procedure of Atienza et al. to keep only the longest bar and the bars that are not classified as noise, and (3) add the negative persistent entropy of the selected bars to the cross-entropy loss. This term is differentiable through the bar lengths via the differential calculus of persistence barcodes, so gradient descent can maximize it. The paper's key move is to use this topological entropy as a proxy for geometric isotropy: pushing all selected bar lengths toward equality empirically flattens the spectrum of the embedding covariance.

What would settle it

Track persistent entropy and top singular-value share across training: if they do not move together, or if random bar selection yields the same anisotropy drop, the claimed mechanism is not the cause.

Watch

Extended reading notes

Core claim

The central claim is that an entropy regularization term computed on the 0-dimensional persistent homology of contextual embeddings reduces the concentration of variance in the top singular directions of the embedding matrix. The authors implement this by taking the [CLS] token from the last hidden layer, reducing the Vietoris-Rips filtration to a minimum spanning tree, selecting 'topological features' with the algorithm of Atienza et al., and adding the negative persistent entropy of the selected bar lengths to the classification loss. They apply the loss separately per class label to preserve the cluster structure of the latent space. In experiments on BERT and RoBERTa fine-tuned on MRPC and CoLA, the first anisotropy component drops by about 0.1–0.2 and the centered anisotropy by up to 0.2, while BERT's CoLA accuracy rises from 0.575 to 0.588. The paper also reports that omitting feature selection removes most of the benefit, which supports the claim that the method exploits existing cluster geometry rather than reparametrizing it.

Load-bearing premise

The load-bearing premise is that maximizing the entropy of the selected tree-edge lengths causes the variance of the embedding matrix to spread away from the top singular directions; the paper offers empirical correlation but no derivation connecting these two quantities, so if that link fails outside the tested settings the method loses its claimed effect.

Editorial extensions

If this is right

The regularization loss can be grafted onto any fine-tuning objective without retraining the base model and without adding inference overhead.
Feature selection is essential: using all bar lengths fails to lower anisotropy and can leave downstream performance flat, whereas the selected-barcode version raises CoLA accuracy for BERT.
The method is model-agnostic, since it only requires a point cloud of embeddings, so it should transfer to other transformer architectures and non-text modalities.
Anisotropy improvements appear on the top singular components and on the centered covariance matrix, meaning the effect is not just a rescaling of embedding norms.
Because the loss is applied per class label, it preserves the cluster structure that prior work identified as useful, avoiding the damage that reparametrization-based methods can cause.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An untested but natural extension is to apply the same loss during pretraining rather than only fine-tuning; if the entropy-anisotropy link holds throughout training, the cone could be prevented from forming in the first place.
The group-wise application suggests a possible use in class-imbalanced or multi-task fine-tuning, where preserving inter-class separation while flattening within-class directions could be even more valuable than in the balanced case.
A formal tie between the MST edge-length distribution and the singular-value spectrum of the embedding Gram matrix is missing; if such a bound exists, it would predict precisely how much anisotropy reduction is achievable for a given barcode.
The same topological regularizer could be tested on other point-cloud representations such as graph node embeddings or vision transformer patch embeddings, where the cone phenomenon has also been observed.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is the first in its cited set to use persistent entropy of selected barcodes as an anisotropy regularizer, and the idea of exploiting cluster geometry during fine-tuning is worth a look. The feature-selection step borrowed from Atienza et al. is nontrivial, and the authors provide code, which is commendable. Table 3 shows consistent anisotropy drops across all four fine-tuning runs, and the centered anisotropy numbers move in the same direction. That is not nothing.

But the load-bearing assumption does not survive contact with the math. Persistent entropy of the 0-dimensional barcode depends only on the multiset of MST edge lengths. It carries no information about the directions of those edges. A configuration of N equally spaced points on a line in R^D has all N-1 MST edges equal, so its persistent entropy is maximal (log(N-1)), while its covariance is rank-one and anisotropy_1 approaches 1. At that maximum the entropy gradient is zero, so the loss supplies no force to spread the points into an isotropic cloud. In other words, the loss's optima include maximally anisotropic configurations. The claimed causal link between entropy maximization and isotropy is not merely underived; it is false in general. The observed anisotropy drops in the four BERT/RoBERTa runs must therefore come from an interaction between the entropy term, the class-wise splitting, and the cross-entropy loss, not from the entropy term doing what the paper says it does.

The empirical reporting is also weaker than the text suggests. Table 3 lists no standard deviations or significance tests, even though the authors ran five seeds. Downstream metrics in Table 2 are mixed: BERT-MRPC gets slightly worse, and all other differences are within one standard deviation of the baseline. Implementation details that matter for reproducibility—the entropy loss weight, gradient flow through the discrete selection step, and the persistent-homology library—are absent.

This paper is not ready for peer review. A serious referee would need a real theoretical connection between persistent entropy and the spectrum of the embedding covariance, or a much larger empirical study with error bars and ablations that isolate the mechanism. As it stands, the central argument is a fallacy, and the empirical tail does not wag the dog. I would not cite it in its current form, though it could be useful as a reading-group example of how TDA-inspired losses can optimize the wrong quantity.

Referee Report

5 major / 5 minor

Summary. The paper proposes an entropy-based regularization term for fine-tuning transformer models, computed from the persistent entropy of 0-dimensional Vietoris-Rips barcodes (i.e., MST edge lengths) of [CLS] embeddings within each class, with a topological feature-selection step adapted from Atienza et al. The authors fine-tune BERT-base and RoBERTa-base on MRPC and COLA, and report that the regularized models have lower anisotropy and slightly better GLUE scores than the unregularized baselines. The central claim is that entropy regularization with feature selection improves latent-space isotropy and generalization without retraining and without inference overhead.

Significance. If the empirical claims were solid, the method would be a lightweight and model-agnostic alternative to existing isotropy-calibration techniques, and the use of differentiable persistent entropy as a regularizer would be a novel contribution to representation-learning. The paper is also distinctive in trying to preserve, rather than destroy, the cluster structure of the embedding space. However, the manuscript does not establish the theoretical link between persistent entropy and anisotropy, and the reported evidence is weakened by missing statistical tests and an unreported hyperparameter. The idea is promising but the current support is insufficient for acceptance.

major comments (5)

[Section 3, Eq. (1)-(2) and Section 5 claim] Persistent entropy of the 0-dimensional barcode is a function only of the multiset of MST edge lengths, not of the spatial directions of those edges. A configuration of N points equally spaced along a line in R^D has all N-1 MST edges equal, so its persistent entropy attains the theoretical maximum log(N-1), while its covariance matrix is rank-one and anisotropy_1 tends to 1. Moreover, for such a collinear configuration the gradient of the entropy with respect to small perpendicular displacements vanishes, so the loss supplies no force that would spread the points into an isotropic cloud. The manuscript therefore does not establish the claimed causal mechanism by which maximizing persistent entropy improves isotropy; it only reports correlations in four fine-tuning runs. The authors should either provide a theoretical justification specific to their class-wise, batched setting or explicitly reframe the claim as an empirical heuristic and support it with substantially more evidence.
[Section 4, objective and Table 1] The weight of the entropy regularization term relative to the cross-entropy loss is not specified. The final objective L = LCE - sum_i Lent * 1[class=i] has no coefficient for Lent, yet the scale of Lent depends on the number of selected bars and on the distances between embeddings, which vary across tasks and batches. Without reporting this hyperparameter (or a schedule for it), the experiments are not reproducible, and the observed effects could be driven by an arbitrary scaling of the two loss terms.
[Section 5, Tables 2 and 3] Table 3 reports anisotropy values without standard deviations or significance tests, despite the statement in Section 4 that each experiment is repeated 5 times. The claim of "significantly lowering" anisotropy is therefore not supported statistically. The downstream results in Table 2 are also mixed: BERT-MRPC degrades from 0.892 ± 0.002 to 0.889 ± 0.001, and the gains on RoBERTa-MRPC and RoBERTa-COLA are within one standard deviation of the baseline. The paper should report per-run anisotropy values and paired significance tests (e.g., bootstrap or paired t-tests) before asserting consistent improvements.
[Section 3, Algorithm 1] The feature-selection algorithm is described in a way that is hard to follow and appears to contain ambiguities or typographical errors: the sorting condition "li ≥ lj ≥ ln−1 = r" is unclear, the recursive call changes n′ without explicitly describing how L′0 is updated, and the quantity Q becomes undefined when α = r/T = 1, which is precisely the equal-spacing regime relevant to the counterexample above. Since the authors attribute a substantial part of the improvement to the selection step in Section 5, a precise, self-contained description is necessary for reproduction.
[Section 5, concluding paragraph] The sentence "an addition of entropy regularization paired with topological feature selection leads to improvements both in terms of isotropy and generalization" overstates the evidence in Tables 2 and 3, given the missing statistical support and the mixed downstream numbers. The conclusion should be softened to reflect the actual empirical scope, or the authors should provide additional experiments that justify the strong claim.

minor comments (5)

[Abstract] The code link in the abstract is misspelled ("avaliable").
[Section 4 vs Table 1] The text says "the exact parameters used for fine-tuning are shown in Table 1," but Table 1 omits the entropy-loss weight, so the word "exact" is inaccurate.
[Section 4, equations (3)-(4)] The sign convention is confusing: Lent is defined as the positive entropy, then the text says "we take it with a minus sign," and the final objective subtracts Lent. Please clarify the convention explicitly so the reader does not have to infer that the objective maximizes entropy rather than minimizing it.
[Figures 1 and 2] The captions do not identify which curve corresponds to "No reg," "Ent. loss (selected barcodes)," and "Ent. loss (all barcodes)." Add a legend or explicit line-style labels.
[References] Reference [24] appears to duplicate reference [14] (both are the RoBERTa paper). Please consolidate and use consistent spelling of "RoBERTa" throughout (e.g., "RoBERTA-MRPC" vs "RoBERTa-COLA").

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the optimized objective (persistent entropy of selected MST bar lengths) and the evaluation metric (singular-value anisotropy) are distinct functions, with no fitted constants or self-citations bridging them.

full rationale

The paper's derivation chain is self-contained with respect to circularity. The regularization loss Lent is defined as the negative persistent entropy of selected Vietoris-Rips 0-dimensional barcode lengths (i.e., MST edge lengths), while the evaluation metric is anisotropyk(X) = sigma_k^2 / sum_i sigma_i^2 of the embedding matrix. These are mathematically independent quantities: the entropy depends only on the multiset of bar lengths, whereas the anisotropy depends on the singular value decomposition of the embedded point cloud. No parameter is fitted to the anisotropy values, and the method does not co-opt the evaluation metric as its training signal. The feature-selection subroutine is imported from Atienza et al. (an external citation with no author overlap), so there is no self-citation load-bearing chain. The central claim is an empirical one supported by Tables 2 and 3; the absence of error bars on anisotropy and the lack of significance testing are reporting weaknesses, and the mechanistic gap between entropy maximization and isotropy (e.g., collinear equally spaced points maximize entropy while remaining anisotropic) is a correctness or validity concern, not a circularity. Under the stated rules, concerns about mechanism or external validity do not constitute circularity. Therefore the circularity score is 0.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claim rests on prior TDA results and two unvalidated domain assumptions: the causal link between bar-length entropy and singular-value anisotropy, and the applicability of the Atienza feature/noise selection to contextual embeddings. There are no fitted free parameters except the implicit unitary weight of the entropy loss. No new entities are introduced.

free parameters (1)

entropy loss coefficient (implicit weight) = 1 (not stated)
The final objective L = L_CE - sum L_ent contains no explicit weighting hyperparameter; results depend on this implicit scale and no sensitivity analysis is given.

assumptions (5)

ad hoc to paper The longest bar T is always a topological feature and the tiniest bar r is noise.
Stated as 'crucial for the algorithm' in Section 3 and inherited from Atienza et al.; applied to contextual embedding barcodes without validation.
domain assumption Maximizing persistent entropy of selected bars reduces the singular-value anisotropy of the embedding matrix.
No derivation is given; Section 5 provides only empirical correlation.
domain assumption The [CLS] token from the last hidden layer is a sufficient representative of the embedding geometry for the loss and anisotropy measure.
Section 4 collects [CLS] embeddings; no analysis of other tokens or layers is provided.
domain assumption Applying the entropy loss separately to each class label preserves class structure and is appropriate for fine-tuning.
Section 4 justifies this for classification but provides no ablation against applying the loss to the whole space during fine-tuning.
standard math Persistent homology of Vietoris-Rips filtration and its differentiability are valid as background.
The paper relies on standard results in topological data analysis, citing Leygonie et al. for differentiability.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Shrink the longest: improving latent space isotropy with symplicial geometry." pith.science (2026). https://pith.science/paper/76MGTBAH

@misc{pith2026250105502,
  author       = {Pith},
  title        = {Pith review of: Shrink the longest: improving latent space isotropy with symplicial geometry},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/76MGTBAH}},
  note         = {Machine review of arXiv:2501.05502}
}

read the original abstract

Although transformer-based models have been dominating the field of deep learning, various studies of their embedding space have shown that they suffer from "representation degeneration problem": embeddings tend to be distributed in a narrow cone, making the latent space highly anisotropic. Increasing the isotropy has shown to improve performance in downstream tasks both in static and contextual language models. However, most of approaches either add inference overhead or require substantial amount of data for model reparametrization. We propose a novel regularization technique based on simplicial geometry to improve the isotropy of latent representations. The core idea of our method is based on maximizing the persistent entropy of barcodes obtained using Vietoris-Rips filtration from contextual embeddings in the underlying latent space. We demonstrate that the method leads to an increase in downstream performance while significantly lowering the anisotropy during fine-tuning by exploiting existing geometric structures instead of reparametrization.

Figures

Figures reproduced from arXiv: 2501.05502 by the authors.

**Figure 2.** MRPC validation metrics of RoBERTa models during fine-tuning [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 22 canonical work pages

[1]

Persistent entropy for separating topo- logical features from noise in Vietoris-rips complexes

Atienza, N., Gonzalez-Diaz, R., & Rucco, M. Persistent entropy for separating topo- logical features from noise in Vietoris-rips complexes. Journal of Intelligent Infor- mation Systems, 52, 637-655. (2019)

work page 2019
[2]

Cai, X., Huang, J., Bian, Y.-L., & Church K. W. Isotropy in the Contextual Em- bedding Space: Clusters and Manifolds. In International Conference on Learning Representations. (2021) 10 Sergej Kudrjashov Olesya Karpik Eduard Klyshinsky

work page 2021
[3]

J., & Downey, D

Demeter, D., Kimmel, G. J., & Downey, D. Stolen Probability: A Structural Weak- ness of Neural Language Models. Annual Meeting of the Association for Computa- tional Linguistics. (2020)

work page 2020
[4]

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. North American Chapter of the Association for Computational Linguistics. (2019)

work page 2019
[5]

On Isotropy Calibration of Transformer Models

Ding, Y., Martinkus, K., Pascual, D., Clematide, S., & Wattenhofer, R. On Isotropy Calibration of Transformer Models. In Proceedings of the Third Workshop on In- sights from Negative Results in NLP (pp. 1–9). Association for Computational Lin- guistics. (2022)

work page 2022
[6]

How Contextual are Contextualized Word Representations? Com- paring the Geometry of BERT, ELMo, and GPT-2 Embeddings

Ethayarajh, K. How Contextual are Contextualized Word Representations? Com- paring the Geometry of BERT, ELMo, and GPT-2 Embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- ĲCNLP) (pp. 55–65). Association for Computational Li...

work page 2019
[7]

Representation Degenera- tion Problem in Training Natural Language Generation Models

Gao, J., He D., Tan, X., Qin, T., Wang, L., & Liu, T.-Y. Representation Degenera- tion Problem in Training Natural Language Generation Models. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019

work page 2019
[8]

Anisotropy Is Inherent to Self- Attention in Transformers

Godey, N., Villemonte de la Clergerie, E., & Sagot, B. Anisotropy Is Inherent to Self- Attention in Transformers. Conference of the European Chapter of the Association for Computational Linguistics. (2024)

work page 2024

Show all 25 references

[9]

WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach

Huang, J., Tang, D., Zhong, W., Lu, S., Shou, L., Gong, M., Jiang, D., & Duan, D. WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach. Con- ference on Empirical Methods in Natural Language Processing. (2021)

2021
[10]

P., & Ba J

Kingma, D. P., & Ba J. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv: 1412.6980. (2014)

2014 arXiv
[11]

A Framework for Differential Calculus on Persistence Barcodes

Leygonie, J., Oudot, S., & Tillmann, U. A Framework for Differential Calculus on Persistence Barcodes. Foundations of Computational Mathematics, 22, 1069-1131. (2021)

2021
[12]

Li, B., Zhou, H., He, J., Wang, M., Yang, Y., & Li, L. (2020). On the Sentence Em- beddings from Pre-trained Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 9119-9130). Association for Computational Linguistics

2020
[13]

G., Cao, R., Zheng, J., Ren, J., & Gao, L

Liang, Y. G., Cao, R., Zheng, J., Ren, J., & Gao, L. Learning to Remove: Towards Isotropic Pre-trained BERT Embedding. In International Conference on Artificial Neural Networks. (2021)

2021
[14]

RoBERTa: A Robustly Optimized BERT Pretrain- ing Approach

Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. RoBERTa: A Robustly Optimized BERT Pretrain- ing Approach. arXiv preprint arXiv: 1907.11692. (2019)

2019 arXiv
[15]

Moor, M., Horn, M., Rieck, B., & Borgwardt, K. (2020). Topological Autoen- coders. In Proceedings of the 37th International Conference on Machine Learning (pp. 7045–7054). PMLR

2020
[16]

(2017) Using the Output Embedding to Improve Language Mod- els In Proc

Press O., Wolf L. (2017) Using the Output Embedding to Improve Language Mod- els In Proc. of the 15th Conf. of the European Chapter of the Association for Com- putational Linguistics: Volume 2, Short Papers, Valencia, Spain, ACL, 157–163

2017
[17]

Rajaee, S., & Pilehvar, M. (2021). How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropy. In Findings of the Association for Computational Linguistics: EMNLP 2021 (pp. 3042-3049). Association for Compu- tational Linguistics. Shrink the longest: ...

2021
[18]

Razzhigaev, A., Mikhalchuk, M., Goncharova, E., Oseledets, I., Dimitrov, D., & Kuznetsov, A. (2024). The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models. In Findings of the Association for Computational Linguistics: EACL 2024 pp. 868–874

2024
[19]

Learning topology-preserving data representations

Trofimov, I., Cherniavskii, D., Tulchinskii, E., Balabin, N, Burnaev, E., & Baran- nikov, S. Learning topology-preserving data representations. In The Eleventh In- ternational Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023

2023
[20]

NN., Kaiser, L., & Polosukhin, I

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. NN., Kaiser, L., & Polosukhin, I. Attention Is All You Need. NEURIPS, 2017

2017
[21]

Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understand- ing. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP (pp. 35...

2018
[22]

Improving Neural Language Generation with Spectrum Control

Wang, L., Huang, J., Huang, K., Hu, Z., Wang, G., & Gu, Q. Improving Neural Language Generation with Spectrum Control. In Proc. of Int. Conf. on Learning Representations, 2020

2020
[23]

Understanding and Improving Layer Normalization

Xu, J., Sun, X., Zhang, Z., Zhao, G., & Lin, J. Understanding and Improving Layer Normalization. arXiv preprint arXiv: 1911.07013. (2019)

2019 arXiv
[24]

Yinhan L., Myle O., Naman G., Jingfei Du, Mandar J., Danqi C., Omer L., Mike L., Luke Z., & Stoyanov V. (2019). RoBERTa: A Robustly Optimized BERT Pre- training Approach

2019
[25]

Zhang, L., Buntine, W., & Shareghi, E. (2022). On the Effect of Isotropy on VAE Representations of Text. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers) (pp. 694–701). Association for Computational Linguistics

2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Persistent entropy for separating topo- logical features from noise in Vietoris-rips complexes

Atienza, N., Gonzalez-Diaz, R., & Rucco, M. Persistent entropy for separating topo- logical features from noise in Vietoris-rips complexes. Journal of Intelligent Infor- mation Systems, 52, 637-655. (2019)

work page 2019

[2] [2]

Cai, X., Huang, J., Bian, Y.-L., & Church K. W. Isotropy in the Contextual Em- bedding Space: Clusters and Manifolds. In International Conference on Learning Representations. (2021) 10 Sergej Kudrjashov Olesya Karpik Eduard Klyshinsky

work page 2021

[3] [3]

J., & Downey, D

Demeter, D., Kimmel, G. J., & Downey, D. Stolen Probability: A Structural Weak- ness of Neural Language Models. Annual Meeting of the Association for Computa- tional Linguistics. (2020)

work page 2020

[4] [4]

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. North American Chapter of the Association for Computational Linguistics. (2019)

work page 2019

[5] [5]

On Isotropy Calibration of Transformer Models

Ding, Y., Martinkus, K., Pascual, D., Clematide, S., & Wattenhofer, R. On Isotropy Calibration of Transformer Models. In Proceedings of the Third Workshop on In- sights from Negative Results in NLP (pp. 1–9). Association for Computational Lin- guistics. (2022)

work page 2022

[6] [6]

How Contextual are Contextualized Word Representations? Com- paring the Geometry of BERT, ELMo, and GPT-2 Embeddings

Ethayarajh, K. How Contextual are Contextualized Word Representations? Com- paring the Geometry of BERT, ELMo, and GPT-2 Embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- ĲCNLP) (pp. 55–65). Association for Computational Li...

work page 2019

[7] [7]

Representation Degenera- tion Problem in Training Natural Language Generation Models

Gao, J., He D., Tan, X., Qin, T., Wang, L., & Liu, T.-Y. Representation Degenera- tion Problem in Training Natural Language Generation Models. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019

work page 2019

[8] [8]

Anisotropy Is Inherent to Self- Attention in Transformers

Godey, N., Villemonte de la Clergerie, E., & Sagot, B. Anisotropy Is Inherent to Self- Attention in Transformers. Conference of the European Chapter of the Association for Computational Linguistics. (2024)

work page 2024