{"id":"a2495a3d-e420-4903-8179-2ed8bd31633b","arxiv_id":"2501.05502","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A persistent-entropy regularization loss reduces anisotropy of BERT and RoBERTa embeddings during fine-tuning, but downstream accuracy gains are small and mixed across tasks.","lead":"The authors propose a regularizer for fine-tuning transformer models that uses the entropy of topological persistence barcodes to make embedding spaces more isotropic. If it works as claimed, it offers a lightweight way to counter the representation degeneration problem without inference overhead or model reparametrization.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Maximum persistent entropy is achieved by collinear equally spaced points, so the entropy loss has no inherent isotropy-promoting mechanism and the central claim rests solely on four-task empirical evidence.","rationale":"The reader correctly identified the missing derivation connecting persistent entropy to singular-value concentration. My stress-test sharpens this into a concrete impossibility: the entropy objective is a function of edge-length multiplicities only, so it cannot distinguish isotropic from anisotropic point configurations. The equal-line example demonstrates that the loss can be already at its global maximum while anisotropy is maximal, which means the empirical improvement must come from task-specific interactions. This does not refute the observed results, but it refutes the paper's general claim and requires the authors either to provide a theoretical condition under which entropy regularization lowers anisotropy, or to restrict the claim to the tested settings and add significance testing. Given the thin experimental support (no error bars in Table 3, downstream gains mostly within noise), the appropriate verdict remains CONDITIONAL pending a demonstration of the mechanism or a weakened claim.","tokens_in":7362,"tokens_out":10216,"duration_ms":108834,"concrete_test":"Run a controlled synthetic experiment: sample N=64 points equally spaced along a line in R^768 (maximum persistent entropy, anisotropy_1 near 1) and fine-tune a two-layer network on a synthetic classification task with the proposed entropy regularization alone, then with the full objective. Compare anisotropy_1 trajectories. If the entropy term alone does not reduce anisotropy_1 relative to no regularization, or if the gradient of Lent with respect to a perpendicular coordinate perturbation at the equal-line configuration is zero or points back to the line, the causal mechanism is refuted. A cheaper analytic check: compute the Jacobian dLent/depsilon for a single point displaced by epsilon perpendicular to the line; if it is zero or negative for epsilon > 0, the loss stabilizes the anisotropic configuration.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central assertion is that maximizing persistent entropy of selected Vietoris-Rips barcodes improves latent-space isotropy. The load-bearing assumption is that equalizing the MST edge lengths reduces concentration of variance in the top singular direction. This link is not merely underived; it is false in general. Persistent entropy of the 0-dimensional barcode depends only on the multiset of MST edge lengths, not on the directions of those edges. A configuration of N points equally spaced along a line in R^D has all N-1 MST edges equal, so its persistent entropy attains the theoretical maximum log(N-1), while its covariance is rank-one and anisotropy_1 approaches 1. At this maximum, the entropy gradient with respect to edge lengths is zero (up to the scale-invariant direction), so the loss supplies no force that would spread the points into an isotropic cloud; small perpendicular perturbations decrease entropy and the gradient pushes points back toward the line. Thus the proposed regularization does not target the quantity it claims to improve. The observed anisotropy drops in the four BERT/RoBERTa fine-tuning runs must therefore be attributed to an interaction between the entropy term, the feature-selection heuristic, and the cross-entropy loss on those specific datasets. Table 3 reports no error bars and no significance testing, and the downstream gains in Table 2 are within one standard deviation of the baseline for three of four tasks. The conclusion that the method 'leads to improvements in isotropy' as a model-agnostic property is therefore unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an entropy-based regularization term for fine-tuning transformer models, computed from the persistent entropy of 0-dimensional Vietoris-Rips barcodes (i.e., MST edge lengths) of [CLS] embeddings within each class, with a topological feature-selection step adapted from Atienza et al. The authors fine-tune BERT-base and RoBERTa-base on MRPC and COLA, and report that the regularized models have lower anisotropy and slightly better GLUE scores than the unregularized baselines. The central claim is that entropy regularization with feature selection improves latent-space isotropy and generalization without retraining and without inference overhead.","tokens_in":7628,"tokens_out":6381,"duration_ms":65085,"significance":"If the empirical claims were solid, the method would be a lightweight and model-agnostic alternative to existing isotropy-calibration techniques, and the use of differentiable persistent entropy as a regularizer would be a novel contribution to representation-learning. The paper is also distinctive in trying to preserve, rather than destroy, the cluster structure of the embedding space. However, the manuscript does not establish the theoretical link between persistent entropy and anisotropy, and the reported evidence is weakened by missing statistical tests and an unreported hyperparameter. The idea is promising but the current support is insufficient for acceptance.","major_comments":[{"comment":"Persistent entropy of the 0-dimensional barcode is a function only of the multiset of MST edge lengths, not of the spatial directions of those edges. A configuration of N points equally spaced along a line in R^D has all N-1 MST edges equal, so its persistent entropy attains the theoretical maximum log(N-1), while its covariance matrix is rank-one and anisotropy_1 tends to 1. Moreover, for such a collinear configuration the gradient of the entropy with respect to small perpendicular displacements vanishes, so the loss supplies no force that would spread the points into an isotropic cloud. The manuscript therefore does not establish the claimed causal mechanism by which maximizing persistent entropy improves isotropy; it only reports correlations in four fine-tuning runs. The authors should either provide a theoretical justification specific to their class-wise, batched setting or explicitly reframe the claim as an empirical heuristic and support it with substantially more evidence.","section":"Section 3, Eq. (1)-(2) and Section 5 claim"},{"comment":"The weight of the entropy regularization term relative to the cross-entropy loss is not specified. The final objective L = LCE - sum_i Lent * 1[class=i] has no coefficient for Lent, yet the scale of Lent depends on the number of selected bars and on the distances between embeddings, which vary across tasks and batches. Without reporting this hyperparameter (or a schedule for it), the experiments are not reproducible, and the observed effects could be driven by an arbitrary scaling of the two loss terms.","section":"Section 4, objective and Table 1"},{"comment":"Table 3 reports anisotropy values without standard deviations or significance tests, despite the statement in Section 4 that each experiment is repeated 5 times. The claim of \"significantly lowering\" anisotropy is therefore not supported statistically. The downstream results in Table 2 are also mixed: BERT-MRPC degrades from 0.892 ± 0.002 to 0.889 ± 0.001, and the gains on RoBERTa-MRPC and RoBERTa-COLA are within one standard deviation of the baseline. The paper should report per-run anisotropy values and paired significance tests (e.g., bootstrap or paired t-tests) before asserting consistent improvements.","section":"Section 5, Tables 2 and 3"},{"comment":"The feature-selection algorithm is described in a way that is hard to follow and appears to contain ambiguities or typographical errors: the sorting condition \"li ≥ lj ≥ ln−1 = r\" is unclear, the recursive call changes n′ without explicitly describing how L′0 is updated, and the quantity Q becomes undefined when α = r/T = 1, which is precisely the equal-spacing regime relevant to the counterexample above. Since the authors attribute a substantial part of the improvement to the selection step in Section 5, a precise, self-contained description is necessary for reproduction.","section":"Section 3, Algorithm 1"},{"comment":"The sentence \"an addition of entropy regularization paired with topological feature selection leads to improvements both in terms of isotropy and generalization\" overstates the evidence in Tables 2 and 3, given the missing statistical support and the mixed downstream numbers. The conclusion should be softened to reflect the actual empirical scope, or the authors should provide additional experiments that justify the strong claim.","section":"Section 5, concluding paragraph"}],"minor_comments":[{"comment":"The code link in the abstract is misspelled (\"avaliable\").","section":"Abstract"},{"comment":"The text says \"the exact parameters used for fine-tuning are shown in Table 1,\" but Table 1 omits the entropy-loss weight, so the word \"exact\" is inaccurate.","section":"Section 4 vs Table 1"},{"comment":"The sign convention is confusing: Lent is defined as the positive entropy, then the text says \"we take it with a minus sign,\" and the final objective subtracts Lent. Please clarify the convention explicitly so the reader does not have to infer that the objective maximizes entropy rather than minimizing it.","section":"Section 4, equations (3)-(4)"},{"comment":"The captions do not identify which curve corresponds to \"No reg,\" \"Ent. loss (selected barcodes),\" and \"Ent. loss (all barcodes).\" Add a legend or explicit line-style labels.","section":"Figures 1 and 2"},{"comment":"Reference [24] appears to duplicate reference [14] (both are the RoBERTa paper). Please consolidate and use consistent spelling of \"RoBERTa\" throughout (e.g., \"RoBERTA-MRPC\" vs \"RoBERTa-COLA\").","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The theoretical concern about persistent entropy being direction-independent is serious and goes to the heart of the paper's motivation. I nevertheless recommend major revision rather than rejection because the empirical claim could in principle be supported by more thorough experiments, and because the authors could soften the claim to an empirical heuristic. If the resubmission does not address the missing loss weight, standard deviations, and significance testing, or if the anisotropy gains disappear under those tests, the paper should be rejected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is the first in its cited set to use persistent entropy of selected barcodes as an anisotropy regularizer, and the idea of exploiting cluster geometry during fine-tuning is worth a look. The feature-selection step borrowed from Atienza et al. is nontrivial, and the authors provide code, which is commendable. Table 3 shows consistent anisotropy drops across all four fine-tuning runs, and the centered anisotropy numbers move in the same direction. That is not nothing.\n\nBut the load-bearing assumption does not survive contact with the math. Persistent entropy of the 0-dimensional barcode depends only on the multiset of MST edge lengths. It carries no information about the directions of those edges. A configuration of N equally spaced points on a line in R^D has all N-1 MST edges equal, so its persistent entropy is maximal (log(N-1)), while its covariance is rank-one and anisotropy_1 approaches 1. At that maximum the entropy gradient is zero, so the loss supplies no force to spread the points into an isotropic cloud. In other words, the loss's optima include maximally anisotropic configurations. The claimed causal link between entropy maximization and isotropy is not merely underived; it is false in general. The observed anisotropy drops in the four BERT/RoBERTa runs must therefore come from an interaction between the entropy term, the class-wise splitting, and the cross-entropy loss, not from the entropy term doing what the paper says it does.\n\nThe empirical reporting is also weaker than the text suggests. Table 3 lists no standard deviations or significance tests, even though the authors ran five seeds. Downstream metrics in Table 2 are mixed: BERT-MRPC gets slightly worse, and all other differences are within one standard deviation of the baseline. Implementation details that matter for reproducibility—the entropy loss weight, gradient flow through the discrete selection step, and the persistent-homology library—are absent.\n\nThis paper is not ready for peer review. A serious referee would need a real theoretical connection between persistent entropy and the spectrum of the embedding covariance, or a much larger empirical study with error bars and ablations that isolate the mechanism. As it stands, the central argument is a fallacy, and the empirical tail does not wag the dog. I would not cite it in its current form, though it could be useful as a reading-group example of how TDA-inspired losses can optimize the wrong quantity.","headline":"The entropy loss ignores directions, so the central claim that it improves isotropy is mathematically unfounded, and the empirical evidence is too thin to rescue it.","tokens_in":737,"tokens_out":2146,"would_cite":false,"duration_ms":45773,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A persistent-entropy regularizer derived from the Vietoris-Rips filtration of [CLS] embeddings reduces latent-space anisotropy and can improve GLUE scores during fine-tuning of transformer models, with no retraining or inference overhead.","keywords":["latent space isotropy","representation degeneration","persistent homology","persistent entropy","topological data analysis","BERT fine-tuning","regularization","Vietoris-Rips filtration"],"falsifier":"Track persistent entropy and top singular-value share across training: if they do not move together, or if random bar selection yields the same anisotropy drop, the claimed mechanism is not the cause.","tokens_in":7155,"feed_emoji":"📉","tokens_out":8933,"duration_ms":77111,"temperature":0.7,"pith_summary":"Transformer embeddings tend to collapse into a narrow cone, a problem known as representation degeneration. This paper proposes a regularization loss that counteracts that collapse during fine-tuning: it maximizes the persistent entropy of selected barcodes obtained from a Vietoris-Rips filtration of the [CLS] embeddings. The authors show that, paired with a topological feature-selection step, the loss lowers the anisotropy of the embedding matrix on BERT and RoBERTa fine-tuned on MRPC and CoLA, and it improves or maintains downstream accuracy. The significance is that the method works on the existing geometric structure of the latent space, so it requires no reparametrization, no retraining, and no extra cost at inference time. If the claim holds, it offers a cheap, model-agnostic way to improve representation quality in low-data fine-tuning scenarios.","feed_headline":"Persistent-entropy loss shrinks the anisotropy cone of BERT embeddings","feed_subtitle":"Adding a topology-based regularizer during fine-tuning lowers anisotropy and can improve GLUE scores without retraining.","key_machinery":"The central object is the persistent entropy of a 0-dimensional Vietoris-Rips barcode, defined as the Shannon entropy of the normalized edge lengths of the minimum spanning tree built on the embedding points. The machinery works in three steps: (1) compute the barcode from a batch of [CLS] embeddings, (2) apply the feature-selection procedure of Atienza et al. to keep only the longest bar and the bars that are not classified as noise, and (3) add the negative persistent entropy of the selected bars to the cross-entropy loss. This term is differentiable through the bar lengths via the differential calculus of persistence barcodes, so gradient descent can maximize it. The paper's key move is to use this topological entropy as a proxy for geometric isotropy: pushing all selected bar lengths toward equality empirically flattens the spectrum of the embedding covariance.","core_discovery":"The central claim is that an entropy regularization term computed on the 0-dimensional persistent homology of contextual embeddings reduces the concentration of variance in the top singular directions of the embedding matrix. The authors implement this by taking the [CLS] token from the last hidden layer, reducing the Vietoris-Rips filtration to a minimum spanning tree, selecting 'topological features' with the algorithm of Atienza et al., and adding the negative persistent entropy of the selected bar lengths to the classification loss. They apply the loss separately per class label to preserve the cluster structure of the latent space. In experiments on BERT and RoBERTa fine-tuned on MRPC and CoLA, the first anisotropy component drops by about 0.1–0.2 and the centered anisotropy by up to 0.2, while BERT's CoLA accuracy rises from 0.575 to 0.588. The paper also reports that omitting feature selection removes most of the benefit, which supports the claim that the method exploits existing cluster geometry rather than reparametrizing it.","pith_inferences":["An untested but natural extension is to apply the same loss during pretraining rather than only fine-tuning; if the entropy-anisotropy link holds throughout training, the cone could be prevented from forming in the first place.","The group-wise application suggests a possible use in class-imbalanced or multi-task fine-tuning, where preserving inter-class separation while flattening within-class directions could be even more valuable than in the balanced case.","A formal tie between the MST edge-length distribution and the singular-value spectrum of the embedding Gram matrix is missing; if such a bound exists, it would predict precisely how much anisotropy reduction is achievable for a given barcode.","The same topological regularizer could be tested on other point-cloud representations such as graph node embeddings or vision transformer patch embeddings, where the cone phenomenon has also been observed."],"forward_implications":["The regularization loss can be grafted onto any fine-tuning objective without retraining the base model and without adding inference overhead.","Feature selection is essential: using all bar lengths fails to lower anisotropy and can leave downstream performance flat, whereas the selected-barcode version raises CoLA accuracy for BERT.","The method is model-agnostic, since it only requires a point cloud of embeddings, so it should transfer to other transformer architectures and non-text modalities.","Anisotropy improvements appear on the top singular components and on the centered covariance matrix, meaning the effect is not just a rescaling of embedding norms.","Because the loss is applied per class label, it preserves the cluster structure that prior work identified as useful, avoiding the damage that reparametrization-based methods can cause."],"supporting_citations":[{"why":"Supplies the differential calculus on persistence barcodes that makes the entropy loss differentiable.","marker":"[11]"},{"why":"Provides the topological feature-selection algorithm that separates meaningful bars from noise.","marker":"[1]"},{"why":"Defines the anisotropy score from singular-value decomposition used as the evaluation metric.","marker":"[18]"},{"why":"Documents that reparametrization-based isotropy calibration methods fail to improve GLUE tasks consistently, the baseline the paper positions its method against.","marker":"[5]"},{"why":"Establishes that contextual embeddings in BERT, ELMo, and GPT-2 are highly anisotropic, motivating the problem.","marker":"[6]"},{"why":"Shows the latent space consists of locally isotropic clusters, supporting the paper's per-class group-wise loss.","marker":"[2]"},{"why":"Supplies the BERT base model used in the fine-tuning experiments.","marker":"[4]"},{"why":"Supplies the RoBERTa base model used in the fine-tuning experiments.","marker":"[24]"},{"why":"Defines the GLUE benchmark from which the MRPC and CoLA tasks are taken.","marker":"[21]"}],"fun_headline_variants":["Persistent entropy loss shrinks BERT's anisotropy cone","Topology-based regularizer lowers anisotropy, boosts GLUE","Simplicial geometry loss improves latent space isotropy","Entropy from barcodes flattens embedding cone during fine-tuning","No reparametrization: entropy loss makes embeddings isotropic"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that maximizing the entropy of the selected tree-edge lengths causes the variance of the embedding matrix to spread away from the top singular directions; the paper offers empirical correlation but no derivation connecting these two quantities, so if that link fails outside the tested settings the method loses its claimed effect.","fun_headline_variants_meta":{"raw":{"variants":["Persistent entropy loss shrinks BERT's anisotropy cone","Topology-based regularizer lowers anisotropy, boosts GLUE","Simplicial geometry loss improves latent space isotropy","Entropy from barcodes flattens embedding cone during fine-tuning","No reparametrization: entropy loss makes embeddings isotropic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000277,"raw_usage":{"total_tokens":1630,"prompt_tokens":906,"completion_tokens":724,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":640}},"tokens_in":522,"tokens_out":724,"duration_ms":7032,"temperature":1.0,"reasoning_tokens":640,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:13:04.095265+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Track persistent entropy and top singular-value share across training: if they do not move together, or if random bar selection yields the same anisotropy drop, the claimed mechanism is not the cause.","supporting_citations":[{"cited_title":"A Framework for Differential Calculus on Persistence Barcodes","cited_arxiv_id":null,"evidence_quote":"Supplies the differential calculus on persistence barcodes that makes the entropy loss differentiable."},{"cited_title":"Persistent entropy for separating topo- logical features from noise in Vietoris-rips complexes","cited_arxiv_id":null,"evidence_quote":"Provides the topological feature-selection algorithm that separates meaningful bars from noise."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the anisotropy score from singular-value decomposition used as the evaluation metric."},{"cited_title":"On Isotropy Calibration of Transformer Models","cited_arxiv_id":null,"evidence_quote":"Documents that reparametrization-based isotropy calibration methods fail to improve GLUE tasks consistently, the baseline the paper positions its method against."},{"cited_title":"How Contextual are Contextualized Word Representations? Com- paring the Geometry of BERT, ELMo, and GPT-2 Embeddings","cited_arxiv_id":null,"evidence_quote":"Establishes that contextual embeddings in BERT, ELMo, and GPT-2 are highly anisotropic, motivating the problem."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows the latent space consists of locally isotropic clusters, supporting the paper's per-class group-wise loss."},{"cited_title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","cited_arxiv_id":null,"evidence_quote":"Supplies the BERT base model used in the fine-tuning experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the RoBERTa base model used in the fine-tuning experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the GLUE benchmark from which the MRPC and CoLA tasks are taken."}],"review_version":1}