Pith. sign in

REVIEW 2 major objections 5 minor 41 references

Separating representation from token reconstruction lets text encoders scale without losing usable frozen features.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 22:19 UTC pith:E6A6XBJL

load-bearing objection CrossBERT is a clean, well-controlled empirical fix for the frozen-scaling collapse of flat BERT; the bipartite design and CMS are real contributions, and the paper should go to referees. the 2 major comments →

arxiv 2607.04011 v1 pith:E6A6XBJL submitted 2026-07-04 cs.CL cs.AI

Separating Representation from Reconstruction Enables Scalable Text Encoders

Megi Dervishi , Mathurin Videau , Yann LeCun This is my paper
classification cs.CL cs.AI
keywords text encodersmasked language modelingCrossBERTfrozen probingrepresentation learningcomplementary maskingscaling lawsMTEB
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard BERT-style encoders improve at reconstructing masked tokens as they grow, yet their frozen features become harder for simple probes to use on classification and embedding tasks. The paper argues this is not an accident of data or scale but a design problem: a flat network is forced to serve both high-level representation and local token prediction at once, so the last layers overspecialize for reconstruction. CrossBERT splits the job. A deep encoder builds the features; a shallow cross-attention predictor is solely responsible for filling in the masked tokens. With that split, the model tolerates masking half or more of the tokens, can train on a complementary pair of masks so every token contributes a gradient, and shows steadily improving frozen performance as compute grows, on both GLUE probes and MTEB-style embedding tasks.

Core claim

As compute scales, flat BERT representations become increasingly unexploitable by frozen linear and kNN probes (and by frozen contrastive adapters) even while reconstruction loss improves; the cause is coupling of representation learning to the token-reconstruction objective, and a bipartite encoder-plus-cross-attention-predictor architecture restores monotonic scaling and superior frozen GLUE and MTEB performance.

What carries the argument

CrossBERT: a bipartite Masked Autoencoder for text in which a deep encoder processes only visible tokens and a lightweight predictor (roughly one-fourth the depth, cross-attention only) reconstructs the complementary masked tokens from the encoder states, insulating the backbone from reconstruction specialization.

Load-bearing premise

That average-pooled last-layer features scored by linear and nearest-neighbor probes, plus a lightweight frozen contrastive pooler, are a fair and sufficient measure of intrinsic representation quality, so the BERT degradation and CrossBERT gains can be blamed mainly on architectural coupling.

What would settle it

Train matched flat BERT and CrossBERT families at the same data-to-model ratios and compute budgets; if frozen linear/kNN GLUE and frozen MTEB scores of BERT continue to rise monotonically (or match CrossBERT) once last-layer pooling and probe hyperparameters are held fixed, the coupling diagnosis fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper argues that flat BERT-style MLM couples representation learning to token reconstruction, so that frozen last-layer features become less exploitable by linear/kNN probes and frozen contrastive adapters as compute scales, even while perplexity improves. CrossBERT separates the two via a bipartite design: a full encoder that sees only unmasked tokens and a shallow cross-attention-only predictor that reconstructs the complementary masked view. Complementary Masking Strategy (CMS) at ≥50% masking doubles sample efficiency and raises throughput. Controlled same-corpus, same-tokenizer sweeps (8 BERT + 9 CrossBERT models, fixed 20:1 data-to-model ratio) show monotonic frozen GLUE and MTEB(eng,v2) scaling for CrossBERT and degradation for BERT (Figs. 1, 3–4; Tables 1–3, 8).

Significance. If the scaling results hold, the work supplies a concrete architectural diagnosis and fix for a long-standing encoder stagnation problem, with direct practical value for retrieval, data curation, and frozen-embedding pipelines. Strengths include the controlled multi-scale sweep under fixed data-to-model ratio, layer-wise over-specialization evidence (Table 1), masking/CMS/predictor ablations (Tables 2, 4–7), and the clear frozen-vs-unfrozen gap that is large for BERT and near-zero for CrossBERT. The efficiency gains (1.5–2 imes throughput, 2 imes sample efficiency) are independently useful. The evaluation protocol itself (frozen linear/kNN + frozen contrastive pooler) is a useful contribution for future encoder work.

major comments (2)
  1. The central attribution of BERT degradation to architectural coupling (flat MLM) rather than probe/pooling choice is load-bearing but only partially stress-tested. Table 1 shows modest last-layer over-specialization (+1.2–1.8 pts from earlier layers), yet the main claims rest on average-pooled last-layer features (§4.1) and a lightweight frozen contrastive pooler (§4.2). A stronger control would report the same frozen MTEB/GLUE curves when (i) probing intermediate layers of large BERTs or (ii) using the same cross-attention pooler architecture on both backbones without warm-start. Without that, residual confounding by probe choice remains possible, though the same probes are applied to both models and the retrieval collapse of BERT is structural.
  2. §5.3 and Appendix I document training instability and loss spikes for large-scale BERT (and Electra) that are largely absent for CrossBERT. The paper correctly notes that undertraining is unlikely (20:1 ratio), but does not fully separate optimization brittleness from the claimed representation misalignment. A short additional experiment—e.g., the largest BERT re-trained with the same init-std reduction that stabilized CrossBERT 2B, or with Electra-style RTD under identical CMS-free conditions—would strengthen the claim that the frozen-score drop is architectural rather than an optimization artifact.
minor comments (5)
  1. Figure 1 caption and §5.1 should state explicitly that MTEB(eng,v2) is used for the scaling plots while Table 3 reports MTEB(eng,v1) for literature comparison; the switch is easy to miss.
  2. Eq. (8) for CrossBERT FLOPs is clear, but the text should note whether the reported FLOPs already include CMS (two complementary views) or are per-view; this affects the 1.5–2 imes throughput claim.
  3. Table 3 lists ModernBERT/NeoBERT/DeBERTaV3 with much larger data budgets and different finetuning mixtures; a short caveat in the caption would prevent over-reading the absolute MTEB numbers.
  4. Typos / notation: “unexploitableby” (abstract), “L la y ers” / “Cr oss-A ttn” in Fig. 2, and inconsistent “MTEB(eng, v1/v2)” spacing.
  5. Appendix J model tables are valuable; adding the exact masking ratio and whether CMS was on for each row would make the sweep fully self-contained.

Circularity Check

0 steps flagged

No significant circularity: empirical architecture comparison measured on external frozen benchmarks

full rationale

The paper's central claims (BERT frozen-probe degradation with scale under flat MLM; CrossBERT restoring monotonic frozen GLUE/MTEB scaling via bipartite encoder+cross-attention predictor) are established by controlled same-corpus training runs and direct measurement on external benchmarks (GLUE linear/kNN probes, MTEB contrastive frozen/unfrozen). Hyperparameter power laws for BSZ/LR are fitted on small-scale sweeps then applied as training settings; they do not define or force the reported probe scores. Self-citations to the authors' prior OptiBERT work supply only baseline numbers and BERT hyperparameter defaults, not a load-bearing uniqueness or derivation step. Inspiration from external vision MAE papers is architectural, not a smuggled ansatz that makes the frozen-score results true by construction. No equation, definition, or fit reduces the claimed performance gap to its own inputs.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 2 invented entities

The central claim rests on standard transformer/MLM machinery plus a small set of design choices (predictor depth ≈1/4, 50% mask + CMS, 20:1 data-to-model ratio, frozen average-pool probes) that are either taken from vision MAE or fitted on small-scale sweeps. No new physical entities are postulated; the architecture itself is the main invented construct.

free parameters (5)
  • predictor depth / width ratio
    Chosen as ~1/4 encoder depth following MAE; ablated but still a free design choice that affects the reported compute–performance trade-off (Appendix A).
  • masking ratio (CrossBERT 50%, BERT 20%)
    Selected after ablation; higher ratios are enabled by the architecture but remain free parameters that drive the throughput claims (Table 2).
  • data-to-model ratio 20:1
    Fixed for the scaling sweep after citing prior work; not derived from first principles (Section 5.1).
  • BSZ and LR power-law coefficients
    Fitted on 50–700 M models then extrapolated (Section 5.1, Appendix H); the scaling curves depend on these fitted schedules.
  • Ridge λ and kNN k / distance
    Swept and best-of reported for frozen GLUE probes; affects absolute probe scores (Section 4.1).
axioms (4)
  • domain assumption Frozen linear/kNN probes on average-pooled last-layer features measure intrinsic representation quality relevant to downstream use.
    Core evaluation premise of Sections 4.1–4.2; if probes are too weak or pooling is suboptimal, the claimed degradation/gain could be overstated.
  • domain assumption Standard MLM negative log-likelihood on masked tokens is a valid pre-training objective whose only defect is architectural coupling.
    Assumed throughout; Electra RTD comparison shows objective choice also matters, but is treated as secondary.
  • standard math Transformer FLOPs formulas (Eqs. 6–8) correctly capture relative compute of encoder vs. predictor under masking.
    Used to place models on the same FLOPs axis for scaling laws.
  • domain assumption DCLM subset + RoBERTa tokenizer is a representative pre-training distribution for the claimed scaling trends.
    All models share this corpus; generalization beyond it is untested.
invented entities (2)
  • CrossBERT bipartite architecture (encoder + cross-attention-only predictor) no independent evidence
    purpose: Decouple representation learning from token reconstruction so frozen features remain exploitable at scale.
    Core proposed construct; independent evidence is the empirical scaling and ablation results inside the paper.
  • Complementary Masking Strategy (CMS) no independent evidence
    purpose: Process a mask and its complement in one batch to collect gradients on every token and double sample efficiency.
    Enabled by high-masking robustness of CrossBERT; evidence is internal ablation (Table 5).

pith-pipeline@v1.1.0-grok45 · 23624 in / 3262 out tokens · 30863 ms · 2026-07-11T22:19:46.876614+00:00 · methodology

0 comments
read the original abstract

While decoders have rapidly scaled, encoders have remained largely unchanged since BERT. We revisit this disparity by frozen backbone evaluation via probing. Under this lens, the representations of BERT encoders become increasingly $\textit{unexploitable}$ by frozen probes, despite improved perplexity. The misalignment originates in BERT's flat design, which couples representation learning to the token reconstruction loss. We propose $\textbf{CrossBERT}$, a two-part architecture that separates the learning of high-quality encoded representations from the rigid grounding of token reconstruction. This design further enables high masking ratios ($\ge 50\%$) and gradient collection over all tokens via a $\textit{Complementary Masking Strategy}$, respectively increasing throughput by $1.5$ to $2\times$ and sample efficiency by $2\times$. Overall, CrossBERT demonstrates monotonic scaling and superior performance on MTEB(eng, v2) and frozen GLUE benchmarks.

Figures

Figures reproduced from arXiv: 2607.04011 by Mathurin Videau, Megi Dervishi, Yann LeCun.

Figure 1
Figure 1. Figure 1: Top. Frozen evaluation of encoders on GLUE, linear and KNN probes are fitted on the average representation of a frozen backbone. Bottom. MTEB(eng, v2) score. ‘(frozen)’ means only the pooler is finetuned on top of the frozen features of the encoder i.e frozen backbone. ‘(unfrozen)’ means that the full network (including the backbone) is finetuned end-to-end. Both are finetuned only on MS-MARCO for one epoc… view at source ↗
Figure 2
Figure 2. Figure 2: Left. The Complementary Masking Strategy (CMS) augments a batch of tokens into two complementary masked views by replacing tokens with <MASK>. Masked tokens are in red; the unmasked tokens of the two views are in green and blue. Numbers indicate positional indices. Right. CrossBERT predicts one view (green) from the other (blue), and vice-versa. Both views are processed in parallel with an attention mask i… view at source ↗
Figure 3
Figure 3. Figure 3: Scaling trend of BERT vs CrossBERT after contrastive finetuning under frozen and unfrozen backbone for MTEB(eng, v2). All models are trained for one epoch on MS-MARCO as described in Section 5.1 Training efficiency and throughput. CrossBERT demon￾strates superior computational efficiency. In terms of train￾ing throughput, it achieves 207k tokens/sec compared to the baseline’s 123k tokens/sec, representing … view at source ↗
Figure 4
Figure 4. Figure 4: Scaling trends of BERT vs CrossBERT using linear and kNN probing for GLUE under frozen backbone(see Section 5.1). ings that RTD distorts sentence-embedding geometry (Rep et al., 2024; Warner et al., 2025). Two conclusions follow. First, neither swapping the objective on a flat backbone (Electra) nor keeping MLM on a flat backbone (BERT) yields versatile frozen representations; the bipartite separa￾tion of … view at source ↗
Figure 5
Figure 5. Figure 5: Learning curves for different BERT(left) and CrossBERT(right) setup specified in model list Appendix J. Each dot represents the validation loss on wikipedia and dclm(Li et al., 2024). 13 [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Average classification performance across 10 tasks monitored during training for BERT (red, ■) and CrossBERT (blue, •). H. Learning rate and batch size sweeps 1019 FLOPs 10−3 10−2 Learning Rate 0.01 0.1 1 Smoothed loss 1019 FLOPs 105 106 Batch Size 0.01 0.1 1 Smoothed loss 10−3 10−2 Learning Rate 105 106 Batch Size 0.01 0.1 1 Smoothed loss [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: CrossBert heatmap for lr and batch size, Left: flops vs lr, Middle: flops vs batch size, Right: lr vs batch size, the size of the point is proportionnal to total compute budget spend for training. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 2 canonical work pages

  1. [1]

    Training compute-optimal transformer encoder models

    Dervishi, Megi and Allauzen, Alexandre and Synnaeve, Gabriel and LeCun, Yann. Training compute-optimal transformer encoder models. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025. doi:10.18653/v1/2025.emnlp-main.1804

    work page doi:10.18653/v1/2025.emnlp-main.1804 2025

  2. [2]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  3. [3]

    arXiv preprint arXiv:2502.19587 , year=

    NeoBERT: A Next-Generation BERT , author=. arXiv preprint arXiv:2502.19587 , year=

    Pith/arXiv arXiv

  4. [4]

    arXiv preprint arXiv:2402.01613 , year=

    Nomic embed: Training a reproducible long context text embedder , author=. arXiv preprint arXiv:2402.01613 , year=

    Pith/arXiv arXiv

  5. [5]

    Bert: Pre-training of deep bidirectional transformers for language understanding , author=. Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) , pages=

    2019

  6. [6]

    Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pages=

    Mteb: Massive text embedding benchmark , author=. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pages=

  7. [7]

    arXiv preprint arXiv:2401.14391 , year=

    Rethinking patch dependence for masked autoencoders , author=. arXiv preprint arXiv:2401.14391 , year=

    Pith/arXiv arXiv

  8. [8]

    arXiv preprint arXiv:2502.08769 , year=

    Cluster and predict latent patches for improved masked image modeling , author=. arXiv preprint arXiv:2502.08769 , year=

    Pith/arXiv arXiv

  9. [9]

    International conference on machine learning , pages=

    A simple framework for contrastive learning of visual representations , author=. International conference on machine learning , pages=. 2020 , organization=

    2020

  10. [10]

    Proceedings of the IEEE/CVF international conference on computer vision , pages=

    Emerging properties in self-supervised vision transformers , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=

  11. [11]

    arXiv preprint arXiv:2304.07193 , year=

    Dinov2: Learning robust visual features without supervision , author=. arXiv preprint arXiv:2304.07193 , year=

    Pith/arXiv arXiv

  12. [12]

    arXiv preprint arXiv:2508.10104 , year=

    Dinov3 , author=. arXiv preprint arXiv:2508.10104 , year=

    Pith/arXiv arXiv

  13. [13]

    arXiv preprint arXiv:2112.09118 , year=

    Unsupervised dense information retrieval with contrastive learning , author=. arXiv preprint arXiv:2112.09118 , year=

    Pith/arXiv arXiv

  14. [14]

    arXiv preprint arXiv:2205.12035 , year=

    RetroMAE: Pre-training retrieval-oriented language models via masked auto-encoder , author=. arXiv preprint arXiv:2205.12035 , year=

    Pith/arXiv arXiv

  15. [15]

    Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP , pages=

    GLUE: A multi-task benchmark and analysis platform for natural language understanding , author=. Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP , pages=

    2018

  16. [16]

    arXiv preprint arXiv:1611.09268 , year=

    Ms marco: A human generated machine reading comprehension dataset , author=. arXiv preprint arXiv:1611.09268 , year=

    Pith/arXiv arXiv

  17. [17]

    Advances in Neural Information Processing Systems , volume=

    Datacomp-lm: In search of the next generation of training sets for language models , author=. Advances in Neural Information Processing Systems , volume=

  18. [18]

    arXiv preprint arXiv:2212.03533 , year=

    Text embeddings by weakly-supervised contrastive pre-training , author=. arXiv preprint arXiv:2212.03533 , year=

    Pith/arXiv arXiv

  19. [19]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Self-supervised learning from images with a joint-embedding predictive architecture , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  20. [20]

    arXiv preprint arXiv:2105.04906 , year=

    Vicreg: Variance-invariance-covariance regularization for self-supervised learning , author=. arXiv preprint arXiv:2105.04906 , year=

    Pith/arXiv arXiv

  21. [21]

    arXiv preprint arXiv:2403.00504 , year=

    Learning and leveraging world models in visual representation learning , author=. arXiv preprint arXiv:2403.00504 , year=

    Pith/arXiv arXiv

  22. [22]

    arXiv preprint arXiv:2511.08544 , year=

    LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics , author=. arXiv preprint arXiv:2511.08544 , year=

    Pith/arXiv arXiv

  23. [23]

    arXiv preprint arXiv:2504.13181 , year=

    Perception encoder: The best visual embeddings are not at the output of the network , author=. arXiv preprint arXiv:2504.13181 , year=

    Pith/arXiv arXiv

  24. [24]

    arXiv preprint arXiv:2303.15389 , year=

    Eva-clip: Improved training techniques for clip at scale , author=. arXiv preprint arXiv:2303.15389 , year=

    Pith/arXiv arXiv

  25. [25]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Masked autoencoders are scalable vision learners , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  26. [26]

    arXiv preprint arXiv:1907.11692 , year=

    Roberta: A robustly optimized bert pretraining approach , author=. arXiv preprint arXiv:1907.11692 , year=

    Pith/arXiv arXiv 1907

  27. [27]

    Advances in neural information processing systems , volume=

    Pytorch: An imperative style, high-performance deep learning library , author=. Advances in neural information processing systems , volume=

  28. [28]

    Mathurin Videau and Badr Youbi Idrissi and Daniel Haziza and Luca Wehrstedt and Jade Copet and Olivier Teytaud and David Lopez-Paz , title =

  29. [29]

    arXiv preprint arXiv:2104.08663 , year=

    Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models , author=. arXiv preprint arXiv:2104.08663 , year=

    Pith/arXiv arXiv

  30. [30]

    arXiv:2001.08361 , year=

    Scaling laws for neural language models , author=. arXiv:2001.08361 , year=

    Pith/arXiv arXiv 2001

  31. [31]

    and Sifre, Laurent , title =

    Hoffmann, Jordan and Borgeaud, Sebastian and Mensch, Arthur and Buchatskaya, Elena and Cai, Trevor and Rutherford, Eliza and de Las Casas, Diego and Hendricks, Lisa Anne and Welbl, Johannes and Clark, Aidan and Hennigan, Tom and Noland, Eric and Millican, Katie and van den Driessche, George and Damoc, Bogdan and Guy, Aurelia and Osindero, Simon and Simony...

    2022

  32. [32]

    arXiv:2401.02954 , year=

    Deepseek llm: Scaling open-source language models with longtermism , author=. arXiv:2401.02954 , year=

    Pith/arXiv arXiv

  33. [33]

    arXiv:2403.08540 , year=

    Language models scale reliably with over-training and on downstream tasks , author=. arXiv:2403.08540 , year=

    Pith/arXiv arXiv

  34. [34]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    MEXMA: Token-level objectives improve sentence representations , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  35. [35]

    Journal of machine learning research , volume=

    Exploring the limits of transfer learning with a unified text-to-text transformer , author=. Journal of machine learning research , volume=

  36. [36]

    arXiv preprint arXiv:2104.08821 , year=

    Simcse: Simple contrastive learning of sentence embeddings , author=. arXiv preprint arXiv:2104.08821 , year=

    Pith/arXiv arXiv

  37. [37]

    arXiv preprint arXiv:2308.03281 , year=

    Towards general text embeddings with multi-stage contrastive learning , author=. arXiv preprint arXiv:2308.03281 , year=

    Pith/arXiv arXiv

  38. [38]

    Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pages=

    Should you mask 15\ author=. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pages=

  39. [39]

    2020 , eprint=

    ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators , author=. 2020 , eprint=

    2020

  40. [40]

    2023 , eprint=

    DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing , author=. 2023 , eprint=

    2023

  41. [41]

    Are ELECTRA ' s Sentence Embeddings Beyond Repair? The Case of Semantic Textual Similarity

    Rep, Ivan and Duki \'c , David and S najder, Jan. Are ELECTRA ' s Sentence Embeddings Beyond Repair? The Case of Semantic Textual Similarity. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024. doi:10.18653/v1/2024.findings-emnlp.535

    work page doi:10.18653/v1/2024.findings-emnlp.535 2024