Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Do BERT-Like Bidirectional Models Still Perform Better on Text Classification in the Era of LLMs?

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuned BERT-like models still outperform zero-shot and internal-state LLM methods on most of six hard text-classification tasks, and the paper provides a rule for choosing between them.

desk verdict A genuinely useful narrow benchmark buried under an over-broad headline; worth reviewing but the authors need to scope the claims. read the letter →

arxiv 2505.18215 v1 pith:XLAWNSSI submitted 2025-05-23 cs.CL cs.AI

Junyan Zhang , Yiming Huang , Shuliang Liu , Yubo Gao , Xuming Hu This is my paper · ORCID
classification cs.CLcs.AI
keywords BERT-likemodelstextclassificationlargelanguagezero-shotpromptinginternalstateprobingmodelselectionpattern-driventaskstask-awarestrategy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

On six deliberately difficult text-classification benchmarks, fine-tuned BERT-style models beat zero-shot prompting and internal-state probing with 7B/8B open-weight LLMs on the majority of tasks, while using far less compute. The paper argues that the community's reflexive 'use an LLM' approach has prematurely sidelined cheap bidirectional encoders. It groups the six datasets into pattern-driven, rule-based, and knowledge-intensive types, and shows LLMs win only on the knowledge-intensive one (hallucination detection). Based on these groupings it proposes TaMAS, a selection rule that says: choose BERT-like models for pattern-driven or rule-based tasks, and LLMs for tasks needing world knowledge or deep semantics.

What carries the argument

The argument is carried by the experimental protocol: four fine-tuned bidirectional transformers compared against two LLM-based families (zero-shot prompting and internal-state probes, the latter enhanced with Prism) across six datasets. The interpretability analysis uses PCA on hidden states and single linear probes at each layer, showing that BERT-like models produce linearly separable CLS embeddings on pattern-driven tasks while LLM hidden states mix classes under obfuscation. TaMAS, the proposed strategy, converts these observations into a decision rule: use BERT-like models for general or rule-based classification, and LLMs for knowledge-intensive or semantically deep tasks.

What would settle it

Re-run the same six test sets with a 7B/8B LLM fine-tuned (parameter-efficiently) on the same training labels used for the BERT models, keeping metrics fixed; if it beats BERT-like models on the ToxiCloakCN variants or on a majority of tasks, the central claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that for most of the six datasets it studies, fine-tuned BERT-like models (BERT, RoBERTa, ERNIE, ELECTRA) yield higher or comparable accuracy, AUC, and F1 than LLM-based methods, even though the LLM methods include state-of-the-art internal-state probes. The ToxiCloakCN implicit hate speech variants (base, emoji, homophone) are won by BERT-like models by large margins; legal-text and malicious-code classifications are close between BERT-like models and LLM internal-state probes, with direct zero-shot prompting clearly behind; and hallucination detection is the one task where LLM internal-state methods win. The paper interprets this as evidence that the 'LLM-centric' trend in text classification has overlooked a cheap, strong alternative, and that the correct model choice depends on dataset properties rather than on the latest model family.

Load-bearing premise

The comparison is asymmetric: BERT-like models are fine-tuned on the training labels, while the open-weight LLMs are used zero-shot or with probes trained on those labels but never fine-tuned, so a stronger or fine-tuned LLM could change the ranking.

Editorial extensions

If this is right

  • Fine-tuned BERT-like models remain a cost-effective default for many text classification tasks, especially when labels are available.
  • Datasets can be triaged by type—pattern-driven, rule-based, knowledge-intensive—before choosing a model family.
  • LLM internal-state probes are a viable middle ground on rule-based and knowledge-intensive tasks, outperforming zero-shot prompting.
  • The pattern-driven advantage of BERT-like models persists even under homophone and emoji perturbations in the tested data.
  • The proposed TaMAS rule offers a practical way to avoid wasteful LLM usage when an encoder would do.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the comparison were made symmetric by fine-tuning the 7B/8B LLMs, the BERT advantage on pattern-driven tasks might shrink but could still hold, since the paper's PCA evidence points to a representational difference, not just a fine-tuning gap.
  • The taxonomy might extend beyond these six datasets: tasks with community-specific jargon, obfuscation, or sparse labels are plausible candidates for BERT-like models.
  • A cheap, practical takeaway is to run a quick linear-probe separability check on a small sample of hidden states before committing to an LLM pipeline.
  • The paper's rule could be converted into a decision tree or scoring function for new datasets, though the current evidence only supports six tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper compares four fine-tuned BERT-like models (BERT, RoBERTa, ERNIE, ELECTRA) against LLM-based approaches—zero-shot Qwen/LLaMA, GPT-4o querying, and internal-state probes SAPLMA and MM-Probe—on six text classification datasets: three ToxiCloakCN hate-speech variants, LegalText, MaliciousCode, and Hallucination. The central claim is that BERT-like models often outperform LLMs, and the paper further categorizes the six datasets into three types (pattern-driven, rule-based clean, knowledge-intensive) and proposes TaMAS, a task-aware model selection strategy. Table 1 supports the narrow claim that fine-tuned BERT-like models beat the specific zero-shot and frozen-probe LLM methods on the majority of these six datasets, but the broad generalizations in the abstract and Section 4 go beyond what the experimental setup can establish.

Significance. If the findings hold, the paper provides a useful empirical counterpoint to the prevailing 'LLM-centric' narrative, with a compact benchmark covering six moderately challenging classification tasks and an interpretable analysis based on PCA and layerwise probing. The potential strength is that the raw performance table is directly reproducible from the described fixed splits, standard BERT variants, and publicly available datasets. However, the significance is limited by the asymmetric comparison (fine-tuned BERTs vs. zero-shot/frozen-probe LLMs), the absence of uncertainty estimates, and the post hoc derivation of the taxonomy and selection strategy from the same six datasets. The paper is a reasonable empirical study whose claims need to be narrowed or supplemented before they can support the abstract's broad conclusion.

major comments (3)
  1. [§2.1, §B, Abstract] The comparison is asymmetric: BERT-like models are fine-tuned on the training split, whereas the LLM baselines are either zero-shot prompted or used only through internal-state probes with a frozen backbone. The authors state in §B, 'We do not include methods that involve fine-tuning the parameters of the LLMs,' yet the abstract concludes that 'BERT-like models often outperform LLMs.' This conclusion is not supported for fine-tuned LLMs; a LoRA or QLoRA fine-tuned 7B/8B model could plausibly change the ranking on the four datasets where BERT currently leads. The claim should be narrowed to 'fine-tuned BERT-like models outperform zero-shot and frozen-representation LLM methods on these six datasets,' or the experiments should include a fine-tuned LLM baseline.
  2. [Table 1] All reported scores are single runs with no standard deviations, significance tests, or multiple seeds. For LegalText, the AUC gap between RoBERTa (99.2) and SAPLMA (97.7) is small, and for MaliciousCode the F1 scores are tied at 99.7 across several methods. Without uncertainty estimates, the claim that BERT-like models 'consistently outperform' LLM-based approaches is not robust, especially on tasks with near-ceiling performance. At minimum, the authors should report averages and standard deviations over multiple seeds and perform a significance test on the key comparisons.
  3. [§3, §4] The three-way dataset taxonomy and the TaMAS strategy are derived from the same six datasets on which they are evaluated. The paper does not validate the taxonomy on held-out datasets, and TaMAS is not operationalized: it does not provide a decision rule to determine, for a new text classification task, whether the pattern-driven, rule-based, or knowledge-intensive category applies without first running both model families. As a result, the proposed strategy is not falsifiable in its current form. The authors should either formalize a concrete, a priori criterion for assigning a task to a category, or present evidence on additional datasets to show the taxonomy and the selection strategy generalize.
minor comments (5)
  1. [§5, §D, §3] There are several typos: 'proposTaMAS' in §5 should be 'propose TaMAS', 'MaliciousCo' in §D should be 'MaliciousCode', and 'Perturbated' in §3 should be 'Perturbed'.
  2. [Table 1 caption] The abbreviations 'Cat.', 'Meth./Met.', 'BLMs', 'LLM-IS', and 'LLM-Q' are used in the table but not defined in the caption; they should be spelled out at first use.
  3. [Figures 2 and 3] The PCA and probing figures are qualitative; the authors should report a quantitative separability metric, such as linear probe accuracy or silhouette score, to support the claims about linear separability.
  4. [Table 1, §2.1] The GPT-4o row reports only F1 scores on the three ToxiCloakCN splits and is absent for the other three datasets. If these numbers are taken from Xiao et al. (2024), the citation should be explicit in the table; if they are new runs, the missing metrics and datasets should be completed.
  5. [General] No code or data availability statement is provided; given the small scale of the experiments, releasing code would aid reproducibility and make the comparisons easier to verify.

Circularity Check

1 steps flagged · score 2.0 of 10

Core benchmark comparison is independent, but the TaMAS taxonomy is a post-hoc restatement of the observed winners on the same six datasets.

  1. renaming known result [Section 3 (Analysis & Discussions) and Section 4 (TaMAS)]
    "Based on experimental results, this chapter classifies six datasets into three categories and analyzes performance using model hidden states PCA visualization and probing. | ForGeneral Text Classificationtasks that primarily rely on surface-form patterns or shallow semantic features, BERT-like models demonstrate superior parameter efficiency and performance."

    The three-way taxonomy is induced from the observed performance rankings on the same six datasets: ToxiCloakCN tasks (where BERT wins) are labelled 'pattern-driven', code/legal tasks (competitive) are 'rule-based', and the Hallucination task (where LLM internal-state methods win) is 'knowledge-intensive/semantic-deep'. TaMAS then 'reveals' that BERT-like models excel on 'General Text Classification' (pattern-driven) tasks and LLMs excel on knowledge/semantic tasks. Because the category labels are assigned to exactly the datasets where each method already won, the strategy's recommendation is a restatement of Table 1 rather than an independent prediction. No held-out dataset tests the taxonomy, so the proposed selection rule adds no information beyond the inputs used to construct it.

full rationale

The central empirical comparison in Table 1 is independent, externally sourced evidence: the six datasets and baseline methods (BERT/RoBERTa/ERNIE/ELECTRA, SAPLMA, MM-Probe, Qwen2.5-7B-Instruct, LLaMA-3-8B-Instruct, GPT-4o) are public resources, and the reported AUC/Acc/F1 values come from running those systems, not from fitting a parameter that is then renamed as a result. The comparison is asymmetric (fine-tuned BERTs versus zero-shot and fixed-probe 7B/8B LLMs), which limits the breadth of the abstract's 'BERT-like models often outperform LLMs' generalization, but this is a scope/correctness limitation rather than a circular derivation. No load-bearing self-citation chain appears; the cited methods and datasets are from external groups. The one circular-adjacent element is TaMAS: the three dataset types are defined after inspecting the same six performance results, and the strategy's recommendation (use BERT for pattern-driven tasks, LLMs for knowledge/semantic tasks) restates those winners. That post-hoc labeling is a real weakness for generalization, but it does not undermine the underlying benchmark, which stands on its own.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central comparison rests on a small, hand-picked dataset set and on interpretability tools (PCA, linear probes) as evidence for mechanism. The only hand-set parameters are the LLM-IS layer choice and the MM-Probe threshold; no new physical, mathematical, or conceptual entities are introduced. The most important unstated cost is the asymmetry between fine-tuned BERT models and non-fine-tuned LLMs.

free parameters (2)
  • LLM internal-state layer index = 3/4 of total layers
    SAPLMA and MM-Probe use the 3/4th layer for all datasets (Appendix C), chosen based on prior literature rather than tuned per dataset. Performance of internal-state methods depends on this choice.
  • MM-Probe classification threshold = Maximum G-Mean on the training set
    The threshold for MM-Probe is set by maximizing G-Mean on the training distribution (Appendix C). This affects reported accuracy and F1 scores.
assumptions (4)
  • domain assumption The six selected datasets are representative of text classification difficulty categories
    Used to generalize from the benchmark to the three task types and to TaMAS in Sections 3 and 4.
  • domain assumption Linear separability of hidden states, as measured by PCA and linear probes, is a valid explanation for why one method wins
    Section 3 interprets Figures 2 and 3 as causal evidence for BERT strength on pattern tasks and LLM strength on knowledge tasks.
  • domain assumption Zero-shot prompting and probe-based internal-state extraction are the relevant LLM baselines; LLM fine-tuning is excluded
    Section 2.1 and Appendix B justify the exclusion by computational cost, but this scoping is load-bearing for the central claim.
  • domain assumption A single 7:1.5:1.5 split with validation-based model selection is sufficient to estimate generalization
    Appendix B describes the split but no seeds or repeated runs, so variance across splits is not assessed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do BERT-Like Bidirectional Models Still Perform Better on Text Classification in the Era of LLMs?." pith.science (2026). https://pith.science/paper/XLAWNSSI

@misc{pith2026250518215,
  author       = {Pith},
  title        = {Pith review of: Do BERT-Like Bidirectional Models Still Perform Better on Text Classification in the Era of LLMs?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XLAWNSSI}},
  note         = {Machine review of arXiv:2505.18215}
}
read the original abstract

The rapid adoption of LLMs has overshadowed the potential advantages of traditional BERT-like models in text classification. This study challenges the prevailing "LLM-centric" trend by systematically comparing three category methods, i.e., BERT-like models fine-tuning, LLM internal state utilization, and zero-shot inference across six high-difficulty datasets. Our findings reveal that BERT-like models often outperform LLMs. We further categorize datasets into three types, perform PCA and probing experiments, and identify task-specific model strengths: BERT-like models excel in pattern-driven tasks, while LLMs dominate those requiring deep semantics or world knowledge. Based on this, we propose TaMAS, a fine-grained task selection strategy, advocating for a nuanced, task-driven approach over a one-size-fits-all reliance on LLMs.

Figures

Figures reproduced from arXiv: 2505.18215 by the authors.

Figure 1
Figure 1. Illustration of our fine-grained task selection [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparative PCA visualization of hidden states across six datasets: BERT-like models vs. LLMs. T-BASE, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Comparative visualization of hidden states [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Analyzing developer discussions on EU and US privacy legislation compliance in GitHub repositories

    cs.SE 2025-12 conditional novelty 6.0 of 10

    Across 32,820 GitHub issues, developer privacy-law compliance discussions center on consent, user-rights functionality, bugs, and cookies, with erasure, opt-out, and access the most-discussed legal rights.

  2. GENUINE: Graph Enhanced Multi-level Uncertainty Estimation for Large Language Models

    cs.CL 2025-09 conditional novelty 6.0 of 10

    GENUINE uses dependency parse trees and learnable graph pooling to produce uncertainty scores for LLM outputs, claiming AUROC gains of up to 29% over semantic entropy baselines.

Reference graph

Works this paper leans on

24 extracted references · 2 canonical work pages · cited by 2 Pith papers

  1. [1]

    Guillaume Alain and Yoshua Bengio. 2016. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644

    arXiv 2016

  2. [2]

    Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in language models is mediated by a single direction. arXiv preprint arXiv:2406.11717

    arXiv 2024

  3. [3]

    Amos Azaria and Tom Mitchell. 2023. The internal state of an llm knows when it's lying. arXiv preprint arXiv:2304.13734

    arXiv 2023

  4. [4]

    Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. 2024. Inside: Llms' internal states retain the power of hallucination detection. arXiv preprint arXiv:2402.03744

    arXiv 2024

  5. [5]

    Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555

    arXiv 2020

  6. [6]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171--4186

    2019

  7. [7]

    Er1111c. 2024. Malicious code classification. https://huggingface.co/datasets/Er1111c/Malicious_code_classification

    work page 2024

  8. [8]

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and 1 others. 2023. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674

    arXiv 2023

Show all 24 references
  1. [9]

    Guangyuan Jiang, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. 2023. Evaluating and inducing personality in pre-trained language models. Advances in Neural Information Processing Systems, 36:10622--10643

    2023

  2. [10]

    Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024. Llms-as-judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579

    2024 arXiv

  3. [11]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692

    2019 arXiv

  4. [12]

    Samuel Marks and Max Tegmark. 2023. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824

    2023 arXiv

  5. [13]

    openSUSE. 2025. Cavil legal text dataset. https://huggingface.co/datasets/openSUSE/cavil-legal-text

    2025

  6. [14]

    Oscar Skean, Md Rifat Arefin, Yann LeCun, and Ravid Shwartz-Ziv. 2024. Does representation matter? exploring intermediate layers in large language models. arXiv preprint arXiv:2412.09563

    2024 arXiv

  7. [15]

    Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, and 1 others. 2021. Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. arXiv preprint arXiv:2107.02137

    2021 arXiv

  8. [16]

    Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2020. Ernie 2.0: A continual pre-training framework for language understanding. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 8968--8975

    2020

  9. [17]

    Sowmya Vajjala and Shwetali Shimangaud. 2025. Text classification in the llm era--where do we stand? arXiv preprint arXiv:2502.11830

    2025 arXiv

  10. [18]

    Yunze Xiao, Yujia Hu, Kenny Tsu Wei Choo, and Roy Ka-wei Lee. 2024. Toxicloakcn: Evaluating robustness of offensive language detection in chinese with cloaking perturbations. arXiv preprint arXiv:2406.12223

    2024 arXiv

  11. [19]

    Junlin Xie, Zhihong Chen, Ruifei Zhang, Xiang Wan, and Guanbin Li. 2024. Large multimodal agents: A survey. arXiv preprint arXiv:2402.15116

    2024 arXiv

  12. [20]

    Fujie Zhang, Peiqi Yu, Biao Yi, Baolei Zhang, Tong Li, and Zheli Liu. 2024 a . Prompt-guided internal states for hallucination detection of large language models. arXiv preprint arXiv:2411.04847

    2024 arXiv

  13. [21]

    Tianhua Zhang, Hongyin Luo, Yung-Sung Chuang, Wei Fang, Luc Gaitskell, Thomas Hartvigsen, Xixin Wu, Danny Fox, Helen Meng, and James Glass. 2023. Interpretable unified language checking. arXiv preprint arXiv:2304.03728

    2023 arXiv

  14. [22]

    Yazhou Zhang, Mengyao Wang, Chenyu Ren, Qiuchi Li, Prayag Tiwari, Benyou Wang, and Jing Qin. 2024 b . Pushing the limit of llm capacity for text classification. arXiv preprint arXiv:2402.07470

    2024 arXiv

  15. [23]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  16. [24]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.