Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

Text Classification: Neural Networks VS Machine Learning Models VS Pre-trained Models

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read On a 10,917-article news dataset, fine-tuned pretrained transformers such as BERT and DistilBERT consistently outperform every classical neural network and machine learning model tested.

desk verdict A competent course-project benchmark whose pretrained-vs-classical comparison is undermined by test-set-based model selection; the GloVe-over-TF-IDF result is the only robust finding. read the letter →

arxiv 2412.21022 v1 pith:CNERX2K6 submitted 2024-12-30 cs.LG

Christos Petridis This is my paper · ORCID
classification cs.LG
keywords TextclassificationTransformersPre-trainedmodelsNaturallanguageprocessingEmbeddingsGloVeTF-IDFTransferlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper compares 13 text classifiers on a balanced news dataset with 17 coarse and 109 fine categories, asking which approach gives the best accuracy. Its answer is that fine-tuned pretrained transformers, led by BERT at 85.16% accuracy on coarse labels, beat every classical model; the best classical result is SVM with GloVe embeddings at 72.11%. It also finds that GloVe embeddings consistently outperform TF-IDF across all classical models. If these results hold, practitioners should default to fine-tuning a pretrained transformer when accuracy matters, and to GloVe-based classical models only when compute or deployment simplicity is the priority.

What carries the argument

The load-bearing object is the evaluation pipeline itself: a single cleaned text column built from source, title, content, and author, fed to classical models as either TF-IDF or 100-dimensional GloVe vectors, and to seven pretrained transformers through their own tokenizers, all scored by test accuracy on the same split. The mechanism that produces the paper's result is transfer learning: pretrained transformers carry general language representations into the task, whereas classical models depend on the quality of the static embeddings they are given.

What would settle it

Retrain ALBERT on the level-2 task with a validation-based early-stopping rule or a substantially larger epoch budget; if accuracy climbs from 0.0105 into the 0.60-0.75 range typical of the other pretrained models, then the paper's ranking among pretrained models, and possibly its broad 'pretrained transformers outperform' generalization, rests on a training-budget artifact.

Watch

Extended reading notes

Core claim

The paper's central claim is that pretraining plus fine-tuning dominates feature engineering on this task: each of the seven pretrained transformer models clears every classical model on both classification granularities. On the coarse level-1 task, BERT reaches 0.8516 accuracy, RoBERTa 0.8457, and DistilBERT 0.8324, compared with at most 0.7211 for the best classical classifier (SVM with GloVe). On the fine level-2 task, DistilBERT reaches 0.7381 and BERT 0.7321, versus 0.5585 for the best classical model (Logistic Regression with GloVe). The authors further claim that GloVe is consistently better than TF-IDF for all classical models, and that accuracy for every model drops as the number of classes grows from 17 to 109.

Load-bearing premise

The evaluation assumes a fixed six-epoch fine-tuning budget is enough for every pretrained model to reach its representable accuracy, an assumption that ALBERT's near-zero level-2 score calls into question.

Editorial extensions

If this is right

  • Fine-tuned pretrained transformers are the default high-accuracy choice for this kind of news-classification task, with BERT at 0.8516 on level-1 and DistilBERT at 0.7381 on level-2.
  • For classical pipelines, the embedding choice matters more than the algorithm: GloVe beats TF-IDF by roughly 20 accuracy points across level-1 models.
  • The fine-grained 109-class task is harder for all models, and the accuracy drop is large enough that applications needing fine categories should budget for lower accuracy or more data.
  • When compute is limited, DistilBERT offers most of BERT's accuracy in about half the training time and with fewer parameters, while classical GloVe-based models are the cheapest deployable option if roughly 0.72 level-1 accuracy suffices.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If ALBERT's near-zero level-2 score is a training-budget artifact, then the ranking among pretrained models could change with more epochs, even though the broad pretrained-versus-classical conclusion would likely survive.
  • The same protocol could be extended to imbalanced or much smaller datasets; the current dataset's near-perfect balance and roughly 100 examples per level-2 class is a favorable regime for all models.
  • A natural next experiment is an accuracy-versus-compute frontier across epoch budgets, which would tell practitioners whether the extra cost of BERT-class models buys accuracy that matters for their deployment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper reports an empirical comparison of text classification methods on a news dataset with two label granularities (17 level-1 categories and 109 level-2 categories). It compares three traditional neural networks (MLP, RNN, TransformerEncoder) and three classical machine learning models (SVM, Random Forest, Logistic Regression), each with TF-IDF and GloVe embeddings, against seven fine-tuned pretrained transformer models (BERT, DistilBERT, RoBERTa, XLM-RoBERTa, ELECTRA, TinyBERT, ALBERT). The central claim is that pretrained transformer models outperform all traditional models on both classification levels, and that GloVe embeddings consistently outperform TF-IDF for the non-pretrained models. The paper also discusses the level-2 ALBERT result of 0.0105 accuracy as anomalous and attributes it to an insufficient fine-tuning budget.

Significance. If the results were obtained with a sound evaluation protocol, this would be a useful, if not highly novel, benchmark comparison: it covers a broad set of seven pretrained transformers, three neural networks, and three classical classifiers on a fine-grained multi-class task, and it explicitly evaluates two embedding types for the non-pretrained models. The most robust finding is the GloVe-over-TF-IDF advantage, which holds for every classifier and both label levels and is consistent with prior work. The paper is clearly structured and the tables are readable. Its main weakness is not the qualitative message, which is plausible and consistent with the broader literature, but the evaluation protocol, which prevents the reported accuracy numbers from being treated as unbiased estimates of generalization. The paper also provides no statistical uncertainty quantification for the neural and pretrained-model results, unlike the cross-validated classical baselines.

major comments (3)
  1. [§6.1, §6.2, §8.3, §8.4] The comparison is not apples-to-apples because the neural-network and pretrained-model accuracies are selected using the same test set on which they are reported, while the classical-model accuracies come from cross-validation. Section 6.1 states that the learning-rate scheduler (ReduceLROnPlateau) adjusts 'based on the model's performance on the test set,' and Section 6.2 reports accuracy on that same test set. For pretrained models, Section 8.3 says the model is evaluated on a 'separate (unseen) test dataset' and weights are saved whenever test accuracy improves, with the final accuracy reported on that same test set in Section 8.4. No held-out validation split is described anywhere. By contrast, the classical models use 3-fold cross-validation for hyperparameter selection and 5-fold cross-validation for the reported numbers (Sections 7.1 and 7.3). The reported pretrained and neural-network numbers are therefore in-sample selections over checkpoints, whereas the classical numbers are cross-validated estimates. This asymmetry means the central claim that pretrained models outperform traditional models is not established by the reported numbers, even though the qualitative conclusion may well be true. The paper should introduce a proper train/validation/test split, use the validation set for learning-rate scheduling and checkpoint selection, and report on a genuinely held-out test set.
  2. [§8.3, Table 8, §10] The ALBERT level-2 result of 0.0105 accuracy is essentially chance (1/109 ≈ 0.0092), and the paper acknowledges in Section 10 that ALBERT 'probably needs more epochs during fine-tuning' but that the GPU budget did not allow further experiments. Because the checkpoint-selection rule in Section 8.3 saves weights only on test-accuracy improvement, a model that never improves on the test set over six epochs will be evaluated from a degenerate or early state. This is a training-budget artifact, not a meaningful measurement of ALBERT's representable performance. As a result, the ranking among pretrained models (Table 8) and the claim that all seven pretrained models outperform traditional models are not supported. The authors should either fine-tune with a validation-based early-stopping budget sufficient for all models, or report ALBERT level-2 as a failed run and exclude it from the aggregate conclusion.
  3. [§8.4, Tables 7–10] The pretrained-model and neural-network results are single-run accuracies with no standard deviations, confidence intervals, or seeds, whereas the classical-model results are means over 5-fold cross-validation. This makes it impossible to assess whether differences such as BERT's 0.8516 versus XLM-RoBERTa's 0.8214 on level-1, or DistilBERT's 0.7381 versus BERT's 0.7321 on level-2, are meaningful. At minimum, the authors should report multiple seeds (or cross-validation folds) for all deep models and provide a significance test or error bars before making fine-grained claims about which pretrained model performs best.
minor comments (6)
  1. [§4] The sentence 'Table 1 demonstrates some details regarding the dataset and briely describes each column' contains a typo ('briely' should be 'briefly').
  2. [§10] The text refers to '107 classes instead of 17 in level-1', but Section 4.1 states there are 109 level-2 categories; this number should be consistent.
  3. [Table 10 vs Table 8] Table 10 lists ALBERT's level-2 accuracy as 0.010, while Table 8 reports 0.0105; the two tables should use the same rounded or full value.
  4. [Eq. (1)] The denominator of the term-frequency formula is garbled by the formatting; it should read sum over all terms in the document, and the typesetting should be corrected.
  5. [§8.2] The paper mentions that the source code is available in delivered Python notebooks, but does not provide a URL or repository identifier; for reproducibility, the code should be linked or uploaded to a persistent archive.
  6. [Title and header] The manuscript header states 'This paper is NOT published' and includes the course name and instructor; this scaffolding should be removed if the paper is submitted to a journal.

Circularity Check

2 steps flagged · score 6.0 of 10

Test-set accuracy is used for checkpoint and LR selection and then reported as final performance, making the pretrained-vs-classical comparison partially circular.

  1. fitted input called prediction [Section 8.3-8.4, Tables 7-8 (pre-trained models)]
    "During each epoch, the function tracks the training loss and evaluates the model’s performance on a separate (unseen) test dataset, calculating accuracy. If the model achieves a significantly better accuracy during training, its weights are saved and we can restore them. At the end of training, the function restores these best weights for a final evaluation... After training, the best model weights are restored, ensuring optimal performance. The final accuracies after training are shown in Table 7 and Table 8."

    The reported 'final accuracy' is not an independent prediction: the same test split is used to select the best checkpoint, and then the selected model is evaluated on that same split. By construction, the Table 7/8 accuracy is the best (or near-best) test-set accuracy observed during the six epochs, i.e., a selection statistic rather than a held-out generalization estimate. Since the classical ML models are instead reported as 5-fold cross-validation means (Section 7.3), the headline pretrained-vs-classical ranking in Section 10 is driven in part by this in-sample selection, so the evidence for the central claim is partially circular.

  2. fitted input called prediction [Sections 6.1-6.2, Table 2 (standard neural networks)]
    "We apply the torch.optim.lr_scheduler.ReduceLROnPlateau as the learning rate scheduler, which dynamically adjusts the learning rate based on the model’s performance on the test set... After each epoch, the model is being evaluated on unseen data from the test set, where its predictions are compared to the actual labels to calculate accuracy."

    For MLP, RNN, and TransformerEncoder, both the learning-rate schedule and the saved best model state are chosen using test-set accuracy, and Table 2 then reports 'final test accuracy' on that same test set. Thus the neural-network numbers are in-sample selections over training epochs, not unbiased predictions; comparing them to the cross-validated ML results in Tables 4-5 does not provide an apples-to-apples test of the paper's ranking.

full rationale

The paper contains no mathematical derivation or fitted parameter that is renamed as a prediction. Its central conclusion is an empirical summary of measured accuracies, so the usual circularity patterns (self-definition, self-citation chains, imported uniqueness theorems, ansatz-by-citation) are absent. The one load-bearing circular element is the evaluation protocol: for the seven pretrained models and the three standard neural networks, the same test split is used for checkpoint selection (and LR scheduling) and then for the reported final accuracy; by construction those numbers are selected maxima on the test set rather than held-out estimates. The classical ML models, by contrast, are reported via cross-validation. Consequently, the Section 10 claim that pretrained models outperform traditional models rests in part on comparing a test-set-selected statistic against cross-validated statistics, so the comparison is partially circular as an evaluation. This does not prove the ranking false, and the paper openly notes the ALBERT anomaly and fixed-epoch budget as limitations (Section 10), but it does mean the headline numbers are not independent evidence of generalization.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are needed because the paper makes no theoretical derivation; the experimental hyperparameters (embedding dimension, epoch counts, learning rates) are implementation choices rather than fitted constants in a model. The axioms capture the empirical assumptions the benchmark depends on: preprocessing preserves class information, all models receive sufficient training, and accuracy is a meaningful metric on this balanced dataset.

assumptions (3)
  • domain assumption The merged and cleaned text features (source, title, content, author) are sufficient inputs for all compared models.
    Section 5.1 discards id, timestamps, and URLs, assuming content alone determines category. This is plausible but unverified.
  • domain assumption A fixed training budget (150 epochs for standard NNs, 6 epochs for fine-tuning) is sufficient for all models to converge.
    Adopted in Sections 6.1 and 8.3; the ALBERT level-2 collapse (0.0105) suggests this budget is not sufficient for every pretrained model.
  • domain assumption Accuracy is an adequate metric for ranking models on this class-imbalance-free dataset.
    Section 4.1 states the dataset is balanced with 100 instances per level-2 category, so accuracy is a meaningful summary.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Text Classification: Neural Networks VS Machine Learning Models VS Pre-trained Models." pith.science (2026). https://pith.science/paper/CNERX2K6

@misc{pith2026241221022,
  author       = {Pith},
  title        = {Pith review of: Text Classification: Neural Networks VS Machine Learning Models VS Pre-trained Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CNERX2K6}},
  note         = {Machine review of arXiv:2412.21022}
}
read the original abstract

Text classification is a very common task nowadays and there are many efficient methods and algorithms that we can employ to accomplish it. Transformers have revolutionized the field of deep learning, particularly in Natural Language Processing (NLP) and have rapidly expanded to other domains such as computer vision, time-series analysis and more. The transformer model was firstly introduced in the context of machine translation and its architecture relies on self-attention mechanisms to capture complex relationships within data sequences. It is able to handle long-range dependencies more effectively than traditional neural networks (such as Recurrent Neural Networks and Multilayer Perceptrons). In this work, we present a comparison between different techniques to perform text classification. We take into consideration seven pre-trained models, three standard neural networks and three machine learning models. For standard neural networks and machine learning models we also compare two embedding techniques: TF-IDF and GloVe, with the latter consistently outperforming the former. Finally, we demonstrate the results from our experiments where pre-trained models such as BERT and DistilBERT always perform better than standard models/algorithms.

Figures

Figures reproduced from arXiv: 2412.21022 by the authors.

Figure 1
Figure 1. Distribution of the classes in our dataset. It is also [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Training Loss (on the left) and Test Accuracy (on [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Training Loss (on the left) and Test Accuracy (on [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Training Loss (on the left) and Test Accuracy (on [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Training Loss (on the left) and Test Accuracy (on [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Pipeline for preparing the data for classifiers (neural [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 8
Figure 8. Figure 8: Test accuracy during training (6 epochs). [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 7
Figure 7. Figure 7: Test accuracy during training (6 epochs). [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Extracting Overlapping Microservices from Monolithic Code via Deep Semantic Embeddings and Graph Neural Network-Based Soft Clustering

    cs.SE 2025-08 reject novelty 6.0 of 10

    Mo2oM assigns classes to overlapping microservices using UniXcoder embeddings and NOCD soft clustering, claiming large gains in modularity metrics over hard-clustering baselines on four monoliths.

  2. CaresAI at SMM4H-HeaRD 2026: Predicting TNM Staging

    cs.CL 2026-07 conditional novelty 2.0 of 10

    LightGBM with TF-IDF predicts T/N/M stage from TCGA pathology reports with strong internal AUROC but weaker generalization on a second held-out test set.

Reference graph

Works this paper leans on

23 extracted references · 12 canonical work pages · cited by 2 Pith papers

  1. [1]

    Sabeen Ahmed, Ian E Nielsen, Aakash Tripathi, Shamoon Siddiqui, Ravi P Ra- machandran, and Ghulam Rasool. 2023. Transformers in time-series analysis: A tutorial. Circuits, Systems, and Signal Processing 42, 12 (2023), 7433–7466

    work page 2023

  2. [2]

    Anguita, Luca Ghelardoni, Alessandro Ghio, L

    D. Anguita, Luca Ghelardoni, Alessandro Ghio, L. Oneto, and Sandro Ridella

    work page

  3. [3]

    K Clark. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555 (2020)

    arXiv 2020

  4. [4]

    A Conneau. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116 (2019)

    arXiv 2019

  5. [5]

    Diogo Cortiz. 2022. Exploring Transformers models for Emotion Recognition: a comparision of BERT, DistilBERT, RoBERTa, XLNET and ELECTRA. InProceed- ings of the 2022 3rd International Conference on Control, Robotics and Intelligent System (Virtual Event, China) (CCRIS ’22). Association for Computing Machinery, New York, NY, USA, 230–234. https://doi.org/1...

    work page arXiv 2022

  6. [6]

    Sonain Jamil, Md Jalil Piran, and Oh-Jin Kwon. 2023. A comprehensive survey of transformers for computer vision. Drones 7, 5 (2023), 287

    work page 2023

  7. [7]

    Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351 (2019)

    arXiv 2019

  8. [8]

    Thorsten Joachims et al. 1997. A probabilistic analysis of the Rocchio algorithm with TFIDF for text categorization. In ICML, Vol. 97. Citeseer, 143–151

    work page 1997

Show all 23 references
  1. [9]

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, Vol. 1. Minneapolis, Minnesota, 2

    2019

  2. [10]

    Kamran Kowsari, Kiana Jafari Meimandi, Mojtaba Heidarysafa, Sanjana Mendu, Laura Barnes, and Donald Brown. 2019. Text classification algorithms: A survey. Information 10, 4 (2019), 150

    2019

  3. [11]

    Z Lan. 2019. Albert: A lite bert for self-supervised learning of language represen- tations. arXiv preprint arXiv:1909.11942 (2019)

    2019 arXiv

  4. [12]

    Yinhan Liu. 2019. Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692 364 (2019)

    2019 arXiv

  5. [13]

    Larry R Medsker, Lakhmi Jain, et al. 2001. Recurrent neural networks. Design and Applications 5, 64-67 (2001), 2

    2001

  6. [14]

    Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Narjes Nikzad, Meysam Chenaghlu, and Jianfeng Gao. 2021. Deep learning–based text classification: a comprehensive review. ACM computing surveys (CSUR) 54, 3 (2021), 1–40

    2021

  7. [15]

    Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) . 1532–1543

    2014

  8. [16]

    Philipp Probst, Anne-Laure Boulesteix, and Bernd Bischl. 2019. Tunability: Im- portance of hyperparameters of machine learning algorithms. Journal of Machine Learning Research 20, 53 (2019), 1–32

    2019

  9. [17]

    Nihar Ranjan, Kaushal Mundada, Kunal Phaltane, and Saim Ahmad. 2016. A Survey on Techniques in NLP. International Journal of Computer Applications 134, 8 (2016), 6–9

    2016

  10. [18]

    V Sanh. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019)

    2019 arXiv

  11. [19]

    Andreas Steiner, André Susano Pinto, Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey Gritsenko, Matthias Minderer, Anthony Sherbondy, Shangbang Long, et al . 2024. PaliGemma 2: A Family of Versatile VLMs for Transfer. arXiv preprint arXiv:2412.03555 (2024)

    2024 arXiv

  12. [20]

    Xiaobing Sun, Xiangyue Liu, Jiajun Hu, and Junwu Zhu. 2014. Empirical studies on the nlp techniques for source code data preprocessing. In Proceedings of the 2014 3rd international workshop on evidential assessment of software technologies . 32–39

    2014

  13. [21]

    Hind Taud and Jean-Franccois Mas. 2018. Multilayer perceptron (MLP).Geomatic approaches for modeling land change scenarios (2018), 451–455

    2018

  14. [22]

    A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems (2017)

    2017

  15. [2012]

    InThe European Symposium on Artificial Neural Networks

    The ’K’ in K-fold Cross Validation. InThe European Symposium on Artificial Neural Networks

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.