{"id":"ce8c2e31-1e72-4285-9f39-0bbea125ccea","arxiv_id":"2505.18215","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Fine-tuned BERT-like models outperform zero-shot and internal-state LLM methods on four of six challenging text classification datasets.","lead":"This paper compares fine-tuned BERT-style models with LLM-based methods on six text classification datasets. It reports that BERT-style models often win on pattern-driven tasks, and proposes a rule for when to use each approach.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on an asymmetric comparison: fine-tuned BERT models are pitted against zero-shot and internal-state 7B/8B LLMs; adding LLM fine-tuning or stronger models could overturn the rankings.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing issue: the LLM side is limited to zero-shot prompting and fixed internal-state probes on 7B/8B open-weight models, while BERT-like models receive full supervised fine-tuning. This asymmetry is explicit in the paper, but it directly controls whether the broad statement 'BERT-like models often outperform LLMs' holds. If the comparison were extended to include a fine-tuned LLM, the rankings on the 'pattern-driven' tasks (ToxiCloakCN variants, LegalText, MaliciousCode) could plausibly change, since LLM fine-tuning has been shown to adapt well to classification tasks. The paper's own evidence is not internally inconsistent—Table 1 supports that fine-tuned BERTs beat the chosen LLM baselines on four of six tasks—but the abstract and conclusion make a stronger claim than the evidence supports. The missing GPT-4o results in Table 1 and the absence of multiple seeds or error bars exacerbate the problem. These are addressable limitations rather than fatal flaws, so a CONDITIONAL verdict is appropriate; since the reader already arrived at CONDITIONAL, my stress-test does not change the verdict. The check I propose is direct: fine-tune the same LLMs and see if the ranking reverses. If it does, the headline must be narrowed; if it does not, the paper's central claim survives this particular objection.","tokens_in":7827,"tokens_out":3821,"duration_ms":36253,"concrete_test":"Run LoRA or QLoRA fine-tuning of Qwen2.5-7B-Instruct and LLaMA-3-8B-Instruct on all six tasks using the same 7:1.5:1.5 splits and report AUC/Acc/F1. If any fine-tuned LLM matches or exceeds the best BERT-like model on ToxiCloakCNBase, ToxiCloakCNEmoji, ToxiCloakCNHomo, or LegalText, then the claim that BERT-like models 'consistently outperform LLM-based approaches' fails for fine-tuned LLMs and must be restricted to zero-shot prompting. Reporting per-seed statistics across at least five seeds would also clarify whether the observed margins are stable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim—'BERT-like models often outperform LLMs'—is supported only for the specific, narrow LLM instantiations used: Qwen2.5-7B-Instruct and LLaMA-3-8B-Instruct in zero-shot prompting, plus internal-state probes (SAPLMA, MM-Probe) trained on the same training labels. Meanwhile, all BERT-like models are fine-tuned on the 70% training split (Section 2.1, Appendix B). The authors explicitly exclude LLM fine-tuning for cost reasons, but this makes the abstract and Section 4's 'consistently outperform LLM-based approaches' too broad. The comparison does not include a single fine-tuned LLM, and the one strong API model tried (GPT-4o) has incomplete results (only F1 on three ToxiCloakCN tasks, Table 1). If a LoRA/QLoRA-fine-tuned 7B/8B model or a stronger zero-shot model outperforms BERT on the four tasks where BERT currently leads, the central claim would need to be narrowed to 'fine-tuned BERTs beat zero-shot and fixed-probe LLMs on these six tasks.' The paper's Table 1 is internally consistent and supports the narrow reading, but the broad generalization in the abstract and conclusion is not yet established. The three-way taxonomy and TaMAS are also derived post hoc from the same six datasets, with no held-out validation, so they do not independently support the generalization.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares four fine-tuned BERT-like models (BERT, RoBERTa, ERNIE, ELECTRA) against LLM-based approaches—zero-shot Qwen/LLaMA, GPT-4o querying, and internal-state probes SAPLMA and MM-Probe—on six text classification datasets: three ToxiCloakCN hate-speech variants, LegalText, MaliciousCode, and Hallucination. The central claim is that BERT-like models often outperform LLMs, and the paper further categorizes the six datasets into three types (pattern-driven, rule-based clean, knowledge-intensive) and proposes TaMAS, a task-aware model selection strategy. Table 1 supports the narrow claim that fine-tuned BERT-like models beat the specific zero-shot and frozen-probe LLM methods on the majority of these six datasets, but the broad generalizations in the abstract and Section 4 go beyond what the experimental setup can establish.","tokens_in":8100,"tokens_out":5843,"duration_ms":61732,"significance":"If the findings hold, the paper provides a useful empirical counterpoint to the prevailing 'LLM-centric' narrative, with a compact benchmark covering six moderately challenging classification tasks and an interpretable analysis based on PCA and layerwise probing. The potential strength is that the raw performance table is directly reproducible from the described fixed splits, standard BERT variants, and publicly available datasets. However, the significance is limited by the asymmetric comparison (fine-tuned BERTs vs. zero-shot/frozen-probe LLMs), the absence of uncertainty estimates, and the post hoc derivation of the taxonomy and selection strategy from the same six datasets. The paper is a reasonable empirical study whose claims need to be narrowed or supplemented before they can support the abstract's broad conclusion.","major_comments":[{"comment":"The comparison is asymmetric: BERT-like models are fine-tuned on the training split, whereas the LLM baselines are either zero-shot prompted or used only through internal-state probes with a frozen backbone. The authors state in §B, 'We do not include methods that involve fine-tuning the parameters of the LLMs,' yet the abstract concludes that 'BERT-like models often outperform LLMs.' This conclusion is not supported for fine-tuned LLMs; a LoRA or QLoRA fine-tuned 7B/8B model could plausibly change the ranking on the four datasets where BERT currently leads. The claim should be narrowed to 'fine-tuned BERT-like models outperform zero-shot and frozen-representation LLM methods on these six datasets,' or the experiments should include a fine-tuned LLM baseline.","section":"§2.1, §B, Abstract"},{"comment":"All reported scores are single runs with no standard deviations, significance tests, or multiple seeds. For LegalText, the AUC gap between RoBERTa (99.2) and SAPLMA (97.7) is small, and for MaliciousCode the F1 scores are tied at 99.7 across several methods. Without uncertainty estimates, the claim that BERT-like models 'consistently outperform' LLM-based approaches is not robust, especially on tasks with near-ceiling performance. At minimum, the authors should report averages and standard deviations over multiple seeds and perform a significance test on the key comparisons.","section":"Table 1"},{"comment":"The three-way dataset taxonomy and the TaMAS strategy are derived from the same six datasets on which they are evaluated. The paper does not validate the taxonomy on held-out datasets, and TaMAS is not operationalized: it does not provide a decision rule to determine, for a new text classification task, whether the pattern-driven, rule-based, or knowledge-intensive category applies without first running both model families. As a result, the proposed strategy is not falsifiable in its current form. The authors should either formalize a concrete, a priori criterion for assigning a task to a category, or present evidence on additional datasets to show the taxonomy and the selection strategy generalize.","section":"§3, §4"}],"minor_comments":[{"comment":"There are several typos: 'proposTaMAS' in §5 should be 'propose TaMAS', 'MaliciousCo' in §D should be 'MaliciousCode', and 'Perturbated' in §3 should be 'Perturbed'.","section":"§5, §D, §3"},{"comment":"The abbreviations 'Cat.', 'Meth./Met.', 'BLMs', 'LLM-IS', and 'LLM-Q' are used in the table but not defined in the caption; they should be spelled out at first use.","section":"Table 1 caption"},{"comment":"The PCA and probing figures are qualitative; the authors should report a quantitative separability metric, such as linear probe accuracy or silhouette score, to support the claims about linear separability.","section":"Figures 2 and 3"},{"comment":"The GPT-4o row reports only F1 scores on the three ToxiCloakCN splits and is absent for the other three datasets. If these numbers are taken from Xiao et al. (2024), the citation should be explicit in the table; if they are new runs, the missing metrics and datasets should be completed.","section":"Table 1, §2.1"},{"comment":"No code or data availability statement is provided; given the small scale of the experiments, releasing code would aid reproducibility and make the comparisons easier to verify.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a compact empirical study with a provocative title. The central claim as stated in the abstract is stronger than the evidence: the absence of fine-tuned LLM baselines and of uncertainty quantification means the broad conclusion is not yet established. I would support publication after the authors either add LoRA baselines and error bars or substantially narrow the claims. The paper may be better suited to an empirical NLP workshop than a general journal, but the topic is timely and the raw comparison is useful."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing you should know: this paper's central claim, as worded in the abstract, is broader than the evidence. What it actually shows is that, on six selected datasets, fine-tuned BERT-like models beat zero-shot and internal-state probes from 7B/8B open-weight LLMs on four datasets, while hallucination detection goes the other way. That narrow result is real and worth having.\n\nWhat's new: the three-way comparison—fine-tuned BERT, LLM internal-state probes (SAPLMA, MM-Probe), and zero-shot querying—on these six datasets, with PCA and probing analysis, is not something I've seen at this resolution. The paper is transparent about the setup: Section 2.1 and Appendix B explicitly say LLM fine-tuning is excluded for cost, and the datasets are listed with links. Table 1 is internally consistent. The PCA visualizations and the probing figure give a plausible picture of why BERT handles pattern-preserving perturbed text well. The generation examples in Table 4 nicely illustrate the hallucination-detection mechanism.\n\nThe soft spots are real but not fatal. Most importantly, the comparison is asymmetric: BERT models are fine-tuned on training labels, LLMs are zero-shot or probed with fixed weights. That's stated, but the abstract says 'BERT-like models often outperform LLMs,' which overclaims. A LoRA-fine-tuned 7B model could shift the rankings, and we don't know because it wasn't tried. GPT-4o results are incomplete (only F1 on three ToxiCloakCN cells). No error bars or multiple seeds, so the differences between some models could be noise—notably on LegalText and MaliciousCode where several methods sit in the high 90s. The three-way taxonomy and TaMAS are derived from the same six datasets they are supposed to predict, with no held-out validation, so treat them as interpretive, not predictive.\n\nThe paper is a useful empirical data point for practitioners choosing between fine-tuned encoders and off-the-shelf LLMs on classification tasks. It deserves a serious referee, but the authors should be pushed to scope the claims to the actual setup, add repeated runs or significance tests, and ideally add one fine-tuned LLM baseline. I'd send it to review with a request for major revision, not desk-reject it.","headline":"A genuinely useful narrow benchmark buried under an over-broad headline; worth reviewing but the authors need to scope the claims.","tokens_in":8653,"tokens_out":1906,"would_cite":true,"duration_ms":19661,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned BERT-like models still outperform zero-shot and internal-state LLM methods on most of six hard text-classification tasks, and the paper provides a rule for choosing between them.","keywords":["BERT-like models","text classification","large language models","zero-shot prompting","internal state probing","model selection","pattern-driven tasks","task-aware strategy"],"falsifier":"Re-run the same six test sets with a 7B/8B LLM fine-tuned (parameter-efficiently) on the same training labels used for the BERT models, keeping metrics fixed; if it beats BERT-like models on the ToxiCloakCN variants or on a majority of tasks, the central claim fails.","tokens_in":7605,"feed_emoji":"⚖️","tokens_out":10278,"duration_ms":66330,"temperature":0.7,"pith_summary":"On six deliberately difficult text-classification benchmarks, fine-tuned BERT-style models beat zero-shot prompting and internal-state probing with 7B/8B open-weight LLMs on the majority of tasks, while using far less compute. The paper argues that the community's reflexive 'use an LLM' approach has prematurely sidelined cheap bidirectional encoders. It groups the six datasets into pattern-driven, rule-based, and knowledge-intensive types, and shows LLMs win only on the knowledge-intensive one (hallucination detection). Based on these groupings it proposes TaMAS, a selection rule that says: choose BERT-like models for pattern-driven or rule-based tasks, and LLMs for tasks needing world knowledge or deep semantics.","feed_headline":"Fine-tuned BERT still beats zero-shot LLMs on most tasks","feed_subtitle":"A six-dataset study gives a rule for when to pick cheap BERT models over LLMs.","key_machinery":"The argument is carried by the experimental protocol: four fine-tuned bidirectional transformers compared against two LLM-based families (zero-shot prompting and internal-state probes, the latter enhanced with Prism) across six datasets. The interpretability analysis uses PCA on hidden states and single linear probes at each layer, showing that BERT-like models produce linearly separable CLS embeddings on pattern-driven tasks while LLM hidden states mix classes under obfuscation. TaMAS, the proposed strategy, converts these observations into a decision rule: use BERT-like models for general or rule-based classification, and LLMs for knowledge-intensive or semantically deep tasks.","core_discovery":"The paper's central claim is that for most of the six datasets it studies, fine-tuned BERT-like models (BERT, RoBERTa, ERNIE, ELECTRA) yield higher or comparable accuracy, AUC, and F1 than LLM-based methods, even though the LLM methods include state-of-the-art internal-state probes. The ToxiCloakCN implicit hate speech variants (base, emoji, homophone) are won by BERT-like models by large margins; legal-text and malicious-code classifications are close between BERT-like models and LLM internal-state probes, with direct zero-shot prompting clearly behind; and hallucination detection is the one task where LLM internal-state methods win. The paper interprets this as evidence that the 'LLM-centric' trend in text classification has overlooked a cheap, strong alternative, and that the correct model choice depends on dataset properties rather than on the latest model family.","pith_inferences":["If the comparison were made symmetric by fine-tuning the 7B/8B LLMs, the BERT advantage on pattern-driven tasks might shrink but could still hold, since the paper's PCA evidence points to a representational difference, not just a fine-tuning gap.","The taxonomy might extend beyond these six datasets: tasks with community-specific jargon, obfuscation, or sparse labels are plausible candidates for BERT-like models.","A cheap, practical takeaway is to run a quick linear-probe separability check on a small sample of hidden states before committing to an LLM pipeline.","The paper's rule could be converted into a decision tree or scoring function for new datasets, though the current evidence only supports six tasks."],"forward_implications":["Fine-tuned BERT-like models remain a cost-effective default for many text classification tasks, especially when labels are available.","Datasets can be triaged by type—pattern-driven, rule-based, knowledge-intensive—before choosing a model family.","LLM internal-state probes are a viable middle ground on rule-based and knowledge-intensive tasks, outperforming zero-shot prompting.","The pattern-driven advantage of BERT-like models persists even under homophone and emoji perturbations in the tested data.","The proposed TaMAS rule offers a practical way to avoid wasteful LLM usage when an encoder would do."],"supporting_citations":[{"why":"Supplies the BERT baseline that the paper fine-tunes and shows winning on pattern-driven tasks.","marker":"(Devlin et al., 2019)"},{"why":"Supplies the RoBERTa baseline, another BERT-like model used in the comparison.","marker":"(Liu et al., 2019)"},{"why":"Supplies the ELECTRA baseline, a discriminator-pretrained BERT-like model.","marker":"(Clark et al., 2020)"},{"why":"Supplies the ERNIE baselines, knowledge-enhanced BERT-like models used for Chinese and English tasks.","marker":"(Sun et al., 2020, 2021)"},{"why":"Supplies the SAPLMA internal-state probe and the hallucination detection dataset; also provides the claim that LLM hidden states encode truthfulness.","marker":"(Azaria and Mitchell, 2023)"},{"why":"Supplies the MM-Probe mass-mean vector method for classifying LLM internal states.","marker":"(Marks and Tegmark, 2023)"},{"why":"Supplies the Prism approach that enhances the internal-state probes (SAPLMA and MM-Probe).","marker":"(Zhang et al., 2024a)"},{"why":"Supplies the ToxiCloakCN datasets (base, emoji, homophone) and the reported best results incorporated for that benchmark.","marker":"(Xiao et al., 2024)"}],"fun_headline_variants":["BERT beats LLMs on most text classification tasks, study finds","Fine-tuned BERT wins over zero-shot LLMs on most datasets","New task-driven strategy: when to pick BERT over LLMs","Six-dataset study: BERT models beat LLMs, except for one case","BERT still outperforms LLMs in text classification—most of the time"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison is asymmetric: BERT-like models are fine-tuned on the training labels, while the open-weight LLMs are used zero-shot or with probes trained on those labels but never fine-tuned, so a stronger or fine-tuned LLM could change the ranking.","fun_headline_variants_meta":{"raw":{"variants":["BERT beats LLMs on most text classification tasks, study finds","Fine-tuned BERT wins over zero-shot LLMs on most datasets","New task-driven strategy: when to pick BERT over LLMs","Six-dataset study: BERT models beat LLMs, except for one case","BERT still outperforms LLMs in text classification—most of the time"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000248,"raw_usage":{"total_tokens":1505,"prompt_tokens":865,"completion_tokens":640,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":545}},"tokens_in":481,"tokens_out":640,"duration_ms":5404,"temperature":1.0,"reasoning_tokens":545,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:45:28.557544+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same six test sets with a 7B/8B LLM fine-tuned (parameter-efficiently) on the same training labels used for the BERT models, keeping metrics fixed; if it beats BERT-like models on the ToxiCloakCN variants or on a majority of tasks, the central claim fails.","supporting_citations":[],"review_version":1}