Pith. sign in

REVIEW 3 major objections 5 minor 44 references

On the Effectiveness of Large Language Models in Automating Categorization of Scientific Texts

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Pith's one-line read A few-shot-prompted Llama 3.1 sorts 59,344 scientific texts into five research areas with 0.82 accuracy, beating the reported BERT baseline of 0.74 without any fine-tuning.

desk verdict A useful empirical case study of LLMs for coarse research-area tagging, but the head-to-head accuracy claim against BERT is not fully supported as written. read the letter →

arxiv 2502.15745 v1 pith:CKT4IFVE submitted 2025-02-08 cs.CL cs.DLcs.LG

classification cs.CLcs.DLcs.LG
keywords LargelanguagemodelsFieldofresearchclassificationScientifictextPromptengineeringFORCdatasetORKGtaxonomyScholarlypublicationsZero-shotandfew-shotlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether off-the-shelf large language models can take over the labor-intensive job of assigning scientific papers to research areas, a task normally done by trained librarians or supervised classifiers. Using 59,344 papers from the FORC dataset and a five-domain taxonomy, the authors find that a few-shot-prompted Llama 3.1 reaches 0.82 accuracy, beating the reported BERT baseline of 0.74 without any fine-tuning. If the comparison holds up, institutions could classify large literature collections with a generic LLM and a well-written prompt, avoiding the need for per-taxonomy labeled training data. The paper also maps how prompt style and temperature move accuracy, identifying 0.8 as a sweet spot.

What carries the argument

The central mechanism is a few-shot prompt that hands the model the ORKG five-domain taxonomy as a hierarchy and one fully worked example, plus the temperature parameter set to 0.8. The prompt turns the LLM's pretrained world knowledge into a zero-training classifier, while temperature tunes the randomness of the generated label. The evaluation harness compares models on the same 59,344 title-abstract records, using accuracy against the human-assigned FORC labels as the yardstick. The baseline machinery is a standard BERT classifier and a BiLSTM, whose reported accuracies of 0.74 and 0.66 are the numbers the 0.82 must beat.

What would settle it

Reproduce the experiment with BERT and BiLSTM fine-tuned on a train/test split of the same 59,344 title-abstract pairs and the same five-class accuracy definition; if a properly tuned BERT reaches or exceeds 0.82, the claimed LLM advantage is an artifact of an unequal baseline.

Watch

Extended reading notes

Core claim

The paper claims that off-the-shelf open-weights LLMs can classify scientific texts into the five top-level research areas of the ORKG taxonomy without any fine-tuning. On the FORC dataset of 59,344 title-abstract pairs, Llama 3.1 (70B) with a few-shot prompt at temperature 0.8 reaches accuracy 0.82, 0.08 above the reported BERT accuracy of 0.74 and 0.16 above BiLSTM's 0.66. Two prompt types (zero-shot and few-shot) and temperatures from 0.2 to 1.0 were tested across four models, with few-shot prompting and the 0.8 temperature yielding the best results and temperature 1.0 decreasing quality. The authors frame the result as evidence that modern LLMs can replace supervised classifiers for coarse-grained scholarly tagging.

Load-bearing premise

The reported advantage of 0.82 over 0.74 assumes the BERT and BiLSTM baselines were trained and measured under the same conditions as the LLMs, but the paper gives no fine-tuning protocol, train/test split, or metric definition for them.

Editorial extensions

If this is right

  • Institutions running institutional repositories can deploy an off-the-shelf LLM as a first-pass subject tagger, reserving human effort for ambiguous cases.
  • Few-shot prompting with a taxonomy excerpt emerges as a cheap alternative to fine-tuning for coarse-grained classification, at least within the five top-level domains tested.
  • The temperature result suggests that too little output randomness undercuts performance and too much randomness at temperature 1.0 also hurts, so parameter selection matters for this task.
  • The approach is claimed to extend to the lower two levels of the ORKG taxonomy and to other schemes such as ACM or Dewey Decimal Classification, though these extensions were not run in the paper.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the result replicates under a controlled baseline comparison, the practical import is that coarse research-area tagging becomes a prompt-engineering task rather than a model-training task, lowering the barrier for smaller institutions.
  • Because the evaluation uses only title and abstract, accuracy likely varies with abstract length and completeness; a stratified check would show whether the 0.82 is carried by well-formed abstracts.
  • The published comparison may overstate the gap: with fine-tuning details for BERT unspecified, a properly tuned BERT on the same five-class split could close or reverse the 0.08 difference.
  • A class-balanced evaluation is missing; with five highly imbalanced top-level domains, macro-F1 could be much lower than accuracy, so the LLM's practical advantage may concentrate in the majority classes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper evaluates four open-source LLMs (Llama 3.1 70B, Gemma 2 27B, Mistral Nemo 12B, and Phi 3.5 3.8B) on the FORC dataset, using zero-shot and few-shot prompts at five temperatures to classify 59,344 scientific titles and abstracts into five ORKG top-level research areas. It reports that Llama 3.1 with a few-shot prompt at temperature 0.8 achieves 0.82 accuracy, exceeding the reported BERT accuracy of 0.74 and BiLSTM accuracy of 0.66, and concludes that off-the-shelf LLMs can match or surpass fine-tuned transformer classifiers on this task. The paper also includes a small manual error analysis and releases code and data.

Significance. If the comparison were properly controlled, the result would be practically useful: it would show that a parameterized prompt and temperature sweep on a large open model can replace fine-tuned encoders for coarse scholarly tagging, with implications for libraries and research information systems. The paper's strengths are its use of a realistic public dataset, its coverage of several open-weight models, and its explicit release of data and code. However, the central quantitative claim currently rests on an unreported baseline evaluation protocol and a post-hoc temperature selection on the same data used for the headline accuracy. These are fixable methodological gaps rather than fatal flaws, but they are load-bearing for the paper's main comparison.

major comments (3)
  1. [§5.3 and Table 4] The baseline evaluation protocol is not specified: there is no train/test split, no fine-tuning procedure, no hyperparameter settings, no epoch count, and no definition of the accuracy metric. The statement that BERT and BiLSTM were 'implemented using the models available on this platform' is insufficient to establish that the 0.74 and 0.66 numbers in Table 4 were obtained under conditions comparable to the LLM evaluation. If the baselines were evaluated on the same 59,344 texts used to tune the LLM temperature, or if they were not fine-tuned on FORC training data, the headline 0.82-versus-0.74 gap is an artifact. Please report the exact split, training details, and metric, and ideally provide per-class accuracy and standard errors.
  2. [§6 and Tables 5–6] Temperature selection is performed on the same data as the headline evaluation. Section 6 states that all models were evaluated 'with all nearly 60.000 titles and abstracts' and that the temperature that 'seems to work best' was then reported. This makes the 0.82 figure the maximum over five evaluated temperatures rather than an independent test result, which inflates the expected out-of-sample accuracy. Use a held-out validation split for model selection and report the test-set accuracy for the chosen configuration, with confidence intervals or significance tests against the BERT baseline.
  3. [Table 1] The label-matching rule is undefined. The zero-shot example shows Expected Output 'Social and Behavioral Sciences' while the model Output is 'Social Science'; if exact string matching is used, semantically correct labels are counted as errors, while if fuzzy or synonym matching is used, the rule must be disclosed. Please specify the normalization and matching procedure and report how many LLM outputs required normalization, since this directly affects every accuracy value in Tables 5 and 6.
minor comments (5)
  1. [Table 2] Table 2 states that Gemma has '27 million parameters' while the surrounding text gives 27.2B parameters; the correct value is 27.2 billion.
  2. [Concluding remarks] The concluding section refers to 'two baseline BERT models', but the baselines are BERT and BiLSTM, not two BERT variants.
  3. [Table 1] The few-shot prompt references a 'taxonomy of research field extracted from ORKG' but does not include the taxonomy hierarchy, making the few-shot condition non-reproducible from the paper alone; please either include the hierarchy or refer to a stable appendix.
  4. [Throughout] There are several minor inconsistencies and typos: 'LLaMa' versus 'Llama', 'temperature values from 0.2 up to 1.0' versus 'temperature 0.8', and a missing space in 'factors influencing' in Table 1.
  5. [References] The in-text citation 'Enamoto et al.' does not match the bibliography entry 'Enamoto'; please harmonize the spelling.

Circularity Check

1 steps flagged · score 4.0 of 10

Temperature is selected on the full test corpus, so the reported 0.82 is a post-hoc best-of-five rather than an independent prediction; no load-bearing self-citation or derivation-level circularity was found.

  1. fitted input called prediction [Section 6 (Results) and Section 5.1.1 (Dataset, Data Cleaning and Preprocessing)]
    "We evaluated the baseline models and the selected LLMs according to the two explained prompts and temperature values, from 0.2 up to 1.0 with all nearly 60.000 titles and abstracts from the FORC dataset."

    The headline accuracy of 0.82 is reported for temperature 0.8, which the paper identifies as the best among the five tested values on the same full corpus: 'after reaching the best performance at 0.8, quality starts decreasing'. Section 5.1.1 states that title and abstract from all 59,344 texts were used and accuracy was derived on this full set; no held-out split is described. Selecting the temperature on the evaluation set and then reporting its accuracy as the central result is a post-hoc maximum, so the 0.82-versus-0.74 comparison is not an independent prediction of a fixed configuration but a best-of-five selection on the test data. This is the fitted-input-called-prediction pattern, although it affects the headline number rather than the entire empirical comparison.

full rationale

This paper is an empirical evaluation rather than a mathematical derivation, so most circularity patterns do not apply. The only circularity-adjacent step is the temperature selection: the paper evaluates all temperature values on the full FORC corpus and then reports the best observed accuracy as the main result, with no held-out validation or fixed-protocol pre-registration. This makes the exact 0.82 figure statistically inflated and not an unbiased prediction, but it does not make the broader findings (few-shot outperforms zero-shot, larger models perform better, all LLMs beat the reported baselines) equivalent to the inputs by construction. The BERT and BiLSTM baseline accuracies are reported without a fine-tuning protocol or train/test split, which is a serious evaluation-comparability risk but not circularity. The self-citations to the authors' prior work are not load-bearing for the central claim. Overall, the paper has partial circularity in the headline number due to test-set temperature selection, but the core empirical content is not forced by definition or by a self-citation chain.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the reliability of the FORC labels, the single-label assumption of the ORKG top-level taxonomy, an unverified baseline comparability, and an unexplained mapping from free-form LLM outputs to the five labels. The only hand-selected numerical setting is the temperature, chosen post hoc on the evaluation data.

free parameters (1)
  • temperature = 0.8 (best of 0.2, 0.4, 0.6, 0.8, 1.0)
    The headline accuracy 0.82 is obtained at temperature 0.8; this value was selected by running the evaluation on the same test data rather than a held-out validation set, so the reported best-case accuracy depends on this hand-selected hyperparameter.
assumptions (4)
  • domain assumption FORC labels are correct enough to serve as ground truth for accuracy measurement.
    Used without any label-quality audit in Section 5.1; all accuracy numbers are computed against these human-assigned labels.
  • domain assumption The five ORKG top-level domains are mutually exclusive and each text has one correct top-level research area.
    The task is treated as single-label 5-class classification in Sections 5.1 and 6; interdisciplinary or boundary papers would break this assumption.
  • ad hoc to paper Baseline models were trained and evaluated in a way comparable to the LLM evaluation.
    Section 5.3 gives no fine-tuning, train/test split, or evaluation protocol; the fairness of the 0.82 versus 0.74 comparison rests on this unstated assumption.
  • domain assumption LLM outputs can be unambiguously matched to one of the five labels.
    The paper does not describe output parsing or normalization; the prompt examples in Table 1 show answers that are not exactly the taxonomy labels, so a matching rule is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Effectiveness of Large Language Models in Automating Categorization of Scientific Texts." pith.science (2026). https://pith.science/paper/CKT4IFVE

@misc{pith2026250215745,
  author       = {Pith},
  title        = {Pith review of: On the Effectiveness of Large Language Models in Automating Categorization of Scientific Texts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CKT4IFVE}},
  note         = {Machine review of arXiv:2502.15745}
}
read the original abstract

The rapid advancement of Large Language Models (LLMs) has led to a multitude of application opportunities. One traditional task for Information Retrieval systems is the summarization and classification of texts, both of which are important for supporting humans in navigating large literature bodies as they e.g. exist with scientific publications. Due to this rapidly growing body of scientific knowledge, recent research has been aiming at building research information systems that not only offer traditional keyword search capabilities, but also novel features such as the automatic detection of research areas that are present at knowledge intensive organizations in academia and industry. To facilitate this idea, we present the results obtained from evaluating a variety of LLMs in their ability to sort scientific publications into hierarchical classifications systems. Using the FORC dataset as ground truth data, we have found that recent LLMs (such as Meta Llama 3.1) are able to reach an accuracy of up to 0.82, which is up to 0.08 better than traditional BERT models.

Figures

Figures reproduced from arXiv: 2502.15745 by the authors.

Figure 1
Figure 1. Methodology used in the identification of research area [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 33 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

  2. [2]

    Abburi, H., Suesserman, M., Pudota, N., Veeramani, B., Bowen, E., and Bhattacharya, S. (2023). Generative ai text classification using ensemble llm approaches. arXiv preprint arXiv:2309.07755

  3. [3]

    Abu Ahmad, R., Borisova, E., and Rehm, G. (2024). Forc@ nslp2024: Overview and insights from the field of research classification shared task. In International Workshop on Natural Scientific Language Processing and Research Knowledge Graphs , pages 189--204. Springer

  4. [4]

    R., and Al Mamun, F

    Al Nazi, Z., Hossain, M. R., and Al Mamun, F. (2025). Evaluation of open and closed-source llms for low-resource language with zero-shot, few-shot, and chain-of-thought prompting. Natural Language Processing Journal , page 100124

  5. [5]

    and Mann, S

    Auer, S. and Mann, S. (2019). Towards an open research knowledge graph. The Serials Librarian , 76(1-4):35--41

  6. [6]

    J., Gibson, B

    Bird, S., Dale, R., Dorr, B. J., Gibson, B. R., Joseph, M. T., Kan, M.-Y., Lee, D., Powley, B., Radev, D. R., Tan, Y. F., et al. (2008). The acl anthology reference corpus: A reference dataset for bibliographic research in computational linguistics. In LREC

  7. [7]

    Bornmann, L., Haunschild, R., and Mutz, R. (2021). Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases. Humanities and Social Sciences Communications , 8(1):1--15

  8. [8]

    Desale, S. K. and Kumbhar, R. M. (2014). Research on automatic classification of documents in library environment: a literature review. KO KNOWLEDGE ORGANIZATION , 40(5):295--304

Show all 44 references
  1. [9]

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

  2. [10]

    R., Maia, R., Weigang, L., and Filho, G

    Enamoto, L., Santos, A. R., Maia, R., Weigang, L., and Filho, G. P. R. (2022). Multi-label legal text classification with bilstm and attention. International Journal of Computer Applications in Technology , 68(4):369--378

  3. [11]

    Feuerriegel, S., Hartmann, J., Janiesch, C., and Zschech, P. (2024). Generative ai. Business & Information Systems Engineering , 66(1):111--126

  4. [12]

    and Paynter, G

    Frank, E. and Paynter, G. W. (2004). Predicting library of congress classifications from library of congress subject headings. Journal of the American Society for Information Science and Technology , 55(3):214--227

  5. [13]

    Gao, A. (2023). Prompt engineering for large language models. Available at SSRN 4504303

  6. [14]

    B., D'Souza, J., and Auer, S

    Giglou, H. B., D'Souza, J., and Auer, S. (2024). Llms4synthesis: Leveraging large language models for scientific synthesis. arXiv preprint arXiv:2409.18812

  7. [15]

    a ck, J., and Ard \

    Golub, K., Hagelb \"a ck, J., and Ard \"o , A. (2020). Automatic classification of swedish metadata using dewey decimal classification: a comparison of approaches. Journal of Data and Information Science , 5(1):18--38

  8. [16]

    Hacker, P., Engel, A., and Mauer, M. (2023). Regulating chatgpt and other large generative ai models. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency , pages 1112--1123

  9. [17]

    and Knoth, P

    Herrmannova, D. and Knoth, P. (2016). An analysis of the microsoft academic graph. D-lib Magazine , 22(9/10):37

  10. [18]

    Huang, Z., Xu, W., and Yu, K. (2015). Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991

  11. [19]

    Jiang, M., D’Souza, J., Auer, S., and Downie, J. S. (2020). Improving scholarly knowledge representation: Evaluating bert-based models for scientific relation classification. In Digital Libraries at Times of Massive Societal Transition: 22nd International Conference on Asia-Pa...

  12. [20]

    Kalyan, K. S. (2023). A survey of gpt-3 family large language models including chatgpt and gpt-4. Natural Language Processing Journal , page 100048

  13. [21]

    Kinney, R., Anastasiades, C., Authur, R., Beltagy, I., Bragg, J., Buraczynski, A., Cachola, I., Candra, S., Chandrasekhar, Y., Cohan, A., et al. (2023). The semantic scholar open data platform. arXiv preprint arXiv:2301.10140

  14. [22]

    Liu, S., Yu, S., Lin, Z., Pathak, D., and Ramanan, D. (2024). Language models as black-box optimizers for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12687--12697

  15. [23]

    Mahapatra, R., Gayan, M., Jamatia, B., et al. (2024). Artificial intelligence tools to enhance scholarly communication: An exploration based on a systematic review

  16. [24]

    and Chiang, M

    Morgan, J. and Chiang, M. (2024). Ollama . https://ollama.com. Online; accessed 6 August 2024

  17. [25]

    Mosca, E., Abdalla, M. H. I., Basso, P., Musumeci, M., and Groh, G. (2023). Distinguishing fact from fiction: A benchmark dataset for identifying machine-generated scientific papers in the llm era. In Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (...

  18. [26]

    Murphy, K. P. (2022). Probabilistic machine learning: an introduction . MIT press

  19. [27]

    Nah, F., Cai, J., Zheng, R., and Pang, N. (2023). An activity system-based perspective of generative ai: Challenges and research directions. AIS Transactions on Human-Computer Interaction , 15(3):247--267

  20. [28]

    A., and Chakraborty, C

    Pal, S., Bhattacharya, M., Islam, M. A., and Chakraborty, C. (2024). Ai-enabled chatgpt or llm: a new algorithm is required for plagiarism-free scientific writing. International Journal of Surgery , 110(2):1329--1330

  21. [29]

    and Po \'s wiata, R

    Pere kiewicz, M. and Po \'s wiata, R. (2024). A review of the challenges with massive web-mined corpora used in large language models pre-training. arXiv preprint arXiv:2407.07630

  22. [30]

    Pertsas, V., Kasapaki, M., and Constantopoulos, P. (2024). An annotated dataset for transformer-based scholarly information extraction and linguistic linked data generation. In Proceedings of the 9th Workshop on Linked Data in Linguistics@ LREC-COLING 2024 , pages 84--93

  23. [31]

    Rabby, G., Auer, S., D'Souza, J., and Oelen, A. (2024). Fine-tuning and prompt engineering with cognitive knowledge graphs for scholarly knowledge organization. arXiv preprint arXiv:2409.06433

  24. [32]

    Rous, B. (2012). Major update to acm's computing classification system. Communications of the ACM , 55(11):12--12

  25. [33]

    Scott, M. L. (1998). Dewey decimal classification. Libraries Unlimited

  26. [34]

    Shahi, G. K. and Hummel, O. (2024). Enhancing research information systems with identification of domain experts. In Proceedings of the Bibliometric-enhanced Information Retrieval Workshop (BIR) at the European Conference on Information Retrieval (ECIR 2024) , CEUR Workshop Pr...

  27. [35]

    Shahi, G. K. and Nandini, D. (2020). Fake C ovid -- a multilingual cross-domain fact check news dataset for covid-19. In Proceedings of the 14th International AAAI C onference on W eb and S ocial M edia

  28. [36]

    S., Love, J., et al

    Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivi \`e re, M., Kale, M. S., Love, J., et al. (2024). Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295

  29. [37]

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

  30. [38]

    Wang, J. (2009). An extensive study on automated dewey decimal classification. Journal of the American Society for Information Science and Technology , 60(11):2269--2286

  31. [39]

    Wang, K., Shen, Z., Huang, C., Wu, C.-H., Dong, Y., and Kanakia, A. (2020). Microsoft academic graph: When experts are not enough. Quantitative Science Studies , 1(1):396--413

  32. [40]

    Wang, S., Hu, T., Xiao, H., Li, Y., Zhang, C., Ning, H., Zhu, R., Li, Z., and Ye, X. (2024). Gpt, large language models (llms) and generative artificial intelligence (gai) models in geospatial science: a systematic review. International Journal of Digital Earth , 17(1):2353122

  33. [41]

    Wiggins, B. J. (2009). Acquisitions and bibliographic access directorate library of congress report of fiscal year 2009 (fiscal year ended september 30, 2009)

  34. [42]

    Young, J. S. and Lammert, M. (2024). Chatgpt for classification: Evaluation of an automated course mapping method in academic libraries

  35. [43]

    Zhang, C., Tian, L., and Chu, H. (2023). Usage frequency and application variety of research methods in library and information science: Continuous investigation from 1991 to 2021. Information Processing & Management , 60(6):103507

  36. [44]

    X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al

    Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023). A survey of large language models. arXiv preprint arXiv:2303.18223

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.