REVIEW 4 major objections 4 minor 30 references
An Ensemble Classification Approach in A Multi-Layered Large Language Model Framework for Disease Prediction
T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read A majority-voting ensemble of three Arabic BERT models, fine-tuned on LLAMA3-preprocessed patient posts, reaches 80.56% accuracy in seven-class disease prediction.
desk verdict A systematic but under-validated engineering study; the ensemble approach is plausible and the preprocessing analysis is useful, but the evaluation lacks rigor and has an internal inconsistency. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the majority-voting ensemble over a grid of twelve base models: three Arabic BERT architectures (CAMeLBERT, AraBERT, AsafayaBERT) fine-tuned separately on four text representations (original 'post', original+refined, original+NER, original+summarized). LLAMA3 supplies the three preprocessing transformations, but the ensemble itself is a simple vote that aggregates each model's top label for a post. The diversity created by pairing original text with distinct processed views, and then crossing those views with different transformer backbones, is what the paper credits for the accuracy lift.
What would settle it
Re-run the pipeline on the same private dataset with a stratified, leakage-checked train/test split and repeated seeds; if the majority vote does not consistently exceed the best single model beyond chance, or if the gap to 80.56% collapses, the central claim is falsified. A simpler check: see whether a random 50/50 split reproduces the reported accuracies within a few points.
Extended reading notes
Core claim
The central claim is that combining LLM-based preprocessing with fine-tuned Arabic transformer models and a majority-voting ensemble improves disease classification on Arabic social telehealth data. The paper reports the best accuracy of 80.56% with this strategy, observed consistently across ensemble sizes of five, seven, eight, and ten models. The gain over the best single model (75.55% for the refined CAMeLBERT variant) is attributed to diversity among base models and text representations. The study also finds that grouping ensembles by preprocessing type favors refined texts (77.12%) over NER (75.24%) and summarized texts (63.64%), suggesting that over-summarization harms medical classif
Load-bearing premise
The 80.56% figure rests on the privately collected dataset's seven-class annotations being correct and on the train/test split being leakage-free, but the paper reports no split ratio, annotation agreement, or leakage checks.
Editorial extensions
If this is right
- The 80.56% result implies a concrete recipe: refine and pair, don't replace; keeping the original text alongside processed versions outperforms using processed text alone.
- Three diverse models appear sufficient to realize most ensemble gains; expanding to 12 models plateaus at the same accuracy, so practitioners can stop at a handful of diverse members.
- Preprocessing choice dominates model choice in grouped ensembles: refined-text ensembles (77.12%) beat NER and post ensembles, while summarized-text ensembles (63.64%) hurt accuracy, so summary-based compression should be avoided for Arabic medical posts.
- If replicated, the framework establishes a benchmark for Arabic social telehealth disease classification that future systems can compare against.
Reading between the lines
- The reported margin between ensemble (80.56%) and best single model (75.55%) may not be statistically significant given no error bars or repeated-seed evaluation; a cross-validation study would tell whether the ensemble gain is real.
- The same multi-layer pairing scheme should transfer to other low-resource languages that have a BERT-style model and access to a strong LLM, provided the base models are sufficiently diverse.
- Because the dataset is private, external validation on an independent Arabic telehealth corpus is the natural next test; a drop in the ensemble's advantage there would indicate overfitting to this annotation scheme.
- A direct extension: use the ensemble's disagreement signals as a confidence measure to flag posts needing human review, which the current binary vote does not do.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes a multi-layered framework for Arabic disease classification from social telehealth posts. LLAMA3 is used for three preprocessing variants (refinement, summarization, NER), which are paired with original text to fine-tune three Arabic BERT models (CAMeLBERT, AraBERT, AsafayaBERT), yielding twelve base models. Majority-voting ensembles over these models are evaluated on a private seven-class dataset. The paper reports a best accuracy of 80.56% and claims that ensemble strategies improve disease classification performance. It presents this as the first integration of LLM-based preprocessing, fine-tuned Arabic transformers, and ensemble learning for this task.
Significance. If the result is robust, the framework is a useful recipe for Arabic social telehealth classification, and the comparison of preprocessing variants is of practical interest. The paper does not contain a derivation, so there is no circular-fit concern; the claim is empirical. However, the evaluation is entirely internal: one private dataset, no public benchmark, no released code or data, and no fixed evaluation protocol. The paper's strengths are the systematic exploration of ensemble sizes and the direct comparison of preprocessing strategies; these are undermined by the absence of error bars, statistical tests, and a clear model-selection procedure. An honest re-analysis with a pre-registered validation/test split and significance tests would make the contribution much stronger.
major comments (4)
- [Section V, Table II, Figures 3-14] The claimed 80.56% is selected as the maximum over an unstated combinatorial search of ensemble subsets. There is no separation between a model-selection validation set and a held-out test set, no error bars, and no significance tests. Moreover, the text in Section V says this accuracy was 'consistently observed across ensemble sizes of 5, 7, 8, and 10 models' while the caption of Fig. 14 says '4, 6, 7, and 10-model ensembles.' This internal inconsistency indicates the result-selection rule is not well-defined. Without a fixed protocol, the headline number may reflect selection bias rather than a genuine ensemble advantage. Please report a single pre-specified ensemble configuration (or all configurations with test-set means and standard deviations), with significance tests against the best single model (CamelBERT Refined, 75.55%).
- [Section III] The dataset description is incomplete. It gives only the class distribution and states that annotation was supervised by a medical advisor. No total size, no train/validation/test split ratio, no leakage checks, and no inter-annotator agreement are reported. Since this is a private dataset, the reproducibility of the 80.56% figure depends entirely on these details. Add dataset statistics, split procedure, and if possible release the data or provide a detailed data sheet.
- [Sections IV-E and V] There is a mismatch in the number of base models. Section IV-E says fine-tuning three Arabic BERT models and collecting predictions 'from all three models,' but Section V constructs ensembles from twelve base models (3 models × 4 representations). The relationship between the three-model procedure and the twelve-model ensemble is not explained. Also, the paper alternates between 'multi-class, multi-label' (Abstract, Section I) and seven disease types with accuracy as the only metric; if the annotation is single-label, the multi-label terminology should be removed; if it is multi-label, accuracy is not the appropriate evaluation metric and metrics like macro-F1 should be reported.
- [Section V, Figures 3-15] The conclusion that 'ensemble strategies improve disease classification performance' is not supported by the evidence as reported. Figures 3-14 show many ensemble configurations with accuracies around 0.79-0.80, but some groups, e.g., summarized-model ensembles in Fig. 15, perform much worse (0.6364). Without a statistical comparison of the best ensemble to the best individual model, and without reporting variability across folds or seeds, the headline claim remains an observation about one selected configuration, not a general finding.
minor comments (4)
- [Conclusion] Typo: 'accuracya' should be 'accuracy'.
- [Table I] The Arabic text in Table I appears as garbled mojibake in the PDF; ensure the glyphs render correctly.
- [Abstract] The claim 'to the best of our knowledge, this is the first work' is difficult to verify and should be softened or supported by a more thorough comparison with existing Arabic medical text classification benchmarks.
- [Section II] The related work surveys general transformer/LLM results but does not position the proposed method against existing Arabic medical text classification systems; please add direct comparative baselines from the literature.
Circularity Check
No circularity: the reported accuracy is an empirical measurement, not a derived or fitted quantity; selection-bias concerns are a reproducibility issue, not circularity.
full rationale
This paper makes no derivation claim whose conclusion could be equivalent to its inputs. The pipeline is empirical: raw posts are passed through LLAMA3 refinement, summarization, and NER; the outputs are concatenated with the original text; three Arabic BERT models are fine-tuned; and majority voting is applied to the resulting twelve checkpoints. The headline 80.56% accuracy is a reported test-set measurement from Table II, not the output of a formula fitted to the data. Majority voting is not mathematically forced to exceed the best individual model, so the comparison in Table II has independent empirical content. The only self-citation, reference [22], is used to state that the authors previously fine-tuned the same models on the same dataset without ensembling; it motivates novelty but does not supply any load-bearing theorem, parameter, or result. There are genuine methodological concerns — the test set appears to have been used to select the best ensemble among many combinations, and Section V says 0.8056 was observed for ensemble sizes 5, 7, 8, and 10 while the Figure 14 caption says 4, 6, 7, and 10 — but these are selection-bias and reproducibility problems, not circularity. No equation, definition, or citation chain makes the reported accuracy equivalent to the paper's inputs. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (2)
- Training hyperparameters =
lr=1e-4, batch_size=4, epochs=25, dropout=0.05, weight_decay=0.01
- Ensemble composition and size =
best ensemble reaches 0.8056, selected from configs of size 2 to 12
assumptions (5)
- domain assumption LLAMA3 preprocessing (refinement, summarization, NER) preserves the discriminative medical information needed for seven-class disease prediction.
- domain assumption The private dataset annotations are accurate and the train/test split is leakage-free.
- domain assumption The three Arabic transformer checkpoints provide suitable representations for Arabic medical text.
- domain assumption Majority voting is an effective aggregation for the 12 classifiers.
- ad hoc to paper One shared hyperparameter configuration is adequate for all models.
Cite this review
Pith. "Pith review of An Ensemble Classification Approach in A Multi-Layered Large Language Model Framework for Disease Prediction." pith.science (2026). https://pith.science/paper/5YV7AZQQ
@misc{pith2026250902446,
author = {Pith},
title = {Pith review of: An Ensemble Classification Approach in A Multi-Layered Large Language Model Framework for Disease Prediction},
year = {2026},
howpublished = {\url{https://pith.science/paper/5YV7AZQQ}},
note = {Machine review of arXiv:2509.02446}
}
read the original abstract
Social telehealth has made remarkable progress in healthcare by allowing patients to post symptoms and participate in medical consultations remotely. Users frequently post symptoms on social media and online health platforms, creating a huge repository of medical data that can be leveraged for disease classification. Large language models (LLMs) such as LLAMA3 and GPT-3.5, along with transformer-based models like BERT, have demonstrated strong capabilities in processing complex medical text. In this study, we evaluate three Arabic medical text preprocessing methods such as summarization, refinement, and Named Entity Recognition (NER) before applying fine-tuned Arabic transformer models (CAMeLBERT, AraBERT, and AsafayaBERT). To enhance robustness, we adopt a majority voting ensemble that combines predictions from original and preprocessed text representations. This approach achieved the best classification accuracy of 80.56%, thus showing its effectiveness in leveraging various text representations and model predictions to improve the understanding of medical texts. To the best of our knowledge, this is the first work that integrates LLM-based preprocessing with fine-tuned Arabic transformer models and ensemble learning for disease classification in Arabic social telehealth data.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Social media use for health purposes: Systematic review,
J. Chen and Y . Wang, “Social media use for health purposes: Systematic review,” Journal of Medical Internet Research , vol. 23, no. 5, 2020
work page 2020
-
[2]
Mining social media data for biomedical signals and health-related behavior,
R. B. Correia, I. B. Wood, J. Bollen, and L. M. Rocha, “Mining social media data for biomedical signals and health-related behavior,” Annual Review of Biomedical Data Science , vol. 3, no. 1, pp. 433–458, 2020
work page 2020
-
[3]
A. Magge et al., “Overview of the Sixth Social Media Mining for Health Applications (SMM4H) Shared Tasks at NAACL 2021,” in Proc. of the Sixth Social Media Mining for Health Applications (SMM4H) Shared Task at NAACL, Jan. 2021
work page 2021
-
[4]
Y . Guo, A. Ovadje, M. A. Al-Garadi, and A. Sarker, “Evaluating large language models for health-related text classification tasks with public social media data,” Journal of the American Medical Informatics Association, 2024
work page 2024
-
[5]
Text classification algorithms: A survey,
K. Kowsari, K. J. Meimandi, M. Heidarysafa, S. Mendu, L. Barnes, and D. Brown, “Text classification algorithms: A survey,” Information, vol. 10, no. 4, p. 150, Apr. 2019
work page 2019
-
[6]
Repeata- bility of fine-tuning large language models illustrated using QLORA,
S. S. Alahmari, L. O. Hall, P. R. Mouton, and D. B. Goldgof, “Repeata- bility of fine-tuning large language models illustrated using QLORA,” IEEE Access, 2024
work page 2024
-
[7]
Multi-label classification with generative large language models,
N. Niraula, S. Ayhan, B. Chidambaram, and D. Whyatt, “Multi-label classification with generative large language models,” in Proc. of the 43rd Digital Avionics Systems Conference (DASC) , 2024
work page 2024
-
[8]
A fine- tuned BERT-based transfer learning approach for text classification,
R. Qasim, W. H. Bangyal, M. A. Alqarni, and A. A. Almazroi, “A fine- tuned BERT-based transfer learning approach for text classification,” Journal of Healthcare Engineering , 2022
work page 2022
Show all 30 references
-
[9]
Nikolov-Radivchev at SemEval-2019 Task 6: Offensive tweet classification with BERT and ensembles,
V . Radivchev and A. Nikolov, “Nikolov-Radivchev at SemEval-2019 Task 6: Offensive tweet classification with BERT and ensembles,” in Proc. of the 13th Int. Workshop on Semantic Evaluation (SemEval-2019), pp. 691–695, 2019
2019
-
[10]
FBK @ SMM4H 2020: RoBERTa for detecting medications on Twitter,
S. Casola and A. Lavelli, “FBK @ SMM4H 2020: RoBERTa for detecting medications on Twitter,” in Proc. of the Fifth Social Media Mining for Health Applications Workshop & Shared Task , pp. 101–103, 2020
2020
-
[11]
Analyzing ELMo and DistilBERT on socio-political news classification,
B. B ¨uy¨uk¨oz, A. H ¨urriyeto˘glu, and A. ¨Ozg¨ur, “Analyzing ELMo and DistilBERT on socio-political news classification,” in Proc. of the Workshop on Automatic Extraction of Socio-political Events from News , pp. 9–18, USA, May 2020
2020
-
[12]
TorontoCL at CMCL 2021 shared task: RoBERTa with multi-stage fine-tuning for eye-tracking prediction,
B. Li and F. Rudzicz, “TorontoCL at CMCL 2021 shared task: RoBERTa with multi-stage fine-tuning for eye-tracking prediction,” in Proc. of the Workshop on Cognitive Modeling and Computational Linguistics (CMCL), pp. 4–9, 2021
2021
-
[13]
ZYJ123@DravidianLangTech-EACL2021: Offen- sive language identification based on XLM-RoBERTa with DPCNN,
Y . Zhao and X. Tao, “ZYJ123@DravidianLangTech-EACL2021: Offen- sive language identification based on XLM-RoBERTa with DPCNN,” in Proc. of the First Workshop on Speech and Language Technologies for Dravidian Languages, pp. 216–221, Parkville, Victoria: EACL, 2021
2021
-
[14]
YNU @ Dravidian-CodeMix-FIRE2020: XLM- RoBERTa for multi-language sentiment analysis,
X. Ou and H. Li, “YNU @ Dravidian-CodeMix-FIRE2020: XLM- RoBERTa for multi-language sentiment analysis,” in Proc. of the Forum for Information Retrieval Evaluation (FIRE) , pp. 4–9, 2020
2020
-
[15]
ASEM: Enhancing empathy in chatbot through attention-based sentiment and emotion modeling,
O. Hamad, K. Shaban, and A. Hamdi, “ASEM: Enhancing empathy in chatbot through attention-based sentiment and emotion modeling,” in Proc. of the Int. Conf. on Computational Linguistics (COLING) , pp. 1588–1601, 2024
2024
-
[16]
Ensemble of ELECTRA for profiling fake news spreaders,
K. A. Das, A. Baruah, F. A. Barbhuiya, and K. Dey, “Ensemble of ELECTRA for profiling fake news spreaders,” in Proc. of the Forum for Information Retrieval Evaluation (FIRE) , pp. 22–25, 2020
2020
-
[17]
BART-TL: Weakly-supervised topic label generation,
C. Popa and T. Rebedea, “BART-TL: Weakly-supervised topic label generation,” in Proc. of the 16th Conf. of the European Chapter of the ACL (EACL), pp. 1418–1425, 2021
2021
-
[18]
Using BERT and BART for query suggestion,
A. Mustar, S. Lamprier, and B. Piwowarski, “Using BERT and BART for query suggestion,” in Proc. of the CEUR Workshop on NLP , vol. 2621, 2020
2020
-
[19]
BART for knowledge grounded conversations,
M. de Bruyn, E. Lotfi, J. Buhmann, and W. Daelemans, “BART for knowledge grounded conversations,” in Proc. of the CEUR Workshop on NLP, vol. 2666, 2020
2020
-
[20]
LMRPA: Large language model-driven efficient robotic process automation for OCR,
O. H. Abdellatif, A. N. Hassan, and A. Hamdi, “LMRPA: Large language model-driven efficient robotic process automation for OCR,” in The Int. Conf. of Advanced Computing and Informatics , pp. 35–44, Springer, 2024
2024
-
[21]
LLM-SEM: A sentiment- based student engagement metric using LLMs for e-learning platforms,
A. Hamdi, A. A. Mazrou, and M. Shaltout, “LLM-SEM: A sentiment- based student engagement metric using LLMs for e-learning platforms,” in The Int. Conf. of Advanced Computing and Informatics , pp. 145–154, Springer, 2024
2024
-
[22]
A multi-layered large language model framework for disease prediction,
M. Mohamed, R. Emad, and A. Hamdi, “A multi-layered large language model framework for disease prediction,” arXiv preprint arXiv:2502.00063, 2025
2025 arXiv
-
[23]
BERT: Pre-training of deep bidirectional transformers for language understanding,
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. of the 2019 Conf. of the NAACL , pp. 4171–4186, 2019
2019
-
[24]
Contextual se- mantic embeddings based on transformer models for Arabic biomedical questions classification,
I. Ait Talghalit, H. Alami, and S. Ouatik El Alaoui, “Contextual se- mantic embeddings based on transformer models for Arabic biomedical questions classification,” HighTech and Innovation Journal , vol. 5, no. 4, pp. 1024, 2024
2024
-
[25]
StEduCov: An explored and benchmarked dataset on stance detection in tweets towards online education during COVID-19 pandemic,
O. Hamad, A. Hamdi, S. Hamdi, and K. Shaban, “StEduCov: An explored and benchmarked dataset on stance detection in tweets towards online education during COVID-19 pandemic,” Big Data and Cognitive Computing, vol. 6, no. 3, pp. 88, 2022
2022
-
[26]
CLASEG: Advanced multi- classification and segmentation for differential diagnosis of oral lesions using deep learning,
A. Al-Ali, A. Hamdi, M. Elshrif, et al. , “CLASEG: Advanced multi- classification and segmentation for differential diagnosis of oral lesions using deep learning,” Scientific Reports, vol. 15, no. 1, pp. 23016, 2025
2025
-
[27]
Arabic large language models for medical text generation,
A. Allam, S. Ahmed, A. Hamdi, and A. Mohammed, “Arabic large language models for medical text generation,” in 2025 4th Int. Conf. on Computer Technologies (ICCTech), pp. 1–6, 2025
2025
-
[28]
GloVe: Global vectors for word representation,
J. Pennington, R. Socher, and C. D. Manning, “GloVe: Global vectors for word representation,” in Proc. of the 2014 Conf. on Empirical Methods in NLP (EMNLP) , pp. 1532–1543, 2014
2014
-
[29]
AraBERT: Transformer-based model for Arabic language understanding,
W. Antoun, F. Baly, and H. Hajj, “AraBERT: Transformer-based model for Arabic language understanding,” in Proc. of the LREC 2020 Work- shop on Resources for African Indigenous Languages , pp. 9–15, 2020
2020
-
[30]
EDA: Easy data augmentation techniques for boosting performance on text classification tasks,
J. Wei and K. Zou, “EDA: Easy data augmentation techniques for boosting performance on text classification tasks,” Proc. of the 2019 Conf. on Empirical Methods in NLP (EMNLP) , pp. 6382–6388, 2019
2019
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.