REVIEW 2 major objections 7 minor 36 references
How Effectively Can BERT Models Interpret Context and Detect Bengali Communal Violent Text?
T0 review · 2 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The paper claims that an ensemble of fine-tuned BanglaBERT variants detects Bengali communal violent text at macro F1 0.63, up from a 0.36 baseline.
desk verdict A useful, honest low-resource hate-speech paper whose headline F1 gains are probably inflated by model-based test-set selection; worth reviewing, but the authors need to re-evaluate on a random split. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is a fine-tuned BanglaBERT classifier, a transformer language model pre-trained specifically for Bengali, combined at the end into an ensemble of five fine-tuned models whose predictions are merged by mean averaging, majority voting, or an MLP. The data-side mechanism is manual augmentation: after SMOTE, zero-shot, few-shot, and paraphrasing all failed to produce usable minority-class examples, the paper used a fine-tuned BanglaBERT to pre-screen roughly 300,000 collected comments, then manually annotated the model-positive ones, adding 1,794 new instances and rebalancing the four classes. The diagnostic mechanism is the pairing of cosine similarity between class-word embeddings and LIME explanations, which localises the persistent Religio-communal versus Noncommunal confusion to lexical overlap in the embedding space.
What would settle it
Collect a random sample from the same 300,000-comment pool, annotate it with humans only, with no classifier pre-screening, train the identical BanglaBERT and ensemble on that data, and evaluate on a fixed held-out test set; if macro F1 does not reproduce the 0.60 to 0.63 range, the reported gain was an artifact of model-biased data selection.
Extended reading notes
Core claim
The central claim is that four-way classification of Bengali communal violent text can be taken from a baseline macro F1 of 0.36 to 0.60 by fine-tuning BanglaBERT on an augmented dataset, and to 0.63 by ensembling four fine-tuned BanglaBERT variants with a fifth BanglaBERT model and combining their predictions by majority voting. The gains are concentrated in the minority classes: Nondenominational communal F1 rises from 0.16 to 0.69 and Ethno communal from 0.32 to 0.66 after adding the 1,794 manually annotated instances, while BanglaBERT Large and mBERT both underperform the standard model. The paper further claims that the main error pattern is not random: cosine similarity between the embeddings of Noncommunal words like 'human' or 'Bangladesh' and Religio-communal words like 'infidel', 'atheist', 'Allah', or 'Muslim' is very high in BanglaBERT and even higher in BanglaBERT Large, so the models blur the boundary between neutral and religiously charged text. LIME explanations identify a small set of religious trigger words as the dominant cause of Religio versus Noncommunal mistakes, and a manual review of 300 errors attributes 18% of them to annotation problems in the dataset itself.
Load-bearing premise
The manual augmentation step used a fine-tuned BanglaBERT to pre-screen which comments were worth annotating, so the added training data may carry the model's own biases and inflate the reported F1 improvement.
Editorial extensions
If this is right
- With enough manually added examples, minority classes such as Nondenominational and Ethno communal become learnable, while larger or multilingual models do not automatically help: BanglaBERT Large and mBERT both scored lower than standard BanglaBERT.
- Combining several fine-tuned BERT variants pushes macro F1 from 0.60 to 0.63 and specifically reduces the Religio-communal versus Noncommunal confusion, with majority voting slightly outperforming averaging on minority classes.
- The embedding-similarity result implies that any model using BanglaBERT's representations will tend to see neutral text containing religious words as communally violent, so surface lexical cues alone are not sufficient for this task.
- LIME's finding that 65% of Religio-communal misclassifications contain a small set of religious terms suggests that a simple lexicon-based baseline would reproduce the same confusion, and that context-sensitive disambiguation is the real bottleneck.
- Annotator noise found in 18% of the reviewed errors means that part of the measured ceiling is label error rather than model failure, so cleaner annotations should yield further gains independent of architecture.
Reading between the lines
- Because the 1,794 augmented comments were pre-selected by the same BanglaBERT flavour that is later evaluated, a portion of the measured F1 gain could reflect the model confirming its own preferences rather than genuinely learning to generalise; an independent human-only annotation of a random candidate sample would settle this.
- The cosine-similarity result points to a specific cheap intervention worth testing: adding contrastive examples in which religious words co-occur with Noncommunal labels, or debiasing the embedding space, could reduce the dominant confusion more directly than adding more data.
- The methodology should transfer to other low-resource languages with communal-violence dynamics, but the trigger-word sets and embedding geometry would have to be re-derived per language; the reported F1 numbers are unlikely to carry over unchanged.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper addresses detection of Bengali communal violent text on social media. The authors fine-tune BanglaBERT, BanglaBERT Large, and mBERT on a four-class dataset derived from Tasnim et al. [33], expand the minority classes with 1,794 manually annotated comments (pre-screened by a fine-tuned BanglaBERT), and combine five fine-tuned models into ensembles using mean value, voting, and MLP fusion. The headline result is a macro F1 of 0.63 for the voting ensemble, up from a baseline of 0.35–0.36. The paper also presents error analysis, including annotation anomaly review, cosine similarity analysis of word embeddings, and LIME-based interpretability.
Significance. If the quantitative claims were reliable, the paper would be a useful contribution to an under-resourced and socially important task: it provides a new Bengali communal-violence dataset expansion, benchmarks several BERT variants, honestly reports failures of SMOTE/zero-shot/few-shot augmentation, and includes qualitative interpretability analysis. The error-analysis disclosure (18% problematic annotations) and the systematic comparison of augmentation strategies are commendable. However, the central performance claim rests on an evaluation protocol that is contaminated by the model-assisted data selection, so the significance of the reported F1 improvement is currently not established.
major comments (2)
- [Section 4.1, final paragraph] The manual augmentation pipeline uses a fine-tuned BanglaBERT to pre-screen 300,000 comments and only comments predicted as communally violent proceed to manual annotation. The 1,794 added entries are then mixed into the dataset before the 80/20 train/test split, so the test set's minority-class examples are predominantly comments that a BanglaBERT-like model already judged to be violent. Because the final ensemble is composed of BanglaBERT variants, evaluation on this test set measures agreement with the pre-screener's decision boundary rather than generalization to the true distribution of Bengali social media comments. This is a covariate-shift problem, not just label noise. The authors should re-evaluate on a test set drawn from the original 12,791 instances only, and/or on a randomly sampled (unfiltered) subset of the newly collected comments; the difference in macro F1 should be reported.
- [Section 6.1 and Section 5] The error analysis reports that 18% of 300 reviewed misclassifications have problematic annotations, and Section 4.1 does not describe any re-annotation of the new test entries beyond the blind voting process. In addition, no confidence intervals, standard deviations, or significance tests are provided for any F1 score in Tables 6 or 7. The ensemble's improvement over the best single model is 0.03 (0.63 vs 0.60), and the difference between mean value and voting is at most 0.01; given the measured label noise and the absence of uncertainty quantification, this improvement is within plausible noise. The authors should provide bootstrap confidence intervals or multiple-seed results, and ideally re-annotate a held-out test set to measure the effect of label noise.
minor comments (7)
- [Section 4.1, class counts] Section 4.1 states that 1,794 new entries were added, but the listed class counts sum to 1,881 (1,073 + 508 + 300); please reconcile.
- [Section 8 vs Table 6] The Conclusion states the macro F1 improved from 0.36, while Table 6 reports 0.35 for the 'BanglaBert Baseline [No Class Weight]' row; please harmonize.
- [Table 6] The relationship between 'BanglaBert Baseline [No Class Weight]' and 'BanglaBert Baseline' is unclear; state explicitly which rows use class weights.
- [Section 5.1] Section 5.1 says 'F1 scores improved for all classes except Religio communal' after adding paraphrased data, but Table 6 shows Noncommunal staying at 0.44; correct the wording.
- [Section 4.2] Removing stop words is generally not recommended for transformer models such as BERT; justify this preprocessing choice or remove it.
- [Table 10] Table 10 reports cosine similarities but does not specify how the word vectors were extracted (which layer and pooling method); add this detail for reproducibility.
- [Section 6.2.1] The claim that 'the model perceives them as being contextually similar' based on static cosine similarity of word embeddings should be softened, since such similarities do not directly explain the model's contextual predictions; a contextualized embedding analysis would be more appropriate.
Circularity Check
Model-assisted augmentation contaminates the test split: the 0.63 macro-F1 gain is partly self-agreement with the BanglaBERT pre-screener used in Section 4.1, not an independent generalization estimate.
-
fitted input called prediction
[Section 4.1 "Data augmentation"; Section 4.2 "Data preprocessing"; Section 5.2 "Ensemble model"]
"Finally, using a fine-tuned BanglaBERT model, we classified these large amounts of comments as communally violent or not. This classification process was followed by manual annotations through a blind voting system. ... Through this process, we added 1,794 new entries, including 1,073 to the Nondenominational communal class, 508 to the Ethno communal class, and 300 to the Religio communal class. ... Initially, we divided our dataset into 80% train and 20% test data. ... In our research, we used an ensemble model that consisted of five fine-tuned models."
The pipeline makes the evaluation set depend on the model being evaluated. Section 4.1 uses a fine-tuned BanglaBERT to decide which of 300,000 comments are 'communally violent or not'; only pre-screener-positive comments reach manual annotation, so the 1,794 added examples are drawn from one side of BanglaBERT's decision boundary. Section 4.2 then splits the augmented dataset 80/20, putting those model-selected examples into the test set used for the reported F1. Section 5.2's ensemble is four BanglaBERT variations plus a fifth BERT variant, i.e., the same model family as the pre-screener. The 0.36-to-0.63 gain therefore partly measures agreement with the pre-screener's own boundary, not generalization to a random sample of real social-media comments.
full rationale
The only load-bearing circularity I can exhibit is in the data-augmentation/evaluation pipeline. The core modeling recipe—fine-tuning BanglaBERT and other BERT variants on an expert-annotated Bengali corpus—is otherwise self-contained and standard; the dataset citation [33] is provenance, not a self-referential derivation, and no uniqueness theorem or ansatz is smuggled in via citation. However, the central empirical claim (macro F1 0.63, up from 0.36) is not measured on a test set independent of the model family being evaluated. Because a fine-tuned BanglaBERT pre-screened the candidate pool for the 1,794 augmented instances, and because the 80/20 split occurred after augmentation, the test set contains examples selected by the same kind of model whose F1 is reported. This is a selection-bias/self-training circularity: it does not make the score mathematically determined, but it invalidates the claim that the improvement is a clean out-of-sample result. The paper's own error analysis (18% problematic annotations among 300 reviewed misclassifications) further shows the labels are noisy, though that is a correctness issue, not circularity. No significance tests or confidence intervals are reported, so the 0.03 ensemble-over-single-model gap may also be noise. Because the central result still has independent components (original 13k dataset, human labels, comparisons across architectures), I score this as partial circularity rather than complete self-definition.
Assumptions & free parameters
free parameters (4)
- Learning rate =
2e-5
- Batch size =
32
- Class weights =
Inverse frequency
- Early stopping patience =
2 epochs
assumptions (3)
- domain assumption The manually augmented labels are correct.
- domain assumption The dataset from [33] is a reliable ground truth for communal violence.
- domain assumption Macro F1 is an appropriate evaluation metric.
Cite this review
Pith. "Pith review of How Effectively Can BERT Models Interpret Context and Detect Bengali Communal Violent Text?." pith.science (2026). https://pith.science/paper/MJF64AF2
@misc{pith2026250619831,
author = {Pith},
title = {Pith review of: How Effectively Can BERT Models Interpret Context and Detect Bengali Communal Violent Text?},
year = {2026},
howpublished = {\url{https://pith.science/paper/MJF64AF2}},
note = {Machine review of arXiv:2506.19831}
}
read the original abstract
The spread of cyber hatred has led to communal violence, fueling aggression and conflicts between various religious, ethnic, and social groups, posing a significant threat to social harmony. Despite its critical importance, the classification of communal violent text remains an underexplored area in existing research. This study aims to enhance the accuracy of detecting text that incites communal violence, focusing specifically on Bengali textual data sourced from social media platforms. We introduce a fine-tuned BanglaBERT model tailored for this task, achieving a macro F1 score of 0.60. To address the issue of data imbalance, our dataset was expanded by adding 1,794 instances, which facilitated the development and evaluation of a fine-tuned ensemble model. This ensemble model demonstrated an improved performance, achieving a macro F1 score of 0.63, thus highlighting its effectiveness in this domain. In addition to quantitative performance metrics, qualitative analysis revealed instances where the models struggled with context understanding, leading to occasional misclassifications, even when predictions were made with high confidence. Through analyzing the cosine similarity between words, we identified certain limitations in the pre-trained BanglaBERT models, particularly in their ability to distinguish between closely related communal and non-communal terms. To further interpret the model's decisions, we applied LIME, which helped to uncover specific areas where the model struggled in understanding context, contributing to errors in classification. These findings highlight the promise of NLP and interpretability tools in reducing online communal violence. Our work contributes to the growing body of research in communal violence detection and offers a foundation for future studies aiming to refine these techniques for better accuracy and societal impact.
Reference graph
Works this paper leans on
-
[33]
Research progress of zero- shot learning
Sun, X., Gu, J., Sun, H., 2020. Research progress of zero- shot learning. Applied Intelligence 51, 36003614. doi:10.1007/ s10489-020-02075-7
work page 2020
-
[1]
New bounds on the accuracy of majority voting for multiclass classification
Aeeneh, S., Zlatanov, N., Yu, J., 2024. New bounds on the accuracy of majority voting for multiclass classification. IEEE Transactions on Neural Networks and Learning Systems , 1–15doi:10.1109/TNNLS. 2024.3387544
arXiv 2024
-
[2]
Agrawal, D., 2020. Emoji dictionary. URL: https://www.kaggle.com/ divyansh22/emoji-dictionary-1
work page 2020
-
[3]
Akil, A., Sultana, N., Bhattacharjee, A., Shahriyar, R., 2022. BanglaParaphrase: A high-quality Bangla paraphrase dataset, Associ- ation for Computational Linguistics, Online only. pp. 261–272. URL: https://aclanthology.org/2022.aacl-short.33
work page 2022
-
[4]
Hundreds protest in bangladesh over deadly religious violence
Al Jazeera, 2021a. Hundreds protest in bangladesh over deadly religious violence. URL: https://aje.io/7ndrbq. retrieved January 23, 2024
work page 2024
-
[5]
Two hindu men killed, temples vandalised in bangladesh violence
Al Jazeera, 2021b. Two hindu men killed, temples vandalised in bangladesh violence. URL: https://aje.io/uy9gmz. retrieved Febru- ary 18, 2024
work page 2024
-
[6]
Robust hate speech de- tection in social media: A cross -dataset empirical evaluation
Antypas, D., Camacho -Collados, J., 2023. Robust hate speech de- tection in social media: A cross -dataset empirical evaluation. URL: https://arxiv.org/abs/2307.01680, arXiv:2307.01680
arXiv 2023
-
[7]
An- alyzing abusive bangla comments on social media: Nlp & explainable ai, pp
Ashraf, K., Hosen, M.H., Asgar, S., Islam, M.T., Nawar, S., 2024. An- alyzing abusive bangla comments on social media: Nlp & explainable ai, pp. 1–6. doi:10.1109/iCACCESS61735.2024.10499547
Show all 36 references
-
[8]
Bangladeshs forgotten crisis: Land, ethnicity, and violence in chittagong hill tracts
Bashar, I., 2011. Bangladeshs forgotten crisis: Land, ethnicity, and violence in chittagong hill tracts. Counter Terrorist Trends and Analyses 3, 1–5. URL: http://www.jstor.org/stable/26350972
2011
-
[9]
Interpretable multi labeled bengali toxic comments classification using deep learning, pp
Belal, T.A., Shahariar, G.M., Kabir, M.H., 2023. Interpretable multi labeled bengali toxic comments classification using deep learning, pp. 1–6. doi:10.1109/ECCE57851.2023.10101588
2023
-
[10]
Bhattacharjee, A., Hasan, T., Ahmad, W., Mubasshir, K.S., Islam, M.S., Iqbal, A., Rahman, M.S., Shahriyar, R., 2022. BanglaBERT: Language model pretraining and benchmarks for low -resource lan- guage understanding evaluation in Bangla, Association for Compu- tational Linguisti...
2022
-
[11]
Communal conflict, civil war, and the state: Complexities, connections, and the case of sudan
Brosché, J., Elfversson, E., 2012. Communal conflict, civil war, and the state: Complexities, connections, and the case of sudan. African Journal on Conflict Resolution 12, 33 –60. URL: https://www.ajol. info/index.php/ajcr/article/view/78700
2012
-
[12]
Smote for imbalanced classification with python
Brownlee, J., 2021. Smote for imbalanced classification with python. URL: https://machinelearningmastery.com/ smote-oversampling-for-imbalanced-classification/
2021
-
[13]
A weakly supervised Bayesian model for violence detection in social media, Asian Federation of Natural Language Processing, Nagoya, Japan
Cano Basave, A.E., He, Y., Liu, K., Zhao, J., 2013. A weakly supervised Bayesian model for violence detection in social media, Asian Federation of Natural Language Processing, Nagoya, Japan. pp. 109–117. URL: https://aclanthology.org/I13-1013
2013
-
[14]
Bangla hate speech detection on social media using attention -based recurrent neural network
Das, A.K., Asif, A.A., Paul, A., Hossain, M.N., 2021. Bangla hate speech detection on social media using attention -based recurrent neural network. Journal of Intelligent Systems 30, 578 –591. doi:10. 1515/jisys-2020-0060
2021
-
[15]
BERT: pre- training of deep bidirectional transformers for language understand- ing
Devlin, J., Chang, M., Lee, K., Toutanova, K., 2018. BERT: pre- training of deep bidirectional transformers for language understand- ing. CoRR abs/1810.04805. URL: http://arxiv.org/abs/1810.04805
2018 arXiv
-
[16]
Bengali Hate Speech Detection Dataset
Dey, K., Sumon, Cochez, M., Karim, M.R., 2022. Bengali Hate Speech Detection Dataset. UCI Machine Learning Repository. URL: https://doi.org/10.24432/C5PD07
2022 doi
-
[17]
Bengali stop word and phrase detection mecha- nism
Haque, R.U., Mridha, M.F., Hamid, M.A., Abdullah -Al-Wadud, M., Islam, M.S., 2020. Bengali stop word and phrase detection mecha- nism. Arabian Journal for Science and Engineering 45, 33553368. doi:10.1007/s13369-020-04388-8
2020 doi
-
[18]
Few -shot learning
IBM, 2024. Few -shot learning. URL: https://www.ibm.com/topics/ few-shot-learning. retrieved January 29, 2024
2024
-
[19]
Classification benchmarks for under -resourced bengali language based on multichannel convolutional-lstm network
Karim, M.R., Chakravarthi, B.R., McCrae, J.P., Cochez, M., 2020. Classification benchmarks for under -resourced bengali language based on multichannel convolutional-lstm network. URL: https: //arxiv.org/abs/2004.07807 , arXiv:2004.07807
2020 arXiv
-
[20]
Digital 2023: Bangladesh
Kemp, S., 2023. Digital 2023: Bangladesh. URL: https:// datareportal.com/reports/digital-2023-bangladesh
2023
-
[21]
G-bert: An efficient method for identifying hate speech in bengali texts on social media
Keya, A.J., Kabir, M.M., Shammey, N.J., Mridha, M.F., Islam, M.R., Watanobe, Y., 2023. G-bert: An efficient method for identifying hate speech in bengali texts on social media. IEEE Access 11, 79697– 79709. doi:10.1109/ACCESS.2023.3299021
2023
-
[22]
Khang Hsien, Y., Arabee Abdul Salam, Z., Kasinathan, V., 2022. Cyber bullying detection using natural language processing (nlp) and text analytics, in: 2022 IEEE International Conference on Distributed Computing and Electrical Circuits and Electronics (ICDCECE), pp. 1–4. doi:1...
2022
-
[23]
Smote: Practical consideration and limitations
Kim, M., 2023. Smote: Practical consideration and limitations. URL: https://medium.com/@minjukim023/ smote-practical-consideration-limitations-f0d926b661a8
2023
-
[24]
ComMA@ICON: Multi- lingual gender biased and communal language identification task at ICON-2021, NLP Association of India (NLPAI), NIT Silchar
Kumar, R., Ratan, S., Singh, S., Nandi, E., Devi, L.N., Bhagat, A., Dawer, Y., Lahiri, B., Bansal, A., 2021. ComMA@ICON: Multi- lingual gender biased and communal language identification task at ICON-2021, NLP Association of India (NLPAI), NIT Silchar. pp. 1–
2021
-
[25]
URL: https://aclanthology.org/2021.icon-multigen.1
2021
-
[26]
Noorana, M., 2018. Pre electoral violence in 2001 parliamentary elec- tions in bangladesh: Empirical evidence from bagerhat and narayan- ganj districts, social science review: The dhaka university stud- ies 35, 231–246. URL: https://www.researchgate.net/publication/ 353636999
2018
-
[27]
Deep learning algorithms for cyber -bulling detection in social media platforms
Obaida, M.H., Elkaffas, S.M., Guirguis, S.K., 2024. Deep learning algorithms for cyber -bulling detection in social media platforms. IEEE Access 12, 76901–76908. doi:10.1109/ACCESS.2024.3406595
2024
-
[28]
Zero-shot learning (zsl) explained
Polat, G., 2023. Zero-shot learning (zsl) explained. URL: https: //encord.com/blog/zero -shot-learning-explained/
2023
-
[29]
Myanmars religious and ethnic conflicts: No end in sight
ReliefWeb, 2017. Myanmars religious and ethnic conflicts: No end in sight. URL: https://reliefweb.int/report/myanmar/ myanmars-religious-and-ethnic-conflicts-no-end-sight. retrieved January 17, 2024
2017
-
[30]
Hate speech detection in the bengali language: A dataset and its baseline evaluation
Romim, N., Ahmed, M., Talukder, H., Islam, M.S., 2020. Hate speech detection in the bengali language: A dataset and its baseline evaluation. arXiv preprint arXiv:2012.09686 URL: https://arxiv. org/abs/2012.09686
2020 arXiv
-
[31]
The Communal Conflict In India Causes And Peaceful Solutions
Sahai, P., 2020. The Communal Conflict In India Causes And Peaceful Solutions. Rajiv Gandhi Institute for Contemporary Stud- ies, New Delhi. URL: https://www.rgics.org/wp-content/uploads/ Communal-Conflict-in-India.pdf
2020
-
[32]
Changes in social and religious prac- tices of disputing communities after riot: A case study on communal violence in ramu, in: Journal of Philosophy, Culture and Religion, p
Soruar, W., Uddin, M.M., 2021. Changes in social and religious prac- tices of disputing communities after riot: A case study on communal violence in ramu, in: Journal of Philosophy, Culture and Religion, p. 13. doi:10.7176/JPCR/51-03
2021 doi
-
[34]
Mapping violence: Developing an extensive framework to build a bangla sectarian expression dataset from so- cial media interactions
Tasnim, N., Gupta, S.S., Juee, F.I., Tahsin, A., Ghum, P., Fatema, K., Haque, M., Farzana, W., Nasir, P., KhudaBukhsh, A., Sadeque, F., Sushmit, A., 2024. Mapping violence: Developing an extensive framework to build a bangla sectarian expression dataset from so- cial media int...
2024 doi
-
[35]
Automated bengali abusive text clas- sification: Using deep learning techniques, pp
Titli, S.R., Paul, S., 2023. Automated bengali abusive text clas- sification: Using deep learning techniques, pp. 1 –6. doi:10.1109/ ICAECIS58353.2023.10170294
2023
-
[36]
Ensemble MLP Classifier Design
Windeatt, T., 2008. Ensemble MLP Classifier Design. Springer Berlin Heidelberg, Berlin, Heidelberg. pp. 133 –147. doi:10.1007/ 978-3-540-79474-5_6
2008
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.