Pith. sign in

REVIEW 3 major objections 6 minor 21 references

Understanding Public Perception of Crime in Bangladesh: A Transformer-Based Approach with Explainability

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read A multilingual transformer can classify Bangla social-media comments about crime into positive, negative, and neutral sentiment with 97% accuracy.

desk verdict Plausible dataset, standard transformer, unverifiable accuracy: the 97% and the cross-dataset SOTA claim both need to be fixed before this is citable. read the letter →

arxiv 2507.21234 v1 pith:3J3RS3W5 submitted 2025-07-28 cs.CL

classification cs.CL
keywords BanglasentimentanalysisXLM-RoBERTacrimenewssocialmediacommentsexplainableAILIMEdataaugmentationlow-resourcelanguage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that a multilingual transformer, XLM-RoBERTa Base, can reliably read public sentiment in Bangla-language social media comments about crime news. The authors assembled and annotated a new dataset of 28,528 comments from Facebook and YouTube, applied round-trip machine-translation augmentation, and fine-tuned XLM-RoBERTa to label each comment positive, negative, or neutral. They report 97% test accuracy and a 0.97 macro F1, beating LSTM, attention-augmented LSTM, BanglaBERT, and earlier published results on the same task. The contribution matters because Bengali is a low-resource language, and the paper argues that a fine-tuned multilingual transformer can extract policy-relevant signals from public reactions to crime.

What carries the argument

The engine of the argument is XLM-RoBERTa Base, a multilingual transformer pretrained on 100 languages that produces contextual token representations shared across languages; the paper fine-tunes it on the Bangla dataset with a cross-entropy loss, AdamW, and a 2e-5 learning rate. Round-trip machine translation (Bangla to English and back) augments the dataset to reduce sparsity. LIME, a post-hoc explainability tool, provides the interpretability analysis by weighting word fragments according to their effect on individual predictions.

What would settle it

Resplit the 28,528 comments so that every round-trip-translated variant of a source comment stays in the same partition, retrain on the same hyperparameters, and check test accuracy; if it falls substantially below 97%, the reported number depended on near-duplicate leakage rather than genuine generalization.

Watch

Extended reading notes

Core claim

On a newly built dataset of 28,528 Bangla crime-news comments, a fine-tuned XLM-RoBERTa Base model classifies comment sentiment into three classes with 97% accuracy on both validation and test splits and a macro F1 of 0.97 across positive, negative, and neutral classes. The authors compare this model with LSTM (93.4% test), LSTM with attention (94.2%), and BanglaBERT (91.2%), and with previously reported models such as KNN at 96.25% and CNN-BiLSTM at 95.71%, concluding that their model generalizes better despite the informal, code-mixed nature of Bengali social media text. They also apply LIME to individual predictions to show which characters or phrases most influence the classification.

Load-bearing premise

The reported 97% test accuracy assumes that no augmented or duplicate copies of training comments ended up in the test set, because the paper does not state whether the split was made before or after round-trip translation augmentation.

Editorial extensions

If this is right

  • On the reported split, a fine-tuned XLM-RoBERTa model yields 97% test accuracy and macro F1 of 0.97, with every class above 0.95 F1.
  • The comparison against LSTM, attention-LSTM, and BanglaBERT implies that a multilingual transformer can outperform a Bangla-specific pretrained model on this crime-comment domain.
  • The 3.2% F1 improvement from cleaning raw text indicates that preprocessing contributes to the model's performance beyond architecture choice.
  • The confusion matrix shows the model most often mistakes positive and negative comments, which is where future data collection or modeling effort would matter most.
  • The authors conclude that such models can support policy formulation and crime prevention by tracking public sentiment in social media.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not report whether the data split was made before or after augmentation, so a reader cannot yet distinguish genuine generalization from near-duplicate leakage; re-splitting after augmentation would settle this.
  • The same fine-tuning recipe could plausibly transfer to other low-resource languages or to neighboring domains such as health or political discourse, but each transfer would need its own leakage-free evaluation.
  • The reported confusion between positive and negative comments suggests that sarcasm and irony, common in crime-news reactions, remain a boundary that the current three-class setup does not model.
  • Because the dataset draws mainly from urban Facebook and YouTube users, the title's claim about public perception in Bangladesh is broader than the sample frame supports; a demographically stratified collection would test that scope.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces a new Bengali-language dataset of 28,528 social media comments on crime-related news, labeled positive, negative, and neutral, and fine-tunes XLM-RoBERTa Base for sentiment classification. The authors report 97% test accuracy and macro F1, compare against LSTM, LSTM+Attention, BanglaBERT, and selected prior models, and apply a LIME-based explainability analysis. The central claim is that the proposed model outperforms existing state-of-the-art Bengali sentiment analysis methods.

Significance. If the empirical results are sound, this work would provide a useful dataset for a low-resource language and demonstrate the effectiveness of multilingual transformers on Bengali crime-related sentiment. The dataset itself is a potential contribution, especially given the scarcity of labeled Bengali social media data. However, the current manuscript does not provide sufficient evidence to verify the central 97% accuracy claim: the dataset and code are not released, the relationship between augmentation and the train/test split is ambiguous, and the comparison with prior work is not conducted on a common benchmark. These issues are load-bearing because they directly affect whether the reported performance represents generalizable classification skill or an artifact of the evaluation protocol.

major comments (3)
  1. [Section III-D and IV-A, Table V] The paper does not state whether the train/validation/test split was performed before or after the round-trip translation augmentation, nor whether augmented near-duplicates were removed from the test set. Table V shows the test support totals 2,853, which is exactly 10% of the original 28,528 comments; if the augmented pool had been split, a 10% test split would yield more than 2,853 instances. The authors must clarify the split order and any deduplication policy. Without this information, the reported 97% test accuracy cannot be distinguished from memorization of augmented paraphrases and cannot be verified by readers.
  2. [Table VI, Section IV-B] The claim that the proposed model "outperforms existing state-of-the-art methods in Bangla sentiment analysis" is not supported by the evidence in Table VI, because the cited accuracies (KNN on e-commerce reviews, LSTM on COVID-19 tweets, CNN-BiLSTM on BangDSA) come from different datasets, tasks, and class distributions. Cross-corpus accuracy comparison is meaningless as a benchmark. To substantiate the state-of-the-art claim, the authors must either run the baseline models on the same test split of their own dataset or evaluate on a shared public benchmark.
  3. [Sections III-A and III-B] The dataset is a primary contribution, but it is not released, and the annotation methodology is not described beyond class counts. There is no information on annotator training, labeling guidelines, inter-annotator agreement (e.g., Cohen's kappa), or quality control for the 28,528 comments. The paper also acknowledges possible demographic bias but does not quantify it. Without data release and annotation details, the 97% accuracy claim cannot be reproduced or independently assessed, and the dataset's utility is reduced to an in-house resource.
minor comments (6)
  1. [Introduction, References [4] and [5]] Citation numbering is inconsistent with the reference list: crime statistics are attributed to [4] (scikit-learn) and [5] (NumPy) in the Introduction, while the actual sources (Dhaka Tribune, etc.) are not properly cited.
  2. [Section III and Table VI] The reference [19] is used both for the XLM-RoBERTa model architecture (Section III) and for the prior LSTM study by Pran et al. (Table VI); these are distinct works and should have separate reference numbers.
  3. [Section IV-B, Figure 5] The explainability section cites reference [7] (SHAP, Lundberg & Lee) but the described procedure (explainer.explain_instance) and Figure 5 are characteristic of LIME; the citation should be corrected to the appropriate LIME reference.
  4. [Table IV] The validation and test accuracies are reported as identical for LSTM (93.4%), LSTM+Attention (94.2%), and the proposed model (97.0%); this is surprising and should be checked, since random variation usually produces slight differences between validation and test splits.
  5. [Section IV-A] The exact train/validation/test split ratio is not stated; the authors mention using scikit-learn for splitting but do not provide the ratio or random seed, which is necessary for reproducibility.
  6. [Section IV-B, Figure 5] The description of LIME output refers to "influential characters" (ল, ভ, স, etc.), but these appear to be individual Bengali letters rather than whole words or subword tokens; the explanation would be clearer if the model's tokenizer output were shown.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports a supervised empirical evaluation whose central accuracy claim is measured on a held-out test split, not derived from its own inputs by construction.

full rationale

This is an empirical NLP paper rather than a derivation-based one. The central claim, 97% test accuracy and 0.97 macro F1 for XLM-RoBERTa on a newly collected Bengali crime-comment dataset, is obtained by training a model and evaluating it on a test split. No equation in the paper defines the reported accuracy in terms of the training labels or the fitted hyperparameters, and the paper does not invoke a load-bearing uniqueness theorem or a chain of self-citations. Hyperparameters were selected on a development set, but the paper reports separate training, validation, and test numbers and the test performance is not itself a fitted parameter. The SOTA comparison in Table VI compares models trained on different datasets/tasks, which is a benchmark-comparability weakness rather than circularity. The round-trip augmentation described in Section III-D raises a legitimate validity concern: if augmented near-duplicates of training comments leaked into the test set, the 97% figure could partly reflect memorization rather than generalization. However, the paper contains no statement establishing such leakage, and the review rules require exhibiting a specific reduction rather than speculating about possible data handling. Since no circular step can be quoted from the paper, the appropriate finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim is an empirical accuracy number. The paper's own contribution is a dataset plus a fine-tuned model; both depend on the listed assumptions about labeling quality, augmentation semantics, and split hygiene. No free parameters in the scientific sense are derived: the hyperparameters are chosen by grid search and the model weights are learned, not fitted to a target constant.

free parameters (5)
  • Learning rate = 2e-5
    Selected via grid search over {1e-5, 2e-5, 3e-5} on the validation set; the reported accuracy depends on this choice.
  • Batch size = 16
    Selected via grid search over {8, 16, 32} on the validation set.
  • Number of epochs = 15
    Selected via grid search over {10, 15, 20} on the validation set.
  • Weight decay = 0.01
    Set by the authors without a reported tuning search; a hand-set constant influencing the result.
  • Augmentation ratio = unspecified
    The augmentation pipeline (Bangla to English to Bangla via GoogleTrans) is described, but the ratio of augmented samples to original samples is not reported, and whether the split happens before or after augmentation is not stated.
assumptions (4)
  • domain assumption The human annotations of positive/negative/neutral for the 28,528 comments are correct and consistent.
    No annotation protocol, number of annotators, or inter-annotator agreement is reported; the labels are treated as ground truth throughout.
  • domain assumption Round-trip machine translation (Bangla to English to Bangla) preserves the sentiment label of each comment.
    The augmentation in Section III-D assumes that back-translated text retains its sentiment, but no validation of this assumption is provided.
  • domain assumption The test set is disjoint from the training set and contains no augmented near-duplicates.
    Section III-D and IV-A do not describe how the split is made relative to augmentation; if augmented copies appear in the test set, the reported accuracy is inflated.
  • domain assumption XLM-RoBERTa provides a suitable representation for Bangla code-mixed text.
    The paper relies on the pretrained multilingual model without fine-tuning on a Bangla-specific objective; it is a standard assumption but not validated in this work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding Public Perception of Crime in Bangladesh: A Transformer-Based Approach with Explainability." pith.science (2026). https://pith.science/paper/3J3RS3W5

@misc{pith2026250721234,
  author       = {Pith},
  title        = {Pith review of: Understanding Public Perception of Crime in Bangladesh: A Transformer-Based Approach with Explainability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3J3RS3W5}},
  note         = {Machine review of arXiv:2507.21234}
}
read the original abstract

In recent years, social media platforms have become prominent spaces for individuals to express their opinions on ongoing events, including criminal incidents. As a result, public sentiment can shift dynamically over time. This study investigates the evolving public perception of crime-related news by classifying user-generated comments into three categories: positive, negative, and neutral. A newly curated dataset comprising 28,528 Bangla-language social media comments was developed for this purpose. We propose a transformer-based model utilizing the XLM-RoBERTa Base architecture, which achieves a classification accuracy of 97%, outperforming existing state-of-the-art methods in Bangla sentiment analysis. To enhance model interpretability, explainable AI technique is employed to identify the most influential features driving sentiment classification. The results underscore the effectiveness of transformer-based models in processing low-resource languages such as Bengali and demonstrate their potential to extract actionable insights that can support public policy formulation and crime prevention strategies.

Figures

Figures reproduced from arXiv: 2507.21234 by the authors.

Figure 1
Figure 1. Overview of The Framework Table I: Sample Examples of Comments on Recent Crime Events from The Dataset Comments Source Information এই ধোঁকাবাজি গুলো আজ থেকে নয় অনেক আগে থেকেই চলতেছে Collected from Facebook page কিন্তু যারা টাকা ইনভেস্ট করেছে তারা বেঁচে থেকেও মরে গেল। Collected from YouTube news report One limitation of the collected dataset is its potential lack of demographic and geographic diversity. Since the co… view at source ↗
Figure 2
Figure 2. Data Augmentation The AdamW optimization technique is used to train the suggested model, and it works well with transformer-based architectures. The major indicator used for performance evaluation is classification accuracy. Table III: Parameter Details of The Proposed Model Hyperparameters Values Batch Size 16 Learning Rate 0.00002 Number of Epochs 15 Weight decay 0.01 Optimizer AdamW Loss Function Cross-Entropy Ac… view at source ↗
Figure 3
Figure 3. Graph of accuracy and loss for the proposed model [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Confusion Matrix [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 20 canonical work pages

  1. [7]

    A unified approach to inter- preting model predictions,

    S. M. Lundberg and S.-I. Lee, “A unified approach to inter- preting model predictions,” Advances in neural information processing systems, vol. 30, 2017

  2. [19]

    Analysis of bangladeshi people’s emotion during covid-19 in social media using deep learning,

    M. S. A. Pran, M. R. Bhuiyan, S. A. Hossain, and S. Abujar, “Analysis of bangladeshi people’s emotion during covid-19 in social media using deep learning,” in 2020 11th International Conference on Computing, Communication and Networking Technologies (ICCCNT). IEEE, 2020, pp. 1–6

  3. [1]

    Data set for sentiment analysis on bengali news comments and its baseline evaluation,

    M. A.-U.-Z. Ashik, S. Shovon, and S. Haque, “Data set for sentiment analysis on bengali news comments and its baseline evaluation,” in 2019 International conference on bangla speech and language processing (ICBSLP) . IEEE, 2019, pp. 1–5

  4. [2]

    Cricket sentiment analysis from bangla text using recurrent neural network with long short term memory model,

    M. F. Wahid, M. J. Hasan, and M. S. Alom, “Cricket sentiment analysis from bangla text using recurrent neural network with long short term memory model,” in 2019 International Con- ference on Bangla Speech and Language Processing (ICBSLP) . IEEE, 2019, pp. 1–4

  5. [3]

    Abusive comments detection in bangla-english code-mixed and transliterated text,

    M. Jahan, I. Ahamed, M. R. Bishwas, and S. Shatabda, “Abusive comments detection in bangla-english code-mixed and transliterated text,” in 2019 2nd international conference on innovation in engineering and technology (ICIET). IEEE, 2019, pp. 1–6

  6. [4]

    Scikit-learn: Machine learning in python,

    F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg et al. , “Scikit-learn: Machine learning in python,” the Journal of machine Learning research , vol. 12, pp. 2825–2830, 2011

  7. [5]

    Array programming with numpy,

    C. R. Harris, K. J. Millman, S. J. Van Der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith et al. , “Array programming with numpy,” Nature, vol. 585, no. 7825, pp. 357–362, 2020

  8. [6]

    Matplotlib: A 2d graphics environment,

    J. D. Hunter, “Matplotlib: A 2d graphics environment,” Com- puting in science & engineering , vol. 9, no. 03, pp. 90–95, 2007

Show all 21 references
  1. [8]

    Improving sentiment classification in low-resource bengali language utilizing cross-lingual self-supervised learn- ing,

    S. Sazzed, “Improving sentiment classification in low-resource bengali language utilizing cross-lingual self-supervised learn- ing,” in International conference on applications of natural language to information systems . Springer, 2021, pp. 218–230

  2. [9]

    The state and fate of linguistic diversity and inclusion in the nlp world,

    P. Joshi, S. Santy, A. Budhiraja, K. Bali, and M. Choudhury, “The state and fate of linguistic diversity and inclusion in the nlp world,” arXiv preprint arXiv:2004.09095 , 2020

  3. [10]

    Bengali senti- ment analysis of e-commerce product reviews using k-nearest neighbors,

    M. T. Akter, M. Begum, and R. Mustafa, “Bengali senti- ment analysis of e-commerce product reviews using k-nearest neighbors,” in 2021 International conference on information and communication technology for sustainable development (ICICT4SD). IEEE, 2021, pp. 40–44

  4. [11]

    Depression analysis from social media data in bangla language using long short term memory (lstm) recurrent neural network technique,

    A. H. Uddin, D. Bapery, and A. S. M. Arif, “Depression analysis from social media data in bangla language using long short term memory (lstm) recurrent neural network technique,” in 2019 in- ternational conference on computer, communication, chemical, materials and electronic ...

  5. [12]

    Employing machine learning techniques on sen- timent analysis of google play store bangla reviews,

    M. M. J. Soumik, S. S. M. Farhavi, F. Eva, T. Sinha, and M. S. Alam, “Employing machine learning techniques on sen- timent analysis of google play store bangla reviews,” in 2019 22nd International Conference on Computer and Information Technology (ICCIT). IEEE, 2019, pp. 1–5

  6. [13]

    Sentiment analysis on movie review data using machine learning approach,

    A. Rahman and M. S. Hossen, “Sentiment analysis on movie review data using machine learning approach,” in 2019 inter- national conference on bangla speech and language processing (ICBSLP). IEEE, 2019, pp. 1–4

  7. [14]

    The opencv library

    G. Bradski, “The opencv library. ” Dr. Dobb’s Journal: Software Tools for the Professional Programmer , vol. 25, no. 11, pp. 120–123, 2000

  8. [15]

    Data structures for statistical computing in python

    W. McKinney et al. , “Data structures for statistical computing in python. ” SciPy, vol. 445, no. 1, pp. 51–56, 2010

  9. [16]

    Keras: Deep learning library for theano and tensorflow,

    F. Chollet et al. , “Keras: Deep learning library for theano and tensorflow,” URL: https://keras. io/k , vol. 7, no. 8, p. T1, 2015

  10. [17]

    based measurement of customer satisfac- tion in mobile service: Sentiment analysis and vikor approach,

    D. Kang and Y. Park, “based measurement of customer satisfac- tion in mobile service: Sentiment analysis and vikor approach,” Expert Systems with Applications , vol. 41, no. 4, pp. 1041–1050, 2014

  11. [18]

    Tensorflow: Large-scale machine learning on heterogeneous systems (2015), software available from tensorflow. org,

    M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin et al. , “Tensorflow: Large-scale machine learning on heterogeneous systems (2015), software available from tensorflow. org,” 2019

  12. [20]

    Using social media to mine and analyze public opinion related to covid-19 in china,

    X. Han, J. Wang, M. Zhang, and X. Wang, “Using social media to mine and analyze public opinion related to covid-19 in china,” International journal of environmental research and public health, vol. 17, no. 8, p. 2788, 2020

  13. [21]

    Sentiment analysis of bangla language using a new comprehensive dataset bangdsa and the novel feature metric skipbangla-bert,

    M. S. Islam and K. M. Alam, “Sentiment analysis of bangla language using a new comprehensive dataset bangdsa and the novel feature metric skipbangla-bert,” Natural Language Pro- cessing Journal, vol. 7, p. 100069, 2024. 6

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.