REVIEW 5 major objections 5 minor 15 references
RideKE: Leveraging Low-Resource, User-Generated Twitter Content for Sentiment and Emotion Detection in Kenyan Code-Switched Dataset
T0 review · 5 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read The paper introduces RideKE, a 29,000-tweet dataset of Kenyan code-switched ride-hailing content, and shows that XLM-R outperforms three other transformer models on sentiment classification, while emotion detection remains weak.
desk verdict The dataset is a real contribution, but inconsistent numbers and unvalidated ChatGPT labels make the benchmark claims unreliable as written. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying artifact is RideKE, a dataset of 29,623 tweets collected from Kenyan ride-hailing hashtags, filtered by user location and language, and preprocessed into about 70% Kenyan-accented English with the rest in Swahili and Sheng. Its structure matters: a small labeled training set (553 human labels plus 636 automatic labels), a 2,000-tweet human-annotated test set, and a roughly 27,000-tweet unlabeled pool. The other load-bearing mechanism is the semi-supervised loop, in which each pretrained model labels unlabeled samples whose predicted probability exceeds the 75th percentile for a class and then retrains on those pseudo-labels for four epochs. Four transformers — DistilBERT, mBERT, XLM-R, and AfriBERTa — are fine-tuned with a combined categorical cross-entropy loss over sentiment and emotion heads, and evaluated on the human test set.
What would settle it
Re-annotate the 636 automatically labeled tweets and a random sample of high-confidence pseudo-labeled tweets with two fresh Kenyan annotators; if their agreement with the auto-labels is near chance, or if retraining on purely human labels changes which model leads, the paper's central benchmark claim would be contradicted.
Extended reading notes
Core claim
The paper's central discovery is that a modest, partially annotated corpus of noisy social-media text in a low-resource code-switched setting can support meaningful model evaluation: XLM-R reaches 69.2% accuracy and 66.1% F1 on sentiment in the fully supervised condition, outperforming mBERT, DistilBERT, and AfriBERTa on this data. Emotion classification is the harder task, with DistilBERT leading at 59.8% accuracy but only 31% macro F1, and all models showing a systematic pull toward neutral sentiment—AfriBERTa most strongly. The paper also finds that its semi-supervised loop, which uses high-confidence pseudo-labels, yields mixed gains: it helps mBERT on emotion (29.7% vs. 26.5% F1) but does not surpass the best supervised sentiment run, so the authors present semi-supervision as a complement rather than a replacement for human annotation. Taken together, the claim is that RideKE gives the low-resource NLP community a new benchmark and an initial model ranking for Kenyan ride-hailing discourse.
Load-bearing premise
The reported results assume that the automatic labels, both the 636 LLM-generated training labels and the high-confidence pseudo-labels from the models themselves, are accurate enough to stand in for human annotation; if those labels are noisy, the model ranking and accuracy figures could reflect label bias rather than true ability.
Editorial extensions
If this is right
- RideKE becomes a public benchmark for Kenyan code-switched sentiment and emotion classification, giving low-resource NLP a shared evaluation set tied to ride-hailing discourse.
- XLM-R is a strong default for sentiment analysis on this data, reaching the highest accuracy and F1 among the tested models in the supervised setting.
- Semi-supervised training with a 75th-percentile confidence threshold helps some models on some tasks, such as mBERT on emotion, but does not consistently beat supervised training.
- Emotion detection on RideKE remains an open problem: the best macro F1 is about 31%, and happy and fear classes are essentially unlearned by most models.
- All four models over-predict neutral sentiment, and AfriBERTa shows the strongest neutral bias, so model choice affects the label distribution, not just accuracy.
Reading between the lines
- Beyond the paper: a ride-hailing company could deploy a RideKE-tuned sentiment monitor to track complaint spikes in Nairobi, but the documented neutral bias would need calibration before use.
- Beyond the paper: the near-zero F1 for happy and fear suggests those emotion classes are too rare or too ambiguous in the data; collecting targeted tweets for these classes would test whether the low scores reflect data scarcity or label ambiguity.
- Beyond the paper: AfriBERTa's weak performance despite African-language pretraining hints that domain and register, not language-family match, drive results; experimenting with pretraining on Sheng-heavy social-media text would test this.
- Beyond the paper: the 75th-percentile pseudo-label threshold is an arbitrary knob; sweeping this threshold could reveal whether the mixed semi-supervised results are robust or threshold-dependent.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces RideKE, a corpus of 29,623 tweets about Kenyan ride-hailing services (Uber, Bolt, Little Cab), containing a mix of English, Swahili, and Sheng. A subset of 2,553 tweets is human-annotated for sentiment (positive/negative/neutral) and eight emotion categories, and a further 636 tweets are labeled by ChatGPT; 27,090 tweets are used as unlabeled data. The authors fine-tune DistilBERT, mBERT, XLM-R, and AfriBERTa under supervised and semi-supervised settings, and report that XLM-R performs best for sentiment (69.2% accuracy, 66.1% F1) while DistilBERT and mBERT lead on emotion. The contributions claimed are the new dataset, the comparison of four transformer models, and analysis of code-switched Kenyan data.
Significance. If the results are reproducible, RideKE would be a useful resource for low-resource African NLP, since there are few public code-switched sentiment and emotion datasets focused on Kenyan ride-hailing. The human annotation effort is a strength, and the paper makes a concrete attempt to address data scarcity through ChatGPT augmentation and semi-supervised learning. However, the significance is currently limited by unresolved numeric inconsistencies and by the absence of validation for the automatically generated labels, so the central model-ranking claim is not yet established.
major comments (5)
- [Abstract, §5.1, Table 5] The reported headline numbers are internally inconsistent. The abstract and Table 5 give XLM-R supervised sentiment accuracy 69.2% and F1 66.1%, but Section 5.1 says the same condition achieves 62.5% accuracy and 66.7% F1, and gives XLM-R semi-supervised as 62.1% accuracy and 68.3% F1, whereas Table 5 reports 67.2% and 64.1% for that system. The abstract also assigns the 26.5% emotion F1 to mBERT semi-supervised, while Table 5 and Section 5.2 report 29.7% for that condition (26.5% is the supervised value). At a minimum, these figures must be reconciled; ideally the authors should report the exact evaluation configuration (e.g., checkpoint selection, number of runs) that produced each table.
- [§3.5.2, §4.2.1] Section 3.5.2 and Table 4 state that Set A, the supervised training set, consists of 553 human labels and 636 ChatGPT-generated labels; Section 4.2.1 says fine-tuning used 1,189 labeled tweets. Thus 54% of the supervised training signal is machine-generated. The paper reports no validation of these ChatGPT labels, no human agreement on them, and no error analysis. Because the central claim is a model ranking, a ranking obtained from this training mixture may reflect sensitivity to ChatGPT label noise rather than true ability on Kenyan code-switched text. The authors should report ChatGPT-label accuracy against human labels on a held-out subset, and should show that the ranking in Table 5 is stable when training on human labels alone.
- [§4.2.2] Section 4.2.2 describes a semi-supervised loop in which each model labels the unlabeled set with its own predictions and retains only samples above the 75th-percentile confidence threshold. This is a self-referential training signal: the model's confidence is used as a proxy for label correctness, with no external check. The resulting pseudo-labels can reinforce systematic errors, especially for rare emotion classes where the model tends to be overconfident. The paper should report the accuracy of the pseudo-labels on a human-annotated subset, or compare the semi-supervised models against a supervised-only baseline under identical hyperparameters, before interpreting the SSL results in Table 5.
- [§5.2, Tables 7–9] Section 5.2 and Tables 7–9 show that several emotion classes have zero F1 for every model (Happy in Table 7, Fear in Table 8, and Surprise for most models in Table 9). The aggregate emotion F1 values in Table 5 therefore average over classes the models never predict. This makes the emotion ranking fragile: a model that assigns every tweet to a single high-frequency class can appear competitive in accuracy while having zero performance on most of the label set. The authors should report the class distribution in Set B, per-class support, and either use a metric robust to extreme imbalance or restrict the emotion benchmark to classes with enough positive examples.
- [Table 5, Appendix A.5] Table 5 states that results are averages over multiple runs, but no variance, confidence intervals, or significance tests are reported. The differences that drive the ranking are small (e.g., mBERT supervised vs. semi-supervised sentiment F1: 59.6% vs. 59.8%), and without run-level statistics these differences are not interpretable. In addition, Appendix A.5 shows two identical tweet rows from the same date, and the text only says that duplicates were removed from Set C, not from Sets A and B; if duplicated tweets cross the train/test split, the evaluation is optimistic. The authors should report standard deviations across seeds, run a paired test (e.g., McNemar) for the headline comparisons, and clarify or implement duplicate removal for all splits.
minor comments (5)
- [Title/Abstract] The title on the first page reads 'Kenya`n Code-Switched Dataset'; this should be fixed to 'Kenyan'.
- [§3.3, Table 10] Table 10 reports large counts of Indonesian, Norwegian, Danish, and other languages; the paper should explain how these were handled, since the dataset is described as English–Swahili–Sheng code-switched.
- [§4.2.1, Figure 3] Section 4.2.1 references Figure 3 as showing the 'combined categorical cross-entropy loss function'; Figure 3 actually shows training loss curves, not the loss definition, so the reference should be clarified.
- [§5.3, Abstract] The abstract and Section 5.3 refer to 'Afri-BERT' while the model is called AfriBERTa elsewhere; a single consistent name should be used.
- [§5.4] Section 5.4 says semi-supervised mBERT improved sentiment F1 to 59.8% from 59.6%; this difference is well within what could be expected from random seed variation, so it should not be described as a 'clear performance enhancement' without variance estimates.
Circularity Check
No significant circularity: the central model comparison is an independent empirical evaluation on a held-out human-labeled test set; the self-training loop and minor self-citation are not load-bearing.
full rationale
The paper's central claim is a new benchmark dataset plus a ranked comparison of four pretrained transformers. The test set (Set B) is described as 'manually annotated by human annotators' (Section 3.5.2), and the reported accuracy/F1 values in Table 5 are evaluated after training on that held-out test set, so the ranking is not forced by construction. The semi-supervised phase (Section 4.2.2) uses each model's own high-confidence predictions as pseudo-labels for additional training; this is standard self-training, and because the final evaluation is on the human-labeled test set rather than on the pseudo-labels themselves, it does not make the reported results circular. The ChatGPT-augmented training labels (Section 3.5.2) are a validity concern about label noise, not a circularity: they are inputs to training, not outputs renamed as predictions. The only self-citation in the paper is to AfriSpeech-200 (Olatunji et al., 2023) in the literature review, and it is not load-bearing for any derivation. The Figure 2 caption's phrase 'generates pseudo labels for evaluation of performance' is imprecise relative to the methods text and the held-out test design, but there is no equation or definition that reduces the reported numbers to the model's own outputs. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no known result is merely renamed. I therefore find no significant circularity, only a minor non-load-bearing self-citation.
Assumptions & free parameters
free parameters (5)
- pseudo-label confidence threshold =
75th percentile of probability predictions per batch
- learning rate =
1e-5
- batch size =
16 to 64 depending on model size
- dropout rate =
0.1
- number of epochs =
10 supervised, 4 semi-supervised
assumptions (5)
- ad hoc to paper ChatGPT-generated labels (636 samples) are reliable enough to augment training data
- ad hoc to paper Pseudo-labels above the 75th percentile confidence threshold are accurate enough for training
- domain assumption Langdetect correctly identifies code-switched content and filtering to 'English (code-switched)' retains relevant content
- domain assumption The two volunteer annotators' labels with Cohen's Kappa 0.60/0.67 are reliable enough for training and evaluation
- domain assumption A test set of 2,000 tweets is representative of the ride-hailing code-switched Twitter stream
Cite this review
Pith. "Pith review of RideKE: Leveraging Low-Resource, User-Generated Twitter Content for Sentiment and Emotion Detection in Kenyan Code-Switched Dataset." pith.science (2026). https://pith.science/paper/N43HAGTN
@misc{pith2026250206180,
author = {Pith},
title = {Pith review of: RideKE: Leveraging Low-Resource, User-Generated Twitter Content for Sentiment and Emotion Detection in Kenyan Code-Switched Dataset},
year = {2026},
howpublished = {\url{https://pith.science/paper/N43HAGTN}},
note = {Machine review of arXiv:2502.06180}
}
read the original abstract
Social media has become a crucial open-access platform for individuals to express opinions and share experiences. However, leveraging low-resource language data from Twitter is challenging due to scarce, poor-quality content and the major variations in language use, such as slang and code-switching. Identifying tweets in these languages can be difficult as Twitter primarily supports high-resource languages. We analyze Kenyan code-switched data and evaluate four state-of-the-art (SOTA) transformer-based pretrained models for sentiment and emotion classification, using supervised and semi-supervised methods. We detail the methodology behind data collection and annotation, and the challenges encountered during the data curation phase. Our results show that XLM-R outperforms other models; for sentiment analysis, XLM-R supervised model achieves the highest accuracy (69.2\%) and F1 score (66.1\%), XLM-R semi-supervised (67.2\% accuracy, 64.1\% F1 score). In emotion analysis, DistilBERT supervised leads in accuracy (59.8\%) and F1 score (31\%), mBERT semi-supervised (accuracy (59\% and F1 score 26.5\%). AfriBERTa models show the lowest accuracy and F1 scores. All models tend to predict neutral sentiment, with Afri-BERT showing the highest bias and unique sensitivity to empathy emotion. https://github.com/NEtori21/Ride_hailing
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Terrible experience with Uber driver! He was rude and refused to follow the GPS directions #Angry
Anger: Label when the tweet expresses frustration, annoyance, resentment, or strong displeasure toward RHS, drivers, or related issues. Look for keywords and tone indicative of anger. Keywords: angry, furious, annoyed, upset. Example: "Terrible experience with Uber driver! He was rude and refused to follow the GPS directions #Angry"
-
[2]
Just had the best ride ever with the friendliest driver! #HappyCustomer #GreatService
Happy: Label when the tweet reflects joy, satisfaction, contentment, or delight regarding RHS experiences. Look for expressions of happiness, appreciation, or positive feedback. Keywords: happy, delighted, thrilled, satisfied. Example: “Just had the best ride ever with the friendliest driver! #HappyCustomer #GreatService”
-
[3]
My ride is taking an unfamiliar route, and I’m getting worried. Is this safe? #Fear
Fear: Label when the tweet expresses anxiety, worry, concern, or fear about RHS safety, incidents, or perceived risks. Identify cues of fear or apprehension. Keywords: afraid, scared, worried, nervous. Example: "My ride is taking an unfamiliar route, and I’m getting worried. Is this safe? #Fear"
-
[4]
Computers in human behavior, 31:527–541
Sentiment analysis in facebook and its appli- cation to e-learning. Computers in human behavior, 31:527–541. Billian Khalayi Otundo and Martine Grice. 2022. In- tonation in advice-giving in kenyan english and kiswahili. Proceedings of Speech Prosody 2022 , pages 150–154. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan
work page 2022
-
[5]
Wow, my driver gave me a free upgrade to a luxury car! #Surprised #Love
Love: Label when the tweet reflects affection, appreciation, or strong positive emotions toward RHS, drivers, or related aspects. Look for expressions of love or admiration. Keywords: love, adore, appreciate, grateful. Example: "Wow, my driver gave me a free upgrade to a luxury car! #Surprised #Love"
-
[6]
Virtuoso: Massive multilingual speech-text joint semi-supervised learning for text-to-speech. In ICASSP 2023-2023 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and li...
arXiv 2023
-
[7]
Use this label sparingly and only when other emotions are not evident
Neutral: Label when the tweet does not exhibit any strong emotional sentiment or when the emotion is unclear or ambiguous. Use this label sparingly and only when other emotions are not evident. Example: "Just booked my ride for tomorrow morning. #RideHail #PlanningAhead Quality Control Monitor inter-annotator agreement to ensure consistency among annotato...
-
[12]
Example: "Wow, my driver gave me a free upgrade to a luxury car! #Surprised
Suprise: Label when the tweet indicates astonishment, amazement, or unex- pected reactions to RHS experiences.Keywords: surprised, shocked, amazed, unexpected. Example: "Wow, my driver gave me a free upgrade to a luxury car! #Surprised
Show all 15 references
-
[14]
Been waiting for my ride for ages. This is so frustrating!#Frustrated #LateAgain
Frustration: Label when the tweet expresses dissatisfaction, irritation, or being fed up with RHS issues. Identify cues of frustration and annoyance. Keyword: frustrated, annoyed, fed up, irritated. Example: "Been waiting for my ride for ages. This is so frustrating!#Frustrate...
-
[136]
Maite Taboada, Julian Brooke, Milan Tofiloski, Kim- berly V oll, and Manfred Stede
Springer. Maite Taboada, Julian Brooke, Milan Tofiloski, Kim- berly V oll, and Manfred Stede. 2011. Lexicon-based methods for sentiment analysis. Computational lin- guistics, 37(2):267–307. 12 Michelle Terblanche, Kayode Olaleye, and Vukosi Mari- vate. 2024. Prompting towards ...
2011 arXiv
-
[2002]
arXiv preprint cs/0205070
Thumbs up? sentiment classification us- ing machine learning techniques. arXiv preprint cs/0205070. Bo Pang, Lillian Lee, et al. 2008. Opinion mining and sentiment analysis. Foundations and Trends® in information retrieval, 2(1–2):1–135. Viet H Pham, Thang M Pham, Giang Nguyen...
2008 arXiv
-
[2003]
In Proceedings of the 12th International Conference on World Wide Web, pages 519–528
Mining the peanut gallery: Opinion extraction and semantic classification of product reviews. In Proceedings of the 12th International Conference on World Wide Web, pages 519–528. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of dee...
2018 arXiv
-
[2014]
Ain Shams engineering journal , 5(4):1093–1113
Sentiment analysis algorithms and applica- tions: A survey. Ain Shams engineering journal , 5(4):1093–1113. Saif Mohammad. 2016. A practical guide to senti- ment annotation: Challenges and solutions. In Pro- ceedings of the 7th Workshop on Computational Ap- proaches to Subject...
2016 arXiv
-
[2022]
Expert Sys- tems with Applications, 209:118187
Low resource machine translation of English– Manipuri: A semi-supervised approach. Expert Sys- tems with Applications, 209:118187. Stephanie Strassel and Jennifer Tracey. 2016. Lorelei language packs: Data, tools, and resources for tech- nology development in low resource lang...
2016
-
[2023]
arXiv preprint arXiv:2310.14120
Sentiment analysis across multiple african languages: A current benchmark. arXiv preprint arXiv:2310.14120. ONAN Aytu˘g. 2018. Sentiment analysis on twitter based on ensemble of psychological and linguistic feature sets. Balkan Journal of Electrical and Com- puter Engineering,...
2018 arXiv
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.