Pith. sign in

REVIEW 5 major objections 5 minor 15 references

RideKE: Leveraging Low-Resource, User-Generated Twitter Content for Sentiment and Emotion Detection in Kenyan Code-Switched Dataset

T0 review · 5 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Pith's one-line read The paper introduces RideKE, a 29,000-tweet dataset of Kenyan code-switched ride-hailing content, and shows that XLM-R outperforms three other transformer models on sentiment classification, while emotion detection remains weak.

desk verdict The dataset is a real contribution, but inconsistent numbers and unvalidated ChatGPT labels make the benchmark claims unreliable as written. read the letter →

arxiv 2502.06180 v1 pith:N43HAGTN submitted 2025-02-10 cs.CL cs.AI

classification cs.CLcs.AI
keywords code-switchingsentimentanalysisemotiondetectionlow-resourcelanguagesKenyanTwitterShengtransformermodelssemi-supervisedlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RideKE is introduced as the first sentiment-and-emotion dataset built from Kenyan ride-hailing tweets in code-switched English, Swahili, and Sheng, totaling more than 29,000 examples with a small human-annotated core. The paper's central claim is that this low-resource dataset is usable: fine-tuned transformer models can classify sentiment with 69.2% accuracy (XLM-R supervised) and emotion with up to 59.8% accuracy (DistilBERT supervised), establishing a benchmark for Kenyan code-switched text. A secondary finding is that emotion detection is substantially harder than sentiment, with the best macro F1 near 31% and several emotion classes effectively unlearned. The authors argue that the dataset and the model comparisons give the NLP community a public resource for low-resource, code-switched, domain-specific evaluation.

What carries the argument

The carrying artifact is RideKE, a dataset of 29,623 tweets collected from Kenyan ride-hailing hashtags, filtered by user location and language, and preprocessed into about 70% Kenyan-accented English with the rest in Swahili and Sheng. Its structure matters: a small labeled training set (553 human labels plus 636 automatic labels), a 2,000-tweet human-annotated test set, and a roughly 27,000-tweet unlabeled pool. The other load-bearing mechanism is the semi-supervised loop, in which each pretrained model labels unlabeled samples whose predicted probability exceeds the 75th percentile for a class and then retrains on those pseudo-labels for four epochs. Four transformers — DistilBERT, mBERT, XLM-R, and AfriBERTa — are fine-tuned with a combined categorical cross-entropy loss over sentiment and emotion heads, and evaluated on the human test set.

What would settle it

Re-annotate the 636 automatically labeled tweets and a random sample of high-confidence pseudo-labeled tweets with two fresh Kenyan annotators; if their agreement with the auto-labels is near chance, or if retraining on purely human labels changes which model leads, the paper's central benchmark claim would be contradicted.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a modest, partially annotated corpus of noisy social-media text in a low-resource code-switched setting can support meaningful model evaluation: XLM-R reaches 69.2% accuracy and 66.1% F1 on sentiment in the fully supervised condition, outperforming mBERT, DistilBERT, and AfriBERTa on this data. Emotion classification is the harder task, with DistilBERT leading at 59.8% accuracy but only 31% macro F1, and all models showing a systematic pull toward neutral sentiment—AfriBERTa most strongly. The paper also finds that its semi-supervised loop, which uses high-confidence pseudo-labels, yields mixed gains: it helps mBERT on emotion (29.7% vs. 26.5% F1) but does not surpass the best supervised sentiment run, so the authors present semi-supervision as a complement rather than a replacement for human annotation. Taken together, the claim is that RideKE gives the low-resource NLP community a new benchmark and an initial model ranking for Kenyan ride-hailing discourse.

Load-bearing premise

The reported results assume that the automatic labels, both the 636 LLM-generated training labels and the high-confidence pseudo-labels from the models themselves, are accurate enough to stand in for human annotation; if those labels are noisy, the model ranking and accuracy figures could reflect label bias rather than true ability.

Editorial extensions

If this is right

  • RideKE becomes a public benchmark for Kenyan code-switched sentiment and emotion classification, giving low-resource NLP a shared evaluation set tied to ride-hailing discourse.
  • XLM-R is a strong default for sentiment analysis on this data, reaching the highest accuracy and F1 among the tested models in the supervised setting.
  • Semi-supervised training with a 75th-percentile confidence threshold helps some models on some tasks, such as mBERT on emotion, but does not consistently beat supervised training.
  • Emotion detection on RideKE remains an open problem: the best macro F1 is about 31%, and happy and fear classes are essentially unlearned by most models.
  • All four models over-predict neutral sentiment, and AfriBERTa shows the strongest neutral bias, so model choice affects the label distribution, not just accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: a ride-hailing company could deploy a RideKE-tuned sentiment monitor to track complaint spikes in Nairobi, but the documented neutral bias would need calibration before use.
  • Beyond the paper: the near-zero F1 for happy and fear suggests those emotion classes are too rare or too ambiguous in the data; collecting targeted tweets for these classes would test whether the low scores reflect data scarcity or label ambiguity.
  • Beyond the paper: AfriBERTa's weak performance despite African-language pretraining hints that domain and register, not language-family match, drive results; experimenting with pretraining on Sheng-heavy social-media text would test this.
  • Beyond the paper: the 75th-percentile pseudo-label threshold is an arbitrary knob; sweeping this threshold could reveal whether the mixed semi-supervised results are robust or threshold-dependent.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces RideKE, a corpus of 29,623 tweets about Kenyan ride-hailing services (Uber, Bolt, Little Cab), containing a mix of English, Swahili, and Sheng. A subset of 2,553 tweets is human-annotated for sentiment (positive/negative/neutral) and eight emotion categories, and a further 636 tweets are labeled by ChatGPT; 27,090 tweets are used as unlabeled data. The authors fine-tune DistilBERT, mBERT, XLM-R, and AfriBERTa under supervised and semi-supervised settings, and report that XLM-R performs best for sentiment (69.2% accuracy, 66.1% F1) while DistilBERT and mBERT lead on emotion. The contributions claimed are the new dataset, the comparison of four transformer models, and analysis of code-switched Kenyan data.

Significance. If the results are reproducible, RideKE would be a useful resource for low-resource African NLP, since there are few public code-switched sentiment and emotion datasets focused on Kenyan ride-hailing. The human annotation effort is a strength, and the paper makes a concrete attempt to address data scarcity through ChatGPT augmentation and semi-supervised learning. However, the significance is currently limited by unresolved numeric inconsistencies and by the absence of validation for the automatically generated labels, so the central model-ranking claim is not yet established.

major comments (5)
  1. [Abstract, §5.1, Table 5] The reported headline numbers are internally inconsistent. The abstract and Table 5 give XLM-R supervised sentiment accuracy 69.2% and F1 66.1%, but Section 5.1 says the same condition achieves 62.5% accuracy and 66.7% F1, and gives XLM-R semi-supervised as 62.1% accuracy and 68.3% F1, whereas Table 5 reports 67.2% and 64.1% for that system. The abstract also assigns the 26.5% emotion F1 to mBERT semi-supervised, while Table 5 and Section 5.2 report 29.7% for that condition (26.5% is the supervised value). At a minimum, these figures must be reconciled; ideally the authors should report the exact evaluation configuration (e.g., checkpoint selection, number of runs) that produced each table.
  2. [§3.5.2, §4.2.1] Section 3.5.2 and Table 4 state that Set A, the supervised training set, consists of 553 human labels and 636 ChatGPT-generated labels; Section 4.2.1 says fine-tuning used 1,189 labeled tweets. Thus 54% of the supervised training signal is machine-generated. The paper reports no validation of these ChatGPT labels, no human agreement on them, and no error analysis. Because the central claim is a model ranking, a ranking obtained from this training mixture may reflect sensitivity to ChatGPT label noise rather than true ability on Kenyan code-switched text. The authors should report ChatGPT-label accuracy against human labels on a held-out subset, and should show that the ranking in Table 5 is stable when training on human labels alone.
  3. [§4.2.2] Section 4.2.2 describes a semi-supervised loop in which each model labels the unlabeled set with its own predictions and retains only samples above the 75th-percentile confidence threshold. This is a self-referential training signal: the model's confidence is used as a proxy for label correctness, with no external check. The resulting pseudo-labels can reinforce systematic errors, especially for rare emotion classes where the model tends to be overconfident. The paper should report the accuracy of the pseudo-labels on a human-annotated subset, or compare the semi-supervised models against a supervised-only baseline under identical hyperparameters, before interpreting the SSL results in Table 5.
  4. [§5.2, Tables 7–9] Section 5.2 and Tables 7–9 show that several emotion classes have zero F1 for every model (Happy in Table 7, Fear in Table 8, and Surprise for most models in Table 9). The aggregate emotion F1 values in Table 5 therefore average over classes the models never predict. This makes the emotion ranking fragile: a model that assigns every tweet to a single high-frequency class can appear competitive in accuracy while having zero performance on most of the label set. The authors should report the class distribution in Set B, per-class support, and either use a metric robust to extreme imbalance or restrict the emotion benchmark to classes with enough positive examples.
  5. [Table 5, Appendix A.5] Table 5 states that results are averages over multiple runs, but no variance, confidence intervals, or significance tests are reported. The differences that drive the ranking are small (e.g., mBERT supervised vs. semi-supervised sentiment F1: 59.6% vs. 59.8%), and without run-level statistics these differences are not interpretable. In addition, Appendix A.5 shows two identical tweet rows from the same date, and the text only says that duplicates were removed from Set C, not from Sets A and B; if duplicated tweets cross the train/test split, the evaluation is optimistic. The authors should report standard deviations across seeds, run a paired test (e.g., McNemar) for the headline comparisons, and clarify or implement duplicate removal for all splits.
minor comments (5)
  1. [Title/Abstract] The title on the first page reads 'Kenya`n Code-Switched Dataset'; this should be fixed to 'Kenyan'.
  2. [§3.3, Table 10] Table 10 reports large counts of Indonesian, Norwegian, Danish, and other languages; the paper should explain how these were handled, since the dataset is described as English–Swahili–Sheng code-switched.
  3. [§4.2.1, Figure 3] Section 4.2.1 references Figure 3 as showing the 'combined categorical cross-entropy loss function'; Figure 3 actually shows training loss curves, not the loss definition, so the reference should be clarified.
  4. [§5.3, Abstract] The abstract and Section 5.3 refer to 'Afri-BERT' while the model is called AfriBERTa elsewhere; a single consistent name should be used.
  5. [§5.4] Section 5.4 says semi-supervised mBERT improved sentiment F1 to 59.8% from 59.6%; this difference is well within what could be expected from random seed variation, so it should not be described as a 'clear performance enhancement' without variance estimates.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central model comparison is an independent empirical evaluation on a held-out human-labeled test set; the self-training loop and minor self-citation are not load-bearing.

full rationale

The paper's central claim is a new benchmark dataset plus a ranked comparison of four pretrained transformers. The test set (Set B) is described as 'manually annotated by human annotators' (Section 3.5.2), and the reported accuracy/F1 values in Table 5 are evaluated after training on that held-out test set, so the ranking is not forced by construction. The semi-supervised phase (Section 4.2.2) uses each model's own high-confidence predictions as pseudo-labels for additional training; this is standard self-training, and because the final evaluation is on the human-labeled test set rather than on the pseudo-labels themselves, it does not make the reported results circular. The ChatGPT-augmented training labels (Section 3.5.2) are a validity concern about label noise, not a circularity: they are inputs to training, not outputs renamed as predictions. The only self-citation in the paper is to AfriSpeech-200 (Olatunji et al., 2023) in the literature review, and it is not load-bearing for any derivation. The Figure 2 caption's phrase 'generates pseudo labels for evaluation of performance' is imprecise relative to the methods text and the held-out test design, but there is no equation or definition that reduces the reported numbers to the model's own outputs. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no known result is merely renamed. I therefore find no significant circularity, only a minor non-load-bearing self-citation.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The empirical claims rest on unvalidated label sources (ChatGPT, pseudo-labels) and standard hyperparameters; no new entities are proposed.

free parameters (5)
  • pseudo-label confidence threshold = 75th percentile of probability predictions per batch
    Dynamic threshold to filter pseudo-labels; chosen empirically without ablation, affects all semi-supervised results.
  • learning rate = 1e-5
    Fixed across models and phases; not tuned per model despite differing sizes.
  • batch size = 16 to 64 depending on model size
    Chosen for computational efficiency; no ablation.
  • dropout rate = 0.1
    Fixed regularizer; no sensitivity analysis.
  • number of epochs = 10 supervised, 4 semi-supervised
    Empirically selected for the semi-supervised loop based on loss plateau; no reported criterion.
assumptions (5)
  • ad hoc to paper ChatGPT-generated labels (636 samples) are reliable enough to augment training data
    Section 3.5.2: 'We used ChatGPT for automatic labeling to augment the training dataset' with no validation of these labels against human labels.
  • ad hoc to paper Pseudo-labels above the 75th percentile confidence threshold are accurate enough for training
    Section 4.2.2: dynamic threshold is set without measuring pseudo-label accuracy or effect on downstream performance.
  • domain assumption Langdetect correctly identifies code-switched content and filtering to 'English (code-switched)' retains relevant content
    Section 3.3: langdetect detects English, Indonesian, and Swahili but Sheng is manually detected; the filtering step is not validated.
  • domain assumption The two volunteer annotators' labels with Cohen's Kappa 0.60/0.67 are reliable enough for training and evaluation
    Section 3.5.1: only two annotators; agreement is moderate, not high; no adjudication details beyond majority vote.
  • domain assumption A test set of 2,000 tweets is representative of the ride-hailing code-switched Twitter stream
    Section 3.5.2: Set B split, but no description of sampling strategy to ensure representativeness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RideKE: Leveraging Low-Resource, User-Generated Twitter Content for Sentiment and Emotion Detection in Kenyan Code-Switched Dataset." pith.science (2026). https://pith.science/paper/N43HAGTN

@misc{pith2026250206180,
  author       = {Pith},
  title        = {Pith review of: RideKE: Leveraging Low-Resource, User-Generated Twitter Content for Sentiment and Emotion Detection in Kenyan Code-Switched Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N43HAGTN}},
  note         = {Machine review of arXiv:2502.06180}
}
read the original abstract

Social media has become a crucial open-access platform for individuals to express opinions and share experiences. However, leveraging low-resource language data from Twitter is challenging due to scarce, poor-quality content and the major variations in language use, such as slang and code-switching. Identifying tweets in these languages can be difficult as Twitter primarily supports high-resource languages. We analyze Kenyan code-switched data and evaluate four state-of-the-art (SOTA) transformer-based pretrained models for sentiment and emotion classification, using supervised and semi-supervised methods. We detail the methodology behind data collection and annotation, and the challenges encountered during the data curation phase. Our results show that XLM-R outperforms other models; for sentiment analysis, XLM-R supervised model achieves the highest accuracy (69.2\%) and F1 score (66.1\%), XLM-R semi-supervised (67.2\% accuracy, 64.1\% F1 score). In emotion analysis, DistilBERT supervised leads in accuracy (59.8\%) and F1 score (31\%), mBERT semi-supervised (accuracy (59\% and F1 score 26.5\%). AfriBERTa models show the lowest accuracy and F1 scores. All models tend to predict neutral sentiment, with Afri-BERT showing the highest bias and unique sensitivity to empathy emotion. https://github.com/NEtori21/Ride_hailing

Figures

Figures reproduced from arXiv: 2502.06180 by the authors.

Figure 1
Figure 1. Geographical representation of RideKE: diverse local accents collected in tweets, such as Rift Valley (e.g., Eldoret, Nakuru), Central (e.g., Nyeri, Ki￾ambu), Nairobi (e.g., Kasarani, Kileleshwa), Western (e.g., Kakamega, Bungoma), Nyanza (e.g., Kisumu, Kisii), Eastern (e.g., Machakos, Embu) Coast (e.g., Mombasa, Malindi), and North-Eastern (e.g., Garissa, Mandera). landscape is diverse and dynamic (Dwivedi, 2014; C… view at source ↗
Figure 2
Figure 2. Methodology: Overview of the RideKE senti￾ment and emotion analysis framework. Unlabeled and labeled datasets are preprocessed and used to train su￾pervised and semi-supervised models for sentiment and emotion prediction. The semi-supervised learning loop generates pseudo labels for evaluation of performance. 3 Methods and Datasets 3.1 Overview of RideKE Dataset RideKE dataset. as shown in [PITH_FULL_IMAGE:figures/… view at source ↗
Figure 3
Figure 3. Training loss (a) supervised and (b) semi￾supervised learning. 7 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Heatmaps comparing sentiment and emotion predictions across different models. AfriBERT model most frequently predicts neutral sentiment and shows the highest sensitivity for empathy emotions. 5.3 Pretrained Models performance As shown in [PITH_FULL_IMAGE:figures/full_…
Figure 5
Figure 5. Figure 5: Number of tweets per location on a logarith [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 11 canonical work pages

  1. [1]

    Terrible experience with Uber driver! He was rude and refused to follow the GPS directions #Angry

    Anger: Label when the tweet expresses frustration, annoyance, resentment, or strong displeasure toward RHS, drivers, or related issues. Look for keywords and tone indicative of anger. Keywords: angry, furious, annoyed, upset. Example: "Terrible experience with Uber driver! He was rude and refused to follow the GPS directions #Angry"

  2. [2]

    Just had the best ride ever with the friendliest driver! #HappyCustomer #GreatService

    Happy: Label when the tweet reflects joy, satisfaction, contentment, or delight regarding RHS experiences. Look for expressions of happiness, appreciation, or positive feedback. Keywords: happy, delighted, thrilled, satisfied. Example: “Just had the best ride ever with the friendliest driver! #HappyCustomer #GreatService”

  3. [3]

    My ride is taking an unfamiliar route, and I’m getting worried. Is this safe? #Fear

    Fear: Label when the tweet expresses anxiety, worry, concern, or fear about RHS safety, incidents, or perceived risks. Identify cues of fear or apprehension. Keywords: afraid, scared, worried, nervous. Example: "My ride is taking an unfamiliar route, and I’m getting worried. Is this safe? #Fear"

  4. [4]

    Computers in human behavior, 31:527–541

    Sentiment analysis in facebook and its appli- cation to e-learning. Computers in human behavior, 31:527–541. Billian Khalayi Otundo and Martine Grice. 2022. In- tonation in advice-giving in kenyan english and kiswahili. Proceedings of Speech Prosody 2022 , pages 150–154. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan

  5. [5]

    Wow, my driver gave me a free upgrade to a luxury car! #Surprised #Love

    Love: Label when the tweet reflects affection, appreciation, or strong positive emotions toward RHS, drivers, or related aspects. Look for expressions of love or admiration. Keywords: love, adore, appreciate, grateful. Example: "Wow, my driver gave me a free upgrade to a luxury car! #Surprised #Love"

  6. [6]

    In ICASSP 2023-2023 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5

    Virtuoso: Massive multilingual speech-text joint semi-supervised learning for text-to-speech. In ICASSP 2023-2023 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and li...

  7. [7]

    Use this label sparingly and only when other emotions are not evident

    Neutral: Label when the tweet does not exhibit any strong emotional sentiment or when the emotion is unclear or ambiguous. Use this label sparingly and only when other emotions are not evident. Example: "Just booked my ride for tomorrow morning. #RideHail #PlanningAhead Quality Control Monitor inter-annotator agreement to ensure consistency among annotato...

  8. [12]

    Example: "Wow, my driver gave me a free upgrade to a luxury car! #Surprised

    Suprise: Label when the tweet indicates astonishment, amazement, or unex- pected reactions to RHS experiences.Keywords: surprised, shocked, amazed, unexpected. Example: "Wow, my driver gave me a free upgrade to a luxury car! #Surprised

Show all 15 references
  1. [14]

    Been waiting for my ride for ages. This is so frustrating!#Frustrated #LateAgain

    Frustration: Label when the tweet expresses dissatisfaction, irritation, or being fed up with RHS issues. Identify cues of frustration and annoyance. Keyword: frustrated, annoyed, fed up, irritated. Example: "Been waiting for my ride for ages. This is so frustrating!#Frustrate...

  2. [136]

    Maite Taboada, Julian Brooke, Milan Tofiloski, Kim- berly V oll, and Manfred Stede

    Springer. Maite Taboada, Julian Brooke, Milan Tofiloski, Kim- berly V oll, and Manfred Stede. 2011. Lexicon-based methods for sentiment analysis. Computational lin- guistics, 37(2):267–307. 12 Michelle Terblanche, Kayode Olaleye, and Vukosi Mari- vate. 2024. Prompting towards ...

  3. [2002]

    arXiv preprint cs/0205070

    Thumbs up? sentiment classification us- ing machine learning techniques. arXiv preprint cs/0205070. Bo Pang, Lillian Lee, et al. 2008. Opinion mining and sentiment analysis. Foundations and Trends® in information retrieval, 2(1–2):1–135. Viet H Pham, Thang M Pham, Giang Nguyen...

  4. [2003]

    In Proceedings of the 12th International Conference on World Wide Web, pages 519–528

    Mining the peanut gallery: Opinion extraction and semantic classification of product reviews. In Proceedings of the 12th International Conference on World Wide Web, pages 519–528. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of dee...

  5. [2014]

    Ain Shams engineering journal , 5(4):1093–1113

    Sentiment analysis algorithms and applica- tions: A survey. Ain Shams engineering journal , 5(4):1093–1113. Saif Mohammad. 2016. A practical guide to senti- ment annotation: Challenges and solutions. In Pro- ceedings of the 7th Workshop on Computational Ap- proaches to Subject...

  6. [2022]

    Expert Sys- tems with Applications, 209:118187

    Low resource machine translation of English– Manipuri: A semi-supervised approach. Expert Sys- tems with Applications, 209:118187. Stephanie Strassel and Jennifer Tracey. 2016. Lorelei language packs: Data, tools, and resources for tech- nology development in low resource lang...

  7. [2023]

    arXiv preprint arXiv:2310.14120

    Sentiment analysis across multiple african languages: A current benchmark. arXiv preprint arXiv:2310.14120. ONAN Aytu˘g. 2018. Sentiment analysis on twitter based on ensemble of psychological and linguistic feature sets. Balkan Journal of Electrical and Com- puter Engineering,...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.