{"id":"fb6c928f-9fcb-4c2f-90e2-f6b88c8da411","arxiv_id":"2512.01557","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Language detection models are near-perfect on clean news text in Yoruba, Kinyarwanda, and Amharic but perform poorly on code-switched Reddit posts, suggesting curated news data is more reliable for training.","lead":"This paper measures how often Yoruba, Kinyarwanda, and Amharic appear on Reddit and news sites, and tests AI language-detection tools on both. News text is clean and easy for AI; Reddit posts are mostly English or mixed, so detection fails there.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reddit code-switching claim rests on unvalidated detector outputs; models' different label sets make the 'struggle' partly an artifact.","rationale":"The reader's weakest assumption—that model outputs on Reddit are treated as ground truth—is exactly the most load-bearing concern. I considered two other potential attacks: (1) the paper's recommendation that news data be used for 'training context-rich AI models' is not directly tested by any downstream training or generation experiment; and (2) the sample sizes are very small (62–105 news items, 165–833 Reddit posts). Both are genuine limitations, but they do not strike at the empirical core as sharply as the missing gold labels. If the code-switching percentages are wrong, the 'stark contrast' between news and Reddit disappears, and the entire recommendation loses its factual basis. The disjoint label-set issue is a specific mechanism that makes the missing labels especially damaging: AfroLID and Llama are not measuring the same thing, so their disagreement is not evidence of code-switching. Because this concern is substantive but fixable with human annotation, the appropriate disposition remains conditional rather than accept or reject. I therefore recommend no change to the reader's verdict.","tokens_in":7205,"tokens_out":4913,"duration_ms":51868,"concrete_test":"Sample 100 posts per subreddit (300 total) from the scraped Reddit data; have two native speakers per language independently annotate each post as monolingual target language, mixed target/English, English, or other, with adjudication for disagreements. Compare AfroLID and Llama 3.3 70B predictions to these gold labels using macro-F1 and confusion matrices, and report human inter-annotator agreement. If model predictions match human labels (e.g., macro-F1 ≥ 0.9, Cohen's κ ≥ 0.8), the concern is resolved; otherwise the paper's central comparison is unsupported. As a secondary check, annotate 50 posts per news source to verify whether the reported 100% news accuracy is an artifact of source-language priors.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central comparative claim—that detection models are near-perfect on clean news data but 'struggle' with code-switched Reddit—is not supported by the measurements as reported. In §4.2, AfroLID and Llama 3.3 70B are used as detectors, and Tables 1–3 report their predicted label distributions as if these were ground-truth language compositions. No human-annotated gold labels are collected for Reddit posts. This is not a missing extra: the two detectors have disjoint label vocabularies. AfroLID has no English class (stated in §4.2), while Llama's dominant output on Reddit is English (73–97%, Table 3). Thus the observed divergence is partly a byproduct of tool design, not evidence about the texts. 'Struggled' is also never defined: no precision, recall, or F1 is reported; only predicted distributions are given. On the news side, '100% accuracy' is close to tautological because each source is known to write in a specific language (BBC Yoruba, RBA, Fana); applying a detector to a known-monolingual source and finding the expected label does not validate the detector or the 'robust source' claim. The code-switching phenomenon—the paper's key empirical premise—therefore remains unverified. The conclusion in §5.4 that conversational platforms are unreliable because of code-switching depends entirely on this unvalidated measurement. Notably, the authors had access to native speakers (§4.3) for topic annotation but did not use them to annotate the language content that is central to the paper's claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper examines the digital presence of Yoruba, Kinyarwanda, and Amharic by comparing Reddit posts with local news articles. It scrapes three subreddits (r/yoruba, r/amharic, r/Rwanda) and three news sources (BBC Yoruba, RBA, Fana), applies the AfroLID and Llama 3.3 70B language detectors, and manually annotates news article topics. The authors report that Reddit data are sparse and heavily code-switched/English-dominated, while news data are clean and monolingual; the detectors are near-perfect on news but 'struggle' on Reddit. The paper concludes that professionally curated news content is a more reliable source for training context-rich AI models for African languages than conversational platforms.","tokens_in":7548,"tokens_out":5142,"duration_ms":55705,"significance":"If supported, the paper would provide a useful empirical data point for low-resource African NLP: it quantifies the gap between conversational and professionally curated sources and documents the difficulty of language detection on mixed text. The manual topic annotation by native speakers is a genuine strength, as is the comparison of a specialized African-language detector with a general-purpose LLM. The claim itself is plausible and policy-relevant for dataset construction. However, the central empirical contrast is currently under-supported: the Reddit 'code-switching' claim is inferred from unvalidated detector outputs with incompatible label sets, and the news 'accuracy' is close to tautological because sources were selected precisely because they are known to publish in a target language. The paper contains no code, data, or uncertainty quantification, which limits its current value.","major_comments":[{"comment":"The central claim that detectors 'struggled' with Reddit code-switching is not directly supported by the measurements. No human gold labels were collected for Reddit posts; Tables 1–3 report the models' predicted label distributions, not accuracy. Because AfroLID has no English class while Llama outputs English for 69–97% of Reddit posts, the observed divergence is partly an artifact of incompatible label vocabularies. 'Struggled' is never defined—no precision, recall, or F1 is reported. This is load-bearing for the §5.4 conclusion about conversational platforms. I recommend human annotation of a stratified Reddit sample (native speakers were already used for topic classification), followed by per-language precision/recall/F1 and an explicit definition of code-switching.","section":"§4.2, Tables 1–3"},{"comment":"The Kinyarwanda data come from r/Rwanda, which is a country subreddit, not a Kinyarwanda-language community. English dominance there is unsurprising and conflates national affiliation with language use. The §5.4 statement that platforms 'clearly dedicated to specific African linguistic communities' are unreliable overgeneralizes to the Rwandan case. The paper should either find a Kinyarwanda-focused subreddit or explicitly frame the result as 'r/Rwanda' rather than as evidence about Kinyarwanda usage.","section":"§4.1.1, §5.1"},{"comment":"'100% accuracy' on news is close to tautological because each news source was selected for publishing in a known language (BBC Yoruba, RBA, Fana). A detector agreeing with source metadata does not validate the detector's general ability or establish that news is a 'robust' source beyond these three outlets. Report at least a human-verified sample of news posts, or evaluate on additional/held-out sources; otherwise the claim should be softened to 'detector predictions match source-language metadata on this sample.'","section":"§5.2.1, Tables 2 and 4"},{"comment":"The quantitative comparison lacks uncertainty quantification. Sample sizes are modest (e.g., 62–105 news articles; 165–833 Reddit posts), yet percentages are reported to two decimals without counts or confidence intervals. For example, Kinyarwanda news '97.14%' is 2 errors out of 105, which is not meaningfully different from 100% at this sample size. Report raw counts, Wilson intervals or bootstrap confidence intervals, and a significance test for the Reddit-vs-news contrast.","section":"§5.1–5.3"}],"minor_comments":[{"comment":"The abstract mentions a macro-classifier GlotLID, but GlotLID is never used or results reported; only AfroLID and Llama 3.3 70B appear in the methodology. Either include GlotLID results or remove the mention.","section":"Abstract, §4.2"},{"comment":"No data availability statement, code, or scraping dates are provided; the Llama prompt and inference configuration are also omitted. This limits reproducibility for an empirical paper.","section":"§4.1"},{"comment":"The 'qualitative analysis' of Reddit posts is mentioned but no coding protocol, inter-annotator agreement, or example excerpts are given. Specify the procedure or remove the qualitative claim.","section":"§5.1"},{"comment":"Reference [15] cites a GeeksforGeeks page for PRAW instead of the PRAW documentation/API; please cite the actual software. Also, the PRAW references [12]–[14] are listed as subreddit pages, which is acceptable but inconsistent with the style of [9]–[11]; please standardize.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is an exploratory empirical paper with a plausible and useful central observation, but the current evidence for the key code-switching claim is not robust. The issues are fixable: human annotation of a Reddit sample, uncertainty quantification, and a clearer separation of source-selection effects from detector performance. I would encourage a resubmission addressing these points rather than rejecting the idea. The novelty is moderate, but the topic is timely for low-resource NLP."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper offers a plausible, practically useful recommendation—curated news text is cleaner than Reddit for Yoruba, Kinyarwanda, and Amharic—but the quantitative evidence for the ‘code-switching makes detectors struggle’ half of the claim is weaker than the prose suggests. The paper should not be published as is without human-annotated labels for the Reddit sample.\n\nWhat’s new: the specific comparison of news vs. Reddit for these three languages with AfroLID and Llama 3.3 70B is new. The finding that news sources yield near-monolingual data and that subreddits are sparse and English-heavy is consistent with prior work but has not been measured this way for these languages. The news topic annotation is a plus: four native speakers labeled article titles, which grounds the qualitative observations about editorial priorities.\n\nThe paper is clearly written, the methodology is transparent, and the recommendation to treat news media as a training-data source and to build code-switch-aware LID is sensible. The ethical section is fine.\n\nThe soft spots are real and they sit at the center of the argument. The claim that detectors are near-perfect on news but ‘struggle’ on Reddit depends entirely on the detectors’ own outputs. No human labels were collected for Reddit text, even though the authors had native speakers on hand for topic classification. That is a missed opportunity, and it matters because AfroLID and Llama have different label vocabularies: AfroLID has no English class, so its high ‘other African language’ percentages on Reddit are partly a byproduct of tool design, and Llama’s 73–97% English detections are not validated against anything. ‘Struggled’ is never defined—no precision, recall, or F1 is reported. On the news side, 100% accuracy is close to tautological because each source is known to write in its language; it confirms the source is clean, but it does not validate the detectors. The sample sizes are small (62–105 news posts, 165–833 Reddit posts), there are no error bars, and no data or code are released. These are not fatal flaws for an exploratory study, but they should be fixed before this is treated as a solid result.\n\nWho should read it: people working on data collection for African NLP, especially those deciding between scraping social media and using news sources. The recommendation is reasonable and the paper gives some numbers, even if the numbers come with caveats.\n\nMy recommendation: send it to peer review, but with a clear request for human-annotated language labels on a sample of Reddit posts, release of the data, and a defined evaluation metric. The topic is important and the paper is honest; it just needs stronger evidence for its central claim.","headline":"Small, clearly-written study with a plausible practical recommendation, but the central quantitative claim about code-switching rests on unvalidated detector outputs and needs human labels before it can support the conclusion.","tokens_in":8028,"tokens_out":2087,"would_cite":false,"duration_ms":22262,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that professionally curated local news media provide cleaner, more reliable training data for Yoruba, Kinyarwanda, and Amharic than conversational platforms, and that language detection models fail when text is code-switche","keywords":["African languages","language identification","code-switching","Yoruba","Kinyarwanda","Amharic","low-resource NLP","digital language representation"],"falsifier":"Hand-annotate a sample of 100 posts from each of the three subreddits, sentence by sentence, with native speakers. If most sentences are judged to be Yoruba, Kinyarwanda, or Amharic with only embedded English words or phrases, the claim of English-heavy code-switching is falsified; if speakers agree with the models' English-majority judgment, the claim holds.","tokens_in":7140,"feed_emoji":"📰","tokens_out":4483,"duration_ms":43038,"temperature":0.7,"pith_summary":"The paper tries to establish that for Yoruba, Kinyarwanda, and Amharic, the best available source of clean monolingual digital text is professionally edited local news, not social-media conversation. On a year of forum posts, the African languages appeared infrequently and mixed with English and other languages, while news sites yielded text that was almost entirely monolingual and prompted local-language engagement in comment sections. Language detection tools—both a dedicated African-language identifier and a large general-purpose model—performed near-perfectly on the news data but split widely and often flagged English on the forum data. The authors conclude that future African-language AI should be trained on curated news data for clean text and needs new models that treat code-switching as a valid, learnable phenomenon.","feed_headline":"Local news, not Reddit, supplies clean data for African-language AI","feed_subtitle":"Study of Yoruba, Kinyarwanda, and Amharic finds detectors ace news text but stumble on code-switched conversation.","key_machinery":"The method pairs two language-detection tools—AfroLID, a neural identifier covering 517 African languages, and Llama 3.3 70B, a general-purpose multilingual LLM—with two contrastive corpora: one year of posts from Yoruba, Amharic, and Kinyarwanda subreddits, and headlines (plus Facebook comments) from local news outlets. The load-bearing contrast is the gap between the tools' near-perfect scores on the clean news text and their unstable, divergent scores on the conversational text. That divergence is used as evidence that conversational platforms are unreliable sources for monolingual training data and that code-switching is the real obstacle.","core_discovery":"The central discovery is a sharp asymmetry: the same detection tools that identify almost 100% of news headlines in Yoruba, Kinyarwanda, and Amharic find those languages only intermittently in the corresponding subreddits. A year of scraping produced 184, 165, and 833 posts respectively, with AfroLID assigning large shares to other African languages and Llama 3.3 70B labeling 69–97% of the posts as English. The authors' interpretation is that dedicated language communities on these conversational platforms are not writing monolingually; they are code-switching, and this is precisely the register that current detectors mishandle. In contrast, news media and their social comments show that whe","pith_inferences":["If the pattern holds across other African languages, many existing web-crawled corpora are skewed toward formal registers, which would make models seem better than they are on spoken conversation.","A direct test of the paper's conclusion would be to fine-tune or prompt-tune a detector on the news data and evaluate on a human-annotated code-switched sample; the paper did not include such an evaluation, so the improvement remains a prediction.","The paper's own data suggests a practical shortcut for future collections: news organizations' Facebook comments appear to contain more spontaneous local-language conversation than dedicated subreddits, so these comment threads may be the closest available public proxy for conversational data."],"forward_implications":["Clean, monolingual African-language corpora for training and evaluation can be assembled from local news outlets much more cheaply and reliably than by scraping conversational platforms.","Benchmarks built on news text alone will overstate real-world performance: detectors that score 100% on clean text can fall apart on everyday code-switched usage.","Social-media comment sections on local-language news posts may serve as a source of authentic conversational data, since local-language posts elicited local-language responses.","Language technology for these communities should treat code-switching as a first-class modelling target rather than as noise to be filtered out.","The negative feedback loop the paper describes—scarce data, poor models, users avoiding their language—can be interrupted at the data-collection stage by using curated news media."],"fun_headline_variants":["African language AI fails on code-switched Reddit posts","News data, not Reddit, best for African language models","Detectors ace clean news, stumble on Reddit code-switching","Yoruba, Kinyarwanda, Amharic: Reddit data misleads AI","Clean news beats chat for training African-language AI"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper's picture of what is happening on the forums—how much English, how much code-switching—rests entirely on the output of AfroLID and Llama 3.3 70B; without human annotation of those posts, the measured 'struggle' could be the models' error rather than the data's code-switching.","fun_headline_variants_meta":{"raw":{"variants":["African language AI fails on code-switched Reddit posts","News data, not Reddit, best for African language models","Detectors ace clean news, stumble on Reddit code-switching","Yoruba, Kinyarwanda, Amharic: Reddit data misleads AI","Clean news beats chat for training African-language AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1207,"prompt_tokens":813,"completion_tokens":394,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":302}},"tokens_in":557,"tokens_out":394,"duration_ms":4967,"temperature":1.0,"reasoning_tokens":302,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T19:08:28.480593+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hand-annotate a sample of 100 posts from each of the three subreddits, sentence by sentence, with native speakers. If most sentences are judged to be Yoruba, Kinyarwanda, or Amharic with only embedded English words or phrases, the claim of English-heavy code-switching is falsified; if speakers agree with the models' English-majority judgment, the claim holds.","supporting_citations":[],"review_version":1}