{"id":"340e0d87-55db-43c8-9959-d525431eaac8","arxiv_id":"2509.07998","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"On a new 144,000-word annotated dataset for Wolayta and Gofa, BERT-base-uncased embeddings with an LSTM classifier reach 0.72 F1, the best of seven compared approaches.","lead":"This paper tests computer programs that tell apart two very similar Ethiopian languages, Wolayta and Gofa. The best system, English BERT word embeddings plus an LSTM classifier, reached 0.72 F1 on a new 144,000-word annotated dataset, a useful first benchmark for low-resource Omotic language identification.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 0.72 F1 may be an artifact of §4.2's removal of common words: if shared wal-gof words were deleted before annotation, the test set no longer samples the bilingual LID task the abstract claims to solve.","rationale":"The reader's weakest assumption correctly identifies the ground-truth label space as the load-bearing element. My stress-test sharpens this: the issue is not only annotator agreement but the coherence of the label space after the common-word removal step in §4.2. If shared words are removed before annotation, the wal-gof class and the task itself change, making the reported F1 a measure of an easier, filtered problem. This is more decisive than the lack of error bars or the numerical inconsistency between the table (0.62) and the text (0.68) for BiLSTM with attention. I agree with the reader's assessment and recommend keeping the CONDITIONAL verdict: the check above could either validate the 0.72 F1 or show that it is an artifact of preprocessing.","tokens_in":9753,"tokens_out":3369,"duration_ms":32669,"concrete_test":"Ask the authors to release both the original unfiltered dataset and the Kutools-filtered version, then independently annotate a random sample of unfiltered words with the same three-annotator majority protocol and compute Fleiss' kappa. Retrain and evaluate the BERT+LSTM model on an unfiltered test set of 1,000 words that includes common/shared items. If F1 stays within 0.05 of 0.72 and kappa is at least 0.8, the filtering concern does not land; if F1 drops materially or agreement is low, the reported result is not a faithful estimate of bilingual LID for these languages.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 states: \"We used Excel Kutools to remove the matching common words from both Wolayta and Gofa to annotate the data.\" Section 4.3 defines wal-gof as the label for words shared by both languages, and Table 2 lists such words (e.g., Kaallidi, Hara, Eridi, Doonan). If common words were literally removed before annotation, the wal-gof class is either emptied or reduced to residual items that survived a heuristic spreadsheet filter. The 0.72 F1 is then measured on a test set from which the most confusable items have been excluded, so it does not support the abstract's general claim about bilingual word-level LID for Wolayta and Gofa. The paper also reports no inter-annotator agreement (only majority vote, §4.4) and no train/test split details, so one cannot determine whether filtering or annotator noise drove the result. This is a load-bearing concern because the central claim is an empirical comparison of classifiers on a specific task; if the task definition itself was altered during preprocessing, the comparison is internally consistent but measures a truncated problem.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a word-level bilingual language identification (LID) system for Wolayta and Gofa, two closely related Omotic languages spoken in southern Ethiopia. The authors collected 144K words from religious, educational, and social media sources, annotated them as wal, gof, or wal-gof by majority vote of native speakers, and evaluated several models: logistic regression, LSTM and BiLSTM with attention, CNN, CNN+LSTM, CNN+BiLSTM, RoBERTa+LSTM, and BERT-base-uncased+LSTM. They report the best result for BERT-base-uncased+LSTM with macro precision 0.76, recall 0.69, and F1 0.72 on the test set. The paper claims this is the first LID work for the Omotic language family and motivates applications in social-media content analysis and downstream NLP tasks.","tokens_in":9994,"tokens_out":4516,"duration_ms":36803,"significance":"If the dataset and preprocessing are sound, this is a useful first benchmark for an under-resourced language pair, comparing classical, deep, and transformer-hybrid approaches on a new word-level annotation. The strongest contribution is the creation of a new annotated dataset for Wolayta and Gofa and the systematic comparison of multiple architectures. However, the significance is currently limited by unclear dataset release, ambiguous preprocessing, and inconsistent result reporting. The central performance claim is plausible and the top model's F1 is internally consistent with its precision and recall, but the empirical comparison must be verified after the reported issues are resolved.","major_comments":[{"comment":"The preprocessing description contradicts the definition of the wal-gof class. Section 4.2 states: \"We used Excel Kutools to remove the matching common words from both Wolayta and Gofa to annotate the data.\" Section 4.3 then defines wal-gof as a label for words shared by both languages, and Table 2 lists examples such as Kaallidi, Hara, Eridi, and Doonan. If matching common words were literally removed before annotation, the wal-gof class could not contain those examples; if they were not removed, the sentence is misleading. This is load-bearing because the reported 0.72 F1 is measured on the resulting test set, and removing the most confusable shared words would change the task from the claimed bilingual LID problem. The authors should clarify exactly what was removed, how the wal-gof class was constructed after filtering, and report the per-class distribution in the final training and test sets.","section":"Sections 4.2, 4.3 and Table 2"},{"comment":"The result table is internally inconsistent. For Logistic Regression, Table 3 reports precision 0.55 and recall 0.76, whose harmonic mean is 0.64, but the table lists F1 as 0.47. For BiLSTM with attention, precision 0.70 and recall 0.66 give F1 0.68, but the table lists 0.62; Section 6 text says BiLSTM with attention achieved F1 0.68. These discrepancies make the reported comparison unreliable and must be corrected. The central ranking (BERT+LSTM best at 0.72) may survive correction, but the magnitude of differences between models depends on this fix.","section":"Table 3 and Section 6"},{"comment":"The paper omits essential experimental details for reproducibility. It does not report the train/test split ratio, the test set size, the random seed, or whether the split was stratified by source domain. It claims an \"open source dataset\" in Section 1 but provides no repository link or availability statement. Section 4.4 reports only majority voting among three annotators and gives no inter-annotator agreement measure, so the reliability of the ground-truth labels cannot be assessed. These details are necessary to evaluate whether the 0.72 F1 result is stable and whether the dataset can be used by other researchers.","section":"Sections 4.1, 4.4 and 4.5"}],"minor_comments":[{"comment":"There is a typo: \"BiLSTM wiith attention\" should be \"BiLSTM with attention.\"","section":"Table 3"},{"comment":"The keywords line reads \"Keywords:Keywords: LID...\"; the duplicated prefix should be removed.","section":"Keywords line"},{"comment":"Some listed contributions are future plans rather than contributions of this paper, e.g., \"The proposed model will be applied in the other low-resource Ethiopian languages.\" Please rephrase to describe what is actually delivered.","section":"Section 1 contributions"},{"comment":"The dataset distribution is shown only as percentages in Figure 3; please provide exact counts or a table with the number of words per class and per domain, as this is important for interpreting the class imbalance.","section":"Figure 3 and Section 4.5"},{"comment":"Several reference entries are incomplete or inconsistently formatted (e.g., [36] and [37] lack full publication details, and some entries appear multiple times with different numbering). Please check the reference list against the venue's style guidelines.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's reference list contains a high number of self-citations and closely related companion papers (e.g., [6], [8], [12], [14], [16], [18], [20], [36], [37], [41], [43], with some repeated entries). The editor may wish to verify that these citations are all necessary and that the bibliography follows the venue's citation guidelines. In addition, the manuscript claims an open-source dataset but does not give a URL; the editor should require a data availability statement before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a modest but genuinely new piece of work for a language pair nobody has benchmarked before. The authors built a 144K-word annotated corpus for Wolayta and Gofa, ran a standard battery of ML, DL, and transformer+LSTM classifiers, and got a plausible 0.72 macro F1 with BERT-base+LSTM. The architecture is explicitly borrowed from Tonja et al. on Kannada-English, and the authors don't pretend otherwise. That's fine—the contribution is the dataset and the first reported evaluation for these Omotic languages.\n\nThe paper does several things well. The annotation procedure is carefully described for a low-resource setting: three native speakers per language, majority vote, author review. The results table is internally coherent; the precision/recall numbers match the F1 values. The related work situates the task within existing bilingual LID efforts. If the dataset were shipped, this would be a useful baseline for future work on these languages.\n\nNow the soft spots, and they're not minor. Section 4.2 says: \"We used Excel Kutools to remove the matching common words from both Wolayta and Gofa to annotate the data.\" The stress-test note is right: if identical words are removed before annotation, the wal-gof class (shared words) is either emptied or reduced to whatever survives a heuristic spreadsheet filter. The test set then no longer samples the actual bilingual LID task—the hardest cases are gone. The abstract's claim about identifying between the two languages is measured on a truncated problem. This is load-bearing, and the paper never explains what the wal-gof label means after that filtering.\n\nOther issues: the dataset is described as \"open source\" but no link is given. There is no train/test split description, no inter-annotator agreement, no error bars. The BiLSTM attention F1 appears as 0.62 in Table 3 and 0.68 in Section 6—probably a typo, but it needs fixing. Some references look like padding, but that's secondary.\n\nWho should read this? Anyone working on low-resource Ethiopic NLP. But the value depends on the data being released and the preprocessing clarified.\n\nVerdict on peer review: yes, send it out. The data collection effort and the specific language pair deserve referee time. But the revision must address the common-word removal, release the dataset, and fill in the experimental details. As it stands, the result is not independently verifiable.\n\nBest,\n[You]","headline":"A useful first dataset and evaluation for Wolayta-Gofa LID, but the reported 0.72 F1 is undermined by an unexplained preprocessing step that removes shared words and by the fact that the dataset is not released.","tokens_in":10570,"tokens_out":1941,"would_cite":false,"duration_ms":19524,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BERT embeddings plus an LSTM tag Wolayta and Gofa words at 0.72 F1","keywords":["bilingual language identification","word-level LID","Omotic languages","Wolayta","Gofa","BERT","LSTM","low-resource NLP"],"falsifier":"Take a random sample of the original unprocessed data, have fresh native speakers annotate every word without removing common words first, and compute inter-annotator agreement; then retrain the BERT+LSTM model on the unmodified labels and compare F1. If agreement is near chance or the F1 drops substantially, the reported 0.72 depends on the filtering and majority-vote procedure.","tokens_in":9554,"feed_emoji":"🔤","tokens_out":4015,"duration_ms":31688,"temperature":0.7,"pith_summary":"This paper tries to establish that word-level bilingual language identification is possible for Wolayta and Gofa, two closely related Omotic languages of southern Ethiopia whose shared vocabulary makes them hard to tell apart. The authors built and manually annotated a 144,000-word dataset with three labels: Wolayta, Gofa, and common words used by both. Among ten model configurations, a pipeline that feeds BERT-base-uncased embeddings into an LSTM classifier performed best, with a macro F1 score of 0.72 on the test set. If the result holds, it offers a practical tool for detecting which language a social media post or comment is written in and a first benchmark for NLP on these languages.","feed_headline":"BERT plus LSTM tells Wolayta and Gofa apart at 0.72 F1","feed_subtitle":"Word-level test on 144,000 annotated words shows contextual embeddings beat logistic regression, CNNs, and RoBERTa.","key_machinery":"The central mechanism is a hybrid pipeline: BERT-base-uncased produces contextualized word embeddings, and those embeddings are passed to an LSTM layer with 128 units, followed by batch normalization, two dense layers of 768 units with ReLU, a dropout layer of 0.1, and a softmax output over the three classes wal, gof, and wal-gof. The BERT embeddings do the load-bearing work of encoding subtle lexical and morphological differences between the two languages, while the LSTM models the sequential structure of the input and the classifier assigns the language label.","core_discovery":"On the authors' annotated three-class test set, the BERT-base-uncased plus LSTM model achieves 0.76 macro precision, 0.69 recall, and 0.72 F1, beating logistic regression (0.47), LSTM with attention (0.55), BiLSTM with attention (0.62), CNN (0.64), CNN+LSTM (0.66), CNN+BiLSTM (0.69), and RoBERTa+LSTM (0.69). The authors claim that pretrained contextual embeddings combined with an LSTM classifier are an effective approach for word-level bilingual LID of closely related, low-resource Omotic languages, and that the work provides the first dataset and benchmark for this language pair.","pith_inferences":["If the labels are reliable, the same BERT+LSTM recipe should generalize to other closely related Ometo languages, but its margin over simpler models may shrink as the shared vocabulary grows.","Removing common words before annotation likely made the wal-gof class sparse or artificial; on raw, unfiltered text the reported 0.72 F1 could drop, so a test on unmodified data would be needed for deployment.","Reporting per-class precision and recall, plus an inter-annotator agreement score, would show whether the wal-gof class is a genuine linguistic category or mainly an annotation artifact."],"forward_implications":["The proposed model identifies the language of short, word-level texts, which is the setting typical of tweets and comments.","The annotated 144,000-word dataset gives a first resource for further NLP work on Wolayta and Gofa.","Word-level LID can separate languages even when they share a large common vocabulary, as long as contextualized representations are used.","The approach is presented as transferable to other low-resource Ethiopian languages and to applications such as hate-speech detection and machine-translation prefiltering."],"supporting_citations":[{"why":"Supplies the transformer-plus-LSTM method for word-level LID that this paper adapts to Wolayta and Gofa.","marker":"[23]"},{"why":"Establishes code mixing as a challenge for LID on social media and frames the motivation for word-level identification.","marker":"[19]"},{"why":"Documents the specific NLP challenges of Omotic languages, including word irregularity and limited digital data.","marker":"[11]"},{"why":"Provides the religious-domain parallel data used as one of the sources for the new dataset.","marker":"[34]"},{"why":"Places Wolayta and Gofa within the Northern Omotic group, supporting the claim that the two languages are closely related.","marker":"[29]"},{"why":"Shows a deep-learning baseline for word-level LID on another language pair and gives the comparison point for this work.","marker":"[15]"}],"fun_headline_variants":["BERT-LSTM model distinguishes Wolaita and Gofa with 0.72 F1","Word-level LID for Omotic languages: BERT+LSTM wins at 0.72 F1","First benchmark for bilingual Omotic LID: 0.72 F1 from BERT+LSTM","Closely related languages? BERT+LSTM still hits 0.72 F1","Wolaita vs Gofa: BERT+LSTM beats other models at 0.72 F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ground-truth labels are correct: each word was labeled by majority vote among three native speakers with no reported agreement measure, and shared words were removed with Excel Kutools before annotation, so the definition of the common-word class is not demonstrated on naturally occurring data.","fun_headline_variants_meta":{"raw":{"variants":["BERT-LSTM model distinguishes Wolaita and Gofa with 0.72 F1","Word-level LID for Omotic languages: BERT+LSTM wins at 0.72 F1","First benchmark for bilingual Omotic LID: 0.72 F1 from BERT+LSTM","Closely related languages? BERT+LSTM still hits 0.72 F1","Wolaita vs Gofa: BERT+LSTM beats other models at 0.72 F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000644,"raw_usage":{"total_tokens":2910,"prompt_tokens":843,"completion_tokens":2067,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":459,"completion_tokens_details":{"reasoning_tokens":1945}},"tokens_in":459,"tokens_out":2067,"duration_ms":13689,"temperature":1.0,"reasoning_tokens":1945,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:23:21.776017+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the original unprocessed data, have fresh native speakers annotate every word without removing common words first, and compute inter-annotator agreement; then retrain the BERT+LSTM model on the unmodified labels and compare F1. If agreement is near chance or the F1 drops substantially, the reported 0.72 depends on the filtering and majority-vote procedure.","supporting_citations":[{"cited_title":"L., waldeyohannis, M","cited_arxiv_id":null,"evidence_quote":"Provides the religious-domain parallel data used as one of the sources for the new dataset."},{"cited_title":"”Code mix- ing: A challenge for LID in the language of social media.” In Proceedings of the first workshop on computational approaches to code switching, pp","cited_arxiv_id":null,"evidence_quote":"Establishes code mixing as a challenge for LID on social media and frames the motivation for word-level identification."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the specific NLP challenges of Omotic languages, including word irregularity and limited digital data."},{"cited_title":"T., and waldeyohannis, M","cited_arxiv_id":null,"evidence_quote":"Places Wolayta and Gofa within the Northern Omotic group, supporting the claim that the two languages are closely related."},{"cited_title":"G., Tonja, A","cited_arxiv_id":null,"evidence_quote":"Shows a deep-learning baseline for word-level LID on another language pair and gives the comparison point for this work."}],"review_version":2}