{"id":"d24f4810-98ae-4851-82ad-eca4dfdcd73d","arxiv_id":"2501.15773","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Using 100-tree random forests on vectorized text features, Navajo can be separated from Google LangID's false suggestions with 97% accuracy, but the model's behavior on Apache languages shows it cannot distinguish within the Athabaskan family.","lead":"Researchers built a simple random forest classifier that distinguishes Navajo from twenty unrelated languages Google's language ID tool confuses it with, and report near-perfect accuracy on held-out test sentences. It also flags why generalization claims can mislead: the model tends to label related Apache languages as Navajo rather than telling them apart.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Athabaskan robustness claim rests on a false-positive rate: Table 2 reports Apache sentences classified as Navajo, which is model collapse, not evidence of generalization.","rationale":"The paper has a credible narrow result: a 100-tree random forest can separate Navajo from the twenty unrelated languages Google LangID confuses with Navajo, with the confusion matrix and Table 1 supporting 97% overall accuracy and Navajo F1 near 0.99. I read that part of the paper in good faith and do not object to it. The problem is the broader claim in the title, abstract, and Section 3.4 that the model is robust across Athabaskan languages. That claim rests on Table 2, which reports the fraction of Apache sentences classified as Navajo. Under the paper's own task definition, those Apache sentences are not Navajo, so the table measures the false-positive rate for Navajo, not detection accuracy for Apache languages. A trivial always-Navajo classifier reproduces the reported numbers, so the result cannot distinguish family-level generalization from a default-label bias. The reader's verdict already flags the Apache test as too small and notes that the model collapses related languages into Navajo; my stress-test sharpens this by identifying the precise metric error: the paper treats a false positive as evidence of generalization. I therefore keep the CONDITIONAL verdict, with the condition being that the Athabaskan generalization claim must be removed or replaced by evidence from an evaluation in which Apache languages are treated as their own classes, not as Navajo. No ad hominem is intended; the issue is the argument's logic and the interpretation of Table 2.","tokens_in":7256,"tokens_out":3482,"duration_ms":34996,"concrete_test":"Treat each Apache sentence's true language as the label and recompute the classifier's full confusion matrix over its 21 output classes, reporting precision and recall for Western Apache, Mescalero Apache, Jicarilla Apache, and Lipan Apache as separate classes. If correct-class accuracy is near zero while the Navajo column absorbs nearly all Apache sentences, the robustness claim in the abstract and Section 3.4 is refuted. As a control, also run 100+ unseen sentences from a non-Athabaskan language (e.g., Cherokee) through the same classifier; if it is also labeled Navajo at high rates, the observed behavior is a default-label artifact rather than family-level generalization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4 and Table 2 present the percentage of Apache sentences classified as Navajo (Western Apache 96.00%, Mescalero Apache 100%, Jicarilla Apache 92.31%, Lipan Apache 62.16%) as evidence that the classifier 'demonstrates robustness across other Athabaskan languages.' This is the central support for the abstract's claim of broader Athabaskan applicability. However, the classifier was trained for a 21-way distinction between Navajo and twenty unrelated languages (Sections 3.2 and 3.3); for an Apache sentence, the label 'Navajo' is a false positive under the task definition, not a correct detection. A degenerate model that always predicts Navajo would achieve exactly the reported pattern: 100% on Mescalero Apache and 96% on Western Apache. The 107-sentence curated sample (25, 32, 13, and 37 sentences per language) is far too small for a family-level claim, and the Limitations section itself concedes that the binary design ignores intra-group variation. Thus the broad Athabaskan claim is not merely under-supported; the reported metric actually demonstrates the model's inability to distinguish related Athabaskan languages from Navajo, which is the opposite of the stated generalization.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses the absence of Navajo in Google's LangID by training a 100-tree random forest with 5,000 features to classify Navajo versus the twenty languages that LangID most often suggests for Navajo sentences. Using Leipzig corpora for both Navajo and the distractor languages, the model is trained on 153,832 samples and evaluated on a random held-out split of 38,458 samples, where it reaches 97% overall accuracy and Navajo precision/recall/F1 of 1.00/0.99/0.99. The authors then test the same classifier on 107 curated Apache sentences and interpret the percentages classified as Navajo as evidence of robustness across the Athabaskan family. They release code and data on GitHub. The Navajo-only result is plausible, but the paper's broader Athabaskan claim is not supported by the experiment as designed.","tokens_in":7450,"tokens_out":8397,"duration_ms":78367,"significance":"If restricted to the 21-way benchmark, the paper provides a transparent, reproducible baseline: it reports per-class metrics, a full confusion matrix, and public code/data, and it shows that a simple random forest can separate Navajo from its LangID confusables with high accuracy. That is a modest but real contribution to low-resource language identification, particularly as a counterexample to the assumption that such tools require large neural models. The claimed extension to other Athabaskan languages, however, is not merely under-supported; the reported metric measures false positives, so the paper currently overstates its scope. After correcting that interpretation, the remaining contribution is a focused Navajo-language identification system rather than a family-level tool.","major_comments":[{"comment":"The paper's central generalization claim, repeated in the abstract and conclusion, is that the classifier demonstrates robustness across other Athabaskan languages. The only evidence is Table 2, which reports the percentage of Apache sentences classified as Navajo. Under the 21-way task defined in Section 3.2, the classifier has no Apache class, so every Apache sentence that receives the label Navajo is a false positive. A trivial always-Navajo baseline would achieve 100% on every Apache language, so the reported 100% for Mescalero Apache and 96% for Western Apache are consistent with a model that has learned no Apache-specific rejection behavior. These numbers therefore do not show generalization; they show that the model cannot distinguish closely related Apache languages from Navajo, which is the opposite of the stated claim. The abstract, Section 3.4, and the conclusion must be rewritten to remove or invert this interpretation, ideally with a baseline that includes a reject option or an Apache-inclusive classifier.","section":"Section 3.4 / Table 2"},{"comment":"The near-perfect accuracy is computed on a random split of the Leipzig corpora (153,832 training samples and 38,458 test samples). Because the Navajo and distractor sentences come from the same Leipzig sources, the test set is likely to be highly similar to the training text in topic, genre, and orthography. This supports the statement that the classifier separates the 21 classes in this corpus, but it does not by itself support the broader claim of accurate language detection for Navajo text in the wild. If the authors retain real-world applicability language, they should add an out-of-domain evaluation using, for example, a separate Navajo corpus, social media text, or community-collected sentences, or they should clearly restrict the claim to the benchmark distribution.","section":"Section 3.3 (evaluation split)"},{"comment":"The conclusion states that the study demonstrates effective recognition of related Athabaskan languages, while the Limitations section concedes that the design assumes a binary distinction between Navajo and other languages without considering intra-group variation and dialectical differences. This internal tension is not harmless: the family-level claim is precisely what the limitation negates. Section 3.4 goes further and interprets the varying false-positive rates for Western and Mescalero Apache (96-100%) versus Jicarilla and Lipan Apache (62-92%) as evidence about the Western and Eastern Apachean subgroups; with only 13-37 sentences per language and no confidence intervals, this observation cannot support a linguistic generalization. The paper should either add a proper multi-class Apache evaluation or state plainly that the classifier is Navajo-specific.","section":"Conclusion and Limitations"}],"minor_comments":[{"comment":"The phrase near-perfect accuracy (97-100%) is not supported by Section 3.3, which reports a single overall accuracy of 97%; per-language F1 scores in Table 1 range from 0.87 to 0.99, so the meaning of the 97-100% range should be clarified.","section":"Abstract"},{"comment":"The row format concatenates the numeric class label with the language name and the metrics in a way that is hard to read; separate the class index, language name, precision, recall, and F1 into distinct columns.","section":"Table 1"},{"comment":"The x-axis order is arbitrary and only eight of the twenty languages are shown; either show all twenty languages or explicitly state that the figure is a subset.","section":"Figure 2"},{"comment":"The sentence 'The detection for Navajo performed best for Western Apache and Mescalero Apache' is misleading because the values are false-positive rates; a detector that rejects more Apache sentences would be better, so the wording should be revised to 'the highest false-positive rates were observed for...'.","section":"Section 3.4"},{"comment":"The Limitations section refers to Google Translate instead of Google LangID; the terminology should be consistent throughout the paper.","section":"Limitations"},{"comment":"The feature representation is described only as 5,000 features; please specify whether these are character n-grams, word n-grams, or another type, and clarify how the aligned words in Appendix A relate to the four Apache varieties tested in Table 2, since the appendix appears to draw on Chiricahua Apache rather than those four languages.","section":"Section 3.2 and Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The central experimental result on the 21-way Navajo task appears sound and is presented with useful transparency. The main problem is an interpretive error in Section 3.4 that propagates to the title, abstract, and conclusion: the Apache results are false-positive rates, not evidence of family-level generalization. If this claim is corrected by removing or inverting the Athabaskan robustness interpretation, the remaining paper is a modest but publishable systems contribution. If the authors insist on retaining the current framing, I would not recommend acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The narrow result in this paper is real and worth knowing: a simple random forest trained on Navajo plus the twenty languages Google's LangID falsely suggests for Navajo achieves 97% accuracy on a 38k held-out test set, with Navajo at 1.00 precision / 0.99 recall. That is a clean, useful empirical artifact. The dataset construction is sensible — using LangID's own false suggestions as the candidate set directly matches the deployment scenario — and the authors release code and data. Credit where due: this is reproducible work, and the per-class confusion matrix is honest about the confusions that remain.\n\nThe soft spot is not subtle. The paper's abstract says the model shows \"robustness across other Athabaskan languages,\" and Section 3.4 presents Table 2 as evidence. But Table 2 reports the percentage of Apache sentences classified as Navajo when the model was trained for a 21-way distinction between Navajo and unrelated languages. Labeling Apache as Navajo is a false positive under the task definition. A degenerate classifier that always predicts Navajo would score 100% on Mescalero Apache and 96% on Western Apache — almost exactly what they report. So the Apache numbers demonstrate the model's inability to distinguish related Athabaskan languages from Navajo, not its ability to generalize. The 107-sentence curated sample (25, 32, 13, 37 per language) is far too small to support a family-level claim, and the Limitations section itself concedes the binary design ignores intra-group variation. The authors seem to have mistaken a failure mode for a feature.\n\nOther concerns are minor. The Leipzig random split means the 97% figure is in-domain; out-of-domain Navajo, dialectal variation, and code-switching remain untested, and the paper does not overclaim on that front. The method is standard, but that is not a flaw. The self-citation to their own Nüshu paper is contextually appropriate.\n\nWho is this for? Anyone building practical language ID for endangered or under-resourced languages, and researchers interested in using a commercial tool's blind spots to bootstrap training data. The narrow classifier is a solid contribution; the Athabaskan framing needs to be removed or completely reworked.\n\nRecommendation: send to peer review. A serious referee can push the authors to reframe the Apache section and either present it as a limitation or as evidence for future work on closely-related language discrimination. The core Navajo classifier deserves publication.","headline":"The Navajo-vs-20-languages classifier result is solid and reproducible, but the Athabaskan generalization claim is unsupported: the Apache table actually shows model collapse, not robustness.","tokens_in":8010,"tokens_out":991,"would_cite":false,"duration_ms":11348,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 100-tree random forest, trained on Navajo and the twenty languages Google's LangID incorrectly suggests for it, distinguishes Navajo from those languages with near-perfect accuracy (97–100%) and appears to recognize related Athabaskan…","keywords":["language identification","endangered languages","Navajo","Athabaskan languages","random forest","low-resource NLP","Google LangID","linguistic diversity"],"falsifier":"Take the released classifier and apply it to Navajo text that is not from the Leipzig corpus—for example, social media posts, chat messages, or transcribed interviews—and measure accuracy against human judgments; if accuracy falls well below the reported 97–100%, the generalization claim is false. Similarly, collecting a balanced sample of several hundred sentences from each of the four Apache languages and measuring per-language classification rates would settle whether the family-level robustness claim holds beyond the 107-sentence curated sample.","tokens_in":7026,"feed_emoji":"🗣️","tokens_out":5866,"duration_ms":47955,"temperature":0.7,"pith_summary":"This paper asks whether a simple machine-learning model can identify Navajo text when the only competing languages are the twenty languages that Google's language identification tool mistakenly suggests for Navajo. The authors train a 100-tree random forest on Navajo sentences from the Leipzig corpus plus samples from those twenty languages, and report near-perfect accuracy (97–100%, with Navajo precision 1.00, recall 0.99). The point is that a small, transparent, easily replicated model can cover an endangered language that commercial tools ignore, and that the same model may recognize closely related Athabaskan languages such as Western Apache and Mescalero Apache. The paper positions this as a step toward decentralized NLP tools that serve under-resourced languages.","feed_headline":"Simple model identifies Navajo with 97–100% accuracy","feed_subtitle":"A 100-tree random forest beats Google's LangID for an unsupported Native American language.","key_machinery":"The load-bearing object is the random forest classifier, a set of 100 decision trees whose individual votes are aggregated, applied to sentence-level text features encoded as 5,000-dimensional vectors. It does the work of separating Navajo from the twenty confusable languages using a modest, interpretable model, and it is the same model, without retraining, that is applied to the Apache generalization test. The candidate set itself matters: the twenty languages are not chosen for linguistic similarity but because Google's LangID returns them for Navajo input, so the classifier learns to reject exactly the confusions that a real user would encounter.","core_discovery":"The central claim is that language identification for an unsupported endangered language need not wait for large-scale multilingual models: a 100-tree random forest trained on 153,832 sentence samples, vectorized into 5,000 features, distinguishes Navajo from the twenty languages Google's LangID erroneously returns, achieving an overall accuracy of 97% and Navajo F1 of 0.99 on a held-out test set. The authors further claim that this Navajo-trained classifier recognizes related Athabaskan languages, classifying 96% of Western Apache sentences, 100% of Mescalero Apache, 92% of Jicarilla Apache, and 62% of Lipan Apache as Navajo, with the pattern tracking the Western/Eastern Apachean subgrouping. The broader argument is that targeted, family-level tools can provide practical support for endangered languages that centralized platforms omit.","pith_inferences":["The Apache generalization test is only 107 sentences total, so the family-level robustness claim is suggestive rather than established; a larger, balanced sample of each Apache language would be needed to know whether the Western/Eastern subgrouping pattern is real.","The reported accuracy is measured on random splits of the Leipzig corpus, so out-of-domain Navajo text—social media, code-switched speech, dialectal variants, transcribed oral narratives—could perform worse; testing on such data would clarify actual deployment behavior.","A natural extension is to treat the classifier's confidence scores as a signal for which languages are genuinely confusable with Navajo, which could guide linguists toward documenting dialectal or orthographic overlaps.","If the Western/Eastern Apachean performance difference is robust, a model trained on all Athabaskan languages jointly might outperform the single Navajo-trained model and provide a blueprint for other endangered language families."],"forward_implications":["A straightforward random forest, trained on freely available Leipzig corpora, can serve as a practical Navajo language identifier that any community or researcher can run locally, without relying on commercial APIs.","Because the same model flags related Athabaskan languages as Navajo, a single trained classifier could help triage text from several endangered languages until dedicated per-language models are built.","The high precision and recall on the closed set of twenty confusable languages shows that commercial LangID failures can be corrected with a small, targeted supplement rather than a massive retraining effort.","The public release of code and data makes the result directly checkable and reusable by preservation groups working on Navajo and other Athabaskan languages."],"supporting_citations":[{"why":"Supplies the random forest method that is the core classifier.","marker":"Ho, 1995"},{"why":"Provides the Leipzig corpora from which Navajo and the twenty comparison languages are sourced.","marker":"Goldhahn et al., 2012"},{"why":"Documents Google's LangID, the tool whose erroneous Navajo suggestions define the candidate set.","marker":"Caswell et al., 2020"},{"why":"Supplies the Athabaskan family tree used to interpret Western versus Eastern Apachean performance.","marker":"Saxon, 2023"},{"why":"Source of the Western Apache texts used in the generalization test.","marker":"Glosbe, 2024"},{"why":"Source of the Mescalero Apache texts used in the generalization test.","marker":"Library, 2024"},{"why":"Source of the Jicarilla Apache texts used in the generalization test.","marker":"Wikipedia, 2024"},{"why":"Source of the Lipan Apache texts used in the generalization test.","marker":"Indians.org, 2024"}],"fun_headline_variants":["Random forest beats Google for Navajo language ID","Navajo detection: 97–100% accuracy with simple model","Simple tool identifies endangered Athabaskan languages","No big model needed: Navajo ID at 97–100% accuracy","Google misses Navajo, random forest nails it"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that a random split of the Leipzig Navajo corpus and the twenty-language samples yields a test set that represents how Navajo text appears in the wild, so the near-perfect accuracy will carry over to real-world usage.","fun_headline_variants_meta":{"raw":{"variants":["Random forest beats Google for Navajo language ID","Navajo detection: 97–100% accuracy with simple model","Simple tool identifies endangered Athabaskan languages","No big model needed: Navajo ID at 97–100% accuracy","Google misses Navajo, random forest nails it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1411,"prompt_tokens":886,"completion_tokens":525,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":448}},"tokens_in":502,"tokens_out":525,"duration_ms":5463,"temperature":1.0,"reasoning_tokens":448,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:57:19.946923+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the released classifier and apply it to Navajo text that is not from the Leipzig corpus—for example, social media posts, chat messages, or transcribed interviews—and measure accuracy against human judgments; if accuracy falls well below the reported 97–100%, the generalization claim is false. Similarly, collecting a balanced sample of several hundred sentences from each of the four Apache languages and measuring per-language classification rates would settle whether the family-level robustness claim holds beyond the 107-sentence curated sample.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Leipzig corpora from which Navajo and the twenty comparison languages are sourced."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents Google's LangID, the tool whose erroneous Navajo suggestions define the candidate set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Athabaskan family tree used to interpret Western versus Eastern Apachean performance."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the Western Apache texts used in the generalization test."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the Mescalero Apache texts used in the generalization test."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the Jicarilla Apache texts used in the generalization test."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the Lipan Apache texts used in the generalization test."}],"review_version":1}