{"id":"daf72ecc-201b-4d81-9018-bb4c935e1eb9","arxiv_id":"2411.12262","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A study of 100,000 tetun.org logs shows users translate high-resource to Tetun mostly for education and science, a domain mix almost opposite to available Tetun corpora.","lead":"This paper analyzes 100,000 real translation requests to the Tetun-language service tetun.org and finds that users mostly translate educational and scientific material into Tetun on mobile phones. The findings suggest that low-resource MT systems should be built and evaluated around what users actually translate, not around the news-heavy corpora that dominate training data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline domain-mismatch claim rests on classifier labels that are validated only on user logs; the MADLAD-side labels come from a different MT pipeline and are never evaluated, so the large gaps in Tables 3 and 4 may be classification artifacts.","rationale":"The reader's weakest-assumption analysis correctly identifies the classifier reliability and differential error across pipelines as the structural weak point. My stress-test agrees and sharpens it: the problem is not just that overall F1 is moderate, but that the comparison corpus (MADLAD) goes through a separate machine-translation step (MADLAD-3B Tetun-to-English) and is never included in the classifier evaluation. This creates a concrete mechanism by which the observed domain mismatch could be inflated: if the classifier systematically assigns translated news text to 'Government & socio-economic issues' or 'News article & press release', the 83.1% and 61.2% figures could be overestimates for MADLAD, while systematic assignment of user queries to 'Science & research' and 'Education & research material' could inflate the MT side. The paper reports micro and weighted F1 but not per-class precision/recall or any calibration analysis, so we cannot tell from the manuscript whether such systematic biases exist. The proposed test directly addresses this by measuring per-class accuracy on MADLAD and applying a confusion-matrix correction to the headline numbers. I do not see grounds to reject the paper; the data and service are real, the analysis is transparent, and the large effect sizes might survive correction. But the central claim is conditional on this validation, so the reader's CONDITIONAL verdict remains appropriate. My verdict adjustment is therefore UNCHANGED, with the concrete test as the condition for accepting the mismatch as decisive.","tokens_in":13120,"tokens_out":2804,"duration_ms":32904,"concrete_test":"Have two independent annotators label a stratified random sample of 200 MADLAD documents (e.g., 50 per quartile of the MADLAD-3B English translations, plus 50 originals if feasible) using the paper's topic and provenance schemas. Run the same Llama 3.1 8B classifier on the English translations and compute per-class precision/recall for MADLAD. Then apply a confusion-matrix correction (or reweighting) to the reported MADLAD and MT distributions, and recompute the differences in Tables 3 and 4 with bootstrap confidence intervals. If the corrected MT-vs-MADLAD gaps shrink to less than, say, a factor of 2 on the headline categories ('Science & research', 'Education & research material', 'Government', 'News'), the central mismatch claim is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that real tetun.org requests concentrate in 'Science & research' (34.2%) and 'Education & research material' (55.9%) while MADLAD is dominated by 'Government & socio-economic issues' (83.1%) and 'News article & press release' (61.2%), driving the paper's recommendations. This comparison depends entirely on Llama 3.1 8B labels assigned under two different pipelines: for MT logs, the classifier sees the user's high-resource text (source or target); for MADLAD, it sees English produced by MADLAD-3B from Tetun (Section 3.1.4). The classifier's weighted F1 is only 0.76 (topic) and 0.73 (provenance) on a 100-log, single-annotator test set (Section 3.1.1), and this test set contains only user requests, not MADLAD documents. Nothing in the paper checks whether classification error is comparable across the two pipelines or whether per-class precision/recall differs. If the classifier over-tags education/science on short, instructional user texts but over-tags government/news on translated MADLAD articles, the 10-fold gaps in Tables 3 and 4 could largely reflect label bias rather than true domain mismatch. This is load-bearing because without the mismatch, the paper's main recommendation (shift MT evaluation and corpus collection toward education, science, health, and the high-resource-to-Tetun direction) loses its empirical foundation. The paper also acknowledges selection bias in server logs, but that is a separate limitation; the immediate threat is that the primary evidence for the mismatch is unvalidated on the comparison corpus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an observational study of 100,000 translation requests logged by tetun.org, a dedicated Tetun translation service. Using server logs and analytics data, it characterizes users (mostly mobile users in Timor-Leste, plausibly students), translation directions (mostly high-resource into Tetun), input lengths (mostly short texts), and domains. Domain is decomposed into topic and provenance, and labels are produced by Llama 3.1 8B applied to the high-resource side of each translation pair, with a weighted F1 of 0.76 for topic and 0.73 for provenance on a 100-log, single-annotator test set. These labels are compared against the MADLAD-400 Tetun corpus, whose texts are first machine-translated to English with MADLAD-3B and then classified with the same model. The paper reports a large mismatch: Tetun.org inputs are dominated by science/research and education/research material, while MADLAD is dominated by government and news provenance. The paper concludes with recommendations to prioritize the high-resource-to-Tetun direction, educational/scientific domains, short texts, and mobile-friendly MT.","tokens_in":13506,"tokens_out":3691,"duration_ms":40422,"significance":"If the reported domain mismatch is real, this is a valuable contribution: it is among the first large-scale observational studies of actual end-user MT behavior for a low-resource language, and it provides concrete evidence that corpus-driven assumptions about domain coverage can diverge sharply from community needs. The paper also ships a useful artifact in the form of a machine-translated English version of the MADLAD Tetun corpus, and it is careful to state limitations about selection bias and generalizability. The recommendations for evaluation design and corpus collection are actionable. However, the strength of the empirical conclusion currently rests on a classifier validation that does not cover the MADLAD-side pipeline, and the uncertainty in the headline percentages is not quantified. These gaps are substantial enough that the central claim, while plausible and well-motivated, is not yet established to the standard the recommendations require.","major_comments":[{"comment":"The headline domain-mismatch claim is load-bearing, but the MADLAD-side labels are never validated. The classifier's weighted F1 (0.76 topic, 0.73 provenance) is measured only on a 100-log test set of user requests, whereas for MADLAD the same classifier is applied to English translations produced by a different model (MADLAD-3B). The paper does not report per-class precision/recall, nor does it show that classification error is comparable across the two pipelines. If, for example, the classifier over-tags short instructional user texts as 'Science & research' or 'Education & research material' and over-tags translated news articles as 'Government & socio-economic issues' or 'News article & press release', the 10-fold gaps in Tables 3 and 4 could be substantially artifactual. I ask the authors to validate the classifier on a sample of MADLAD documents (after their English translation) and to report a confusion matrix or per-class precision/recall for both pipelines, or otherwise bound the differential error.","section":"§3.1.1, §3.1.4, Tables 3 and 4"},{"comment":"Because domain percentages are multi-label and word-weighted, the paper should report uncertainty. No confidence intervals or significance tests are given for the percentages in Tables 3 and 4, and several adjacent categories are close (e.g., Education at 22.8% vs. Business & work & employment at 21.2%). A bootstrap over documents, or an inter-annotator agreement measure on the 100-log test set, would clarify whether the reported ordering and the 'large discrepancy' with MADLAD are robust. This is especially important given that the classifier itself has only 0.76/0.73 weighted F1.","section":"§3.1.1, §4.3"},{"comment":"The high-resource side of the input-output pair is not symmetric across translation directions. For English/Indonesian/Portuguese-to-Tetun requests, the classifier sees the user's source text; for Tetun-to-English/Indonesian/Portuguese requests, it sees the MT output. The paper does not report classification accuracy separately by direction, although Table 2 shows that roughly 29% of requests have Tetun as the source. Translation errors in the high-resource output could systematically affect labels, especially for short inputs. I request a direction-wise breakdown of classifier performance or an explicit analysis of whether label distributions differ by which side of the pair is the high-resource text.","section":"§3.1.1, §4.2"},{"comment":"The perplexity comparison between MT inputs and the Labadain corpus is confounded by input length and document segmentation. MT inputs have a median of 8 words, and perplexity on very short fragments is typically inflated relative to longer sentences or documents. The paper's claim that MT inputs 'may contain vocabulary, structures, or topics that are underrepresented in scraped monolingual corpora' should be supported by a length-controlled comparison, for example by computing perplexity on matched-length bins or by reporting perplexity for MT inputs and Labadain sentences in the same length range.","section":"§4.4, §4.2"}],"minor_comments":[{"comment":"There are formatting issues in the table, such as '0 .19' and '0 .40' with an extra space; please fix the spacing.","section":"Table 1"},{"comment":"'LLama 3.1 8B' should be 'Llama 3.1 8B' for consistency with the reference list and standard naming.","section":"§3.1.1"},{"comment":"The y-axis tick labels ('0 k', '20 k', etc.) are difficult to read and the figure would benefit from explicit numeric tick values.","section":"Figure 4"},{"comment":"The statement 'Upon acceptance, we will release our translation' should be rephrased to 'Upon publication' or made unconditional, since the artifact is already described as part of the paper's contribution.","section":"§3.1.4"},{"comment":"The paper should clarify explicitly that percentages are word-weighted and multi-label, and that columns therefore need not sum to 100%; the current tables are easy to misread as exclusive categories.","section":"Tables 3 and 4"},{"comment":"The recommendation to prioritize the 'high-resource to low-resource direction' would benefit from specifying which high-resource language(s) are most relevant, since English, Portuguese, and Indonesian differ in their user shares and in their distance to Tetun.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong and timely dataset, and the qualitative user-behavior findings (mobile-heavy, short inputs, direction asymmetry) are likely robust. The domain-mismatch conclusion, however, is the main hook and it depends on a classifier validation that does not cover the MADLAD-side pipeline. This is fixable with additional experiments, so I recommend major revision rather than rejection. I would also encourage the editor to ask for direction-wise and per-class classifier results, and for a length-controlled perplexity analysis, as these are all within the scope of the current manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is worth reading for the data alone: 100,000 server logs from tetun.org, a real MT service for Tetun. That is a genuinely new kind of evidence—surveys and focus groups can't tell you what people actually translate. The headline findings—users mostly translate from English/Portuguese/Indonesian into Tetun, on mobile, in short chunks, and with an apparent emphasis on education and science—are a useful corrective to corpus-driven assumptions about low-resource MT. The authors also do a few things right: they are transparent about selection bias, they got an ethics waiver and anonymized logs, and they release the translated MADLAD corpus upon acceptance. The perplexity analysis in Section 4.4 is a strong independent check: a Tetun LM trained on MADLAD has 11x higher perplexity on MT inputs than on a different monolingual corpus, so the qualitative conclusion that user inputs are out-of-domain does not rest solely on the classifier. That matters because the stress-test concern—that the domain labels on MADLAD come from an unevaluated pipeline—is real but not fatal. The classifier gets only 0.76/0.73 weighted F1 on a 100-log, single-annotator test set, and the MADLAD side is never validated. If the classifier systematically over-labels user texts as education/science and under-labels news in translated MADLAD, the specific 10-fold gaps in Tables 3 and 4 could shrink substantially. But the perplexity gap and the direction asymmetry (Tetun as target vs. source) are robust to that critique. The paper's soft spots are the missing confidence intervals, the lack of per-class precision/recall, and the single-service, single-language scope. None of these undermine the central contribution, but they do mean the exact domain percentages should be treated as provisional. The authors should be pushed to validate the classifier on MADLAD and report CIs. This is a solid observational study that should go to peer review; with revisions on the classifier analysis, it will be a valuable reference for anyone working on low-resource MT or corpus collection.","headline":"First large-scale observational evidence on who uses low-resource MT; the domain-mismatch claim is plausible and partly corroborated, but the classifier needs stronger validation.","tokens_in":14038,"tokens_out":2029,"would_cite":true,"duration_ms":24771,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An analysis of 100,000 real translation requests shows Tetun users mostly translate short educational texts from high-resource languages into Tetun, clashing with the news-dominated corpora used to train low-resource MT systems.","keywords":["machine translation","low-resource languages","Tetun","observational study","domain mismatch","user behavior","corpus analysis","Timor-Leste"],"falsifier":"Manually annotate a random sample of, say, 500 user requests and 500 MADLAD documents with the same label set, compute the domain gap using the manual labels instead of the classifier's, and check whether the gap between user text and corpus narrows; if it drops below the reported levels, the paper's mismatch claim fails. A cheaper check is to re-run the classification pipeline feeding the Tetun side through a Tetun-capable classifier, if one is available, and compare the resulting distributions.","tokens_in":12959,"feed_emoji":"📚","tokens_out":5534,"duration_ms":51023,"temperature":0.7,"pith_summary":"This paper argues that machine translation for the low-resource, institutionalized language Tetun is being built for the wrong purpose. By analyzing 100,000 real requests to the tetun.org service, it finds that users, many of them students on mobile devices, mostly translate short texts from English, Portuguese, or Indonesian into Tetun, with heavy concentration in science, health, and education. Available Tetun corpora, by contrast, are dominated by news about government and law. The paper concludes that MT systems for Tetun should be optimized for the high-resource-to-Tetun direction, for educational domains, and for short inputs, and that evaluation should reflect these real usage patterns.","feed_headline":"Tetun users translate science, not news","feed_subtitle":"Analysis of 100,000 requests shows education-heavy usage that training corpora miss.","key_machinery":"The argument is carried by an observational pipeline that turns server logs into domain statistics. Topic and provenance labels are assigned by a prompt-based large language model (the paper uses Llama 3.1 8B) applied to the high-resource side of each translation pair, after LDA topic discovery helps define the label set; the same classifier is applied to the MADLAD-400 Tetun corpus after machine-translating it to English. Word-count weighting keeps short inputs from being underrepresented. A secondary mechanism is the perplexity comparison: a monolingual Tetun language model trained mostly on MADLAD gives average perplexity 1,774 on real MT inputs versus 153 on an unseen held-out corpus, and lower perplexity on the same news/government topics that dominate training data, tying the corpus mismatch to concrete modeling cost.","core_discovery":"The paper's central claim is a domain mismatch between actual translation demand and available training data, quantified on 100,000 server logs from March-August 2024. When weighted by word count, 'Science & research' appears in 34.2% of MT inputs but only 3.2% of the MADLAD-400 Tetun corpus, and 'Education & research material' appears in 55.9% of inputs versus 7.5% of MADLAD, while news articles and government topics dominate the corpus at 61.2% and 83.1% respectively. Users also translate into Tetun about 70% of the time and strongly prefer short texts, with a median input of 8 words. The paper interprets these numbers as evidence that the high-resource-to-low-resource direction and education-centric domains are what a low-resource MT service actually serves, and that current corpora misrepresent that demand.","pith_inferences":["The same observational approach could be applied to other institutionalized low-resource languages with a public translation service, providing a usage-grounded alternative to surveys and focus groups.","If the domain mismatch generalizes, it suggests that 'low-resource' corpus-building efforts in other settings may be systematically misaligned with actual community demand, and that cheap observational log analysis should precede corpus collection.","The perplexity numbers hint that a language model fine-tuned on user-like educational inputs could substantially reduce error on the real workload, a testable extension the paper does not run.","The paper's finding that users translate from Tetun mostly for daily-life and personal communication suggests that outward-facing communication is a separate use case requiring different optimization from the dominant inward direction."],"forward_implications":["MT evaluation for Tetun should be anchored in educational, science, and health domains rather than news and government, because those are what users actually translate.","The high-resource-to-Tetun direction should receive priority in model development and corpus collection, since it accounts for over 70% of requests.","Short inputs, including single words, are the typical case with a median of 8 words, so evaluation should include them rather than focusing on full sentences and paragraphs.","Perplexity gaps indicate that current corpora leave underrepresented domains poorly modeled, so corpus collection should intentionally cover education, science, and health material.","Mobile-first inference matters because over 90% of usage is on mobile devices, particularly Android."],"supporting_citations":[{"why":"Supplies the finding that domain similarity between training and test data is the largest driver of low-resource MT performance, motivating the paper's domain-mismatch analysis.","marker":"(Khiu et al., 2024)"},{"why":"Provides MADLAD-400, the large Tetun corpus used as the comparison baseline for topic, provenance, and genre distributions.","marker":"(Kudugunta et al., 2023)"},{"why":"Provides the Llama 3.1 8B model used for topic and provenance classification of the MT inputs and the comparison corpus.","marker":"(Dubey et al., 2024)"},{"why":"Supplies the Labadain Tetun corpus used in the perplexity comparison and evidence that most Tetun text on the internet comes from news articles.","marker":"(de Jesus and Nunes, 2024c)"},{"why":"Provides the Goldfish monolingual Tetun language model used to measure perplexity on MT inputs and corpora.","marker":"(Chang et al., 2024)"},{"why":"Supplies the LDA method used for initial topic discovery and for defining the topic label set.","marker":"(Blei et al., 2003)"},{"why":"Defines domain as a combination of topic, provenance, and genre, grounding the paper's classification scheme.","marker":"(Saunders, 2022)"},{"why":"Provides the survey framing that low-resource MT accuracy varies across domains and that such corpora often concentrate in religious texts.","marker":"(Haddow et al., 2022)"}],"fun_headline_variants":["100k Tetun requests reveal a science gap in training data","Tetun users ask for science, corpora give news","What do Tetun users translate? Not what you'd think","Observational study: Tetun translation demand is educational","Science and education dominate Tetun translation, not news"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim depends on the assumption that the classifier's errors are identical on user requests and on the machine-translated comparison corpus, since the classifier only reaches weighted F1 of 0.76 (topic) and 0.73 (provenance) and its error rates on the two pipelines were never compared.","fun_headline_variants_meta":{"raw":{"variants":["100k Tetun requests reveal a science gap in training data","Tetun users ask for science, corpora give news","What do Tetun users translate? Not what you'd think","Observational study: Tetun translation demand is educational","Science and education dominate Tetun translation, not news"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000842,"raw_usage":{"total_tokens":3656,"prompt_tokens":921,"completion_tokens":2735,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":2654}},"tokens_in":537,"tokens_out":2735,"duration_ms":20215,"temperature":1.0,"reasoning_tokens":2654,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:44:46.533401+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually annotate a random sample of, say, 500 user requests and 500 MADLAD documents with the same label set, compute the domain gap using the manual labels instead of the classifier's, and check whether the gap between user text and corpus narrows; if it drops below the reported levels, the paper's mismatch claim fails. A cheaper check is to re-run the classification pipeline feeding the Tetun side through a Tetun-capable classifier, if one is available, and compare the resulting distributions.","supporting_citations":[{"cited_title":"Seza Do g ru \\\"o z, and En-Shiun Lee","cited_arxiv_id":null,"evidence_quote":"Supplies the finding that domain similarity between training and test data is the largest driver of low-resource MT performance, motivating the paper's domain-mismatch analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides MADLAD-400, the large Tetun corpus used as the comparison baseline for topic, provenance, and genre distributions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines domain as a combination of topic, provenance, and genre, grounding the paper's classification scheme."}],"review_version":1}