{"id":"89109c73-9903-4667-8b4d-a79acf22b367","arxiv_id":"2607.05849","paper_version":1,"verdict":"CONDITIONAL","confidence":"UNKNOWN","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":2,"one_line_summary":"A multi-stage pipeline that pivots Traditional Mongolian script through Cyrillic before translation improves MT quality across multiple backbones and target languages, and generates useful synthetic parallel data.","lead":"CoPiT is a translation pipeline that converts Mongolian Traditional Script text into the less ambiguous Cyrillic script before translating into English, Korean, or Russian. It consistently beats direct translation from the Traditional script and lets open-source models rival GPT-4.1 on this task.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Reference translations are derived from the Cyrillic side of parallel pairs, structurally favoring the Cyrillic-pivoting pipeline in reference-based evaluation.","rationale":"The reader identified small datasets, asymmetric fine-tuned-vs-zero-shot comparisons, and low inter-annotator agreement as concerns. These are valid but somewhat generic. The more specific and load-bearing issue is that the reference translations are derived from the Cyrillic side of parallel pairs, which structurally advantages the Cyrillic-pivoting pipeline in the metric that shows the largest gains. The reference-free evaluation partially addresses this, but the paper's own acknowledgment that COMET is unreliable for Traditional Mongolian undermines confidence in COMETKiwi as a cross-check. The human evaluation on 12 sentences is too small to be decisive.\n\nThis does not invalidate the paper's contribution—the pivot approach is well-motivated, the ablations are thorough, and the consistent direction of improvement across multiple settings is suggestive of a real effect. But the magnitude of the claimed gains, particularly the headline comparisons to GPT-4.1, rests on an evaluation design that may be biased toward the proposed method. The verdict should remain CONDITIONAL: the approach is promising and the work is valuable, but the strongest quantitative claims need either Traditional-derived references or a larger, more reliable reference-free evaluation before they can be fully accepted.\n\nThe reader's verdict of CONDITIONAL is appropriate. I agree with that placement but would add the reference-bias concern as a more precise articulation of why the quantitative evidence is not yet fully secure.","tokens_in":21003,"tokens_out":2583,"duration_ms":83409,"concrete_test":"Have bilingual speakers who can read Traditional Mongolian directly (without first converting to Cyrillic) produce fresh reference translations from the Traditional script side of the 1,031-sentence test set, independent of the existing Cyrillic-derived references. Recompute COMET and BLEU for both Direct and CoPiT (Fine-tuned) against these Traditional-derived references. If CoPiT's advantage over direct translation shrinks by more than ~30% relative (e.g., the English COMET gap for Ministral-3 14B narrows from +0.240 to below +0.168), the reference bias is materially affecting the headline claims.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that CoPiT consistently outperforms direct translation rests heavily on Table 1 (reference-based evaluation, 1,031 sentences). However, Section 4.1 states: 'English references are translated from the Cyrillic side and subsequently validated by bilingual speakers.' This means the gold-standard references encode a specific Cyrillic interpretation of each Traditional Mongolian source sentence. CoPiT explicitly routes Traditional→Cyrillic→Target, so its outputs are derived from the same Cyrillic representation that generated the references. Direct translation, by contrast, interprets the Traditional script independently and may produce semantically valid translations that differ from the Cyrillic-derived reference—particularly for the orthographically ambiguous forms the paper itself emphasizes as the core challenge. This creates a systematic evaluation bias: the metric measures proximity to a Cyrillic-grounded interpretation, which is exactly what CoPiT produces by construction.\n\nThe reference-free evaluation (Table 2, 380 news articles) partially mitigates this, as COMETKiwi scores source–hypothesis pairs without references. However, the paper itself acknowledges in Section 4.2.3 that 'semantic metrics such as COMET are unreliable for this language' when discussing Traditional Mongolian, which directly undermines confidence in COMETKiwi scores that take Traditional Mongolian as source input. The human evaluation covers only 12 sentences (Appendix A.2), with fluency inter-annotator agreement as low as α=0.263 for English. Thus each evaluation mode has a significant limitation: reference-based may be biased, reference-free may be unreliable for this script, and human evaluation is too small to be decisive.\n\nThe concern is load-bearing because the largest reported gaps appear in reference-based evaluation (e.g., Ministral-3 14B: COMET 0.707 vs 0.468), and these are the numbers used to support the headline claim that open-source models 'match or outperfo","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"The paper proposes CoPiT, a cognitively motivated pipeline for translating Mongolian text written in the Traditional script into English, Korean, and Russian. The core idea is to route translation through the Cyrillic script, which is more phonemically transparent and better-resourced. The Traditional-to-Cyrillic conversion is factorized into linguistically motivated steps: morphological segmentation, vowel harmony recovery, Latin-assisted normalization, Cyrillic normalization, and sentence-level self-reflection. Each component is independently fine-tuned using word-level lexical data (14,125 entries) and sentence-level revision pairs (2,061). Experiments span multiple backbone models (Qwen-3 4B/30B, Ministral-3 3B/14B, GPT-4.1) and three target languages, evaluated under reference-based, reference-free, and human evaluation protocols. The authors also demonstrate that CoPiT-generated synthetic parallel data (8,034 pairs) enables reverse-direction translation and improves forward translation. Datasets and code are released publicly.","tokens_in":21148,"tokens_out":1038,"duration_ms":101533,"significance":"The paper addresses a genuine and well-motivated problem: the digraphic resource imbalance in Mongolian NLP, where the Traditional script is both orthographically ambiguous and severely under-resourced. The cognitive motivation—mirroring how fluent readers map Traditional forms to Cyrillic—is a reasonable framing. The pipeline is linguistically grounded, with components targeting distinct sources of underspecification (vowel harmony, phonological normalization, sentence-level coherence). The ablation study (Table 3, Tables 9–11) is thorough, isolating both individual and pairwise component contributions. The release of a multi-script parallel corpus and all code is a concrete contribution to the community. The synthetic data generation loop and its validation through both forward and reverse translation experiments (Table 4, Figure 6) demonstrate practical utility beyond the inference-time pipeline.","major_comments":[{"comment":"§4.1, paragraph on Datasets: The reference translations for the reference-based evaluation set (1,031 sentences) are derived from the Cyrillic side of parallel pairs—'English references are translated from the Cyrillic side and subsequently validated by bilingual speakers.' Since CoPiT explicitly routes Traditional→Cyrillic→Target, its outputs are derived from the same Cyrillic representation that generated the references. Direct translation, by contrast, interprets the Traditional script independently and may produce semantically valid translations that diverge from the Cyrillic-grounded reference, particularly for the orthographically ambiguous forms the paper identifies as the core challenge. This creates a systematic structural bias in reference-based evaluation (Table 1) that favors CoPiT by construction. The paper should explicitly acknowledge this bias and clarify that the COMET/B","section":null}],"minor_comments":[{"comment":"Table 1: The BLEU-3/4 column header is unusual; standard practice reports BLEU-4 (or BLEU). Clarify what BLEU-3/4 means—is it n-gram order 3 and 4 reported separately?","section":null},{"comment":"Table 2: The GPT-4.1 row for Russian appears to have a formatting issue where the COMETKiwi value (0.429) runs into the Adeq. column.","section":null},{"comment":"§4.2.2: The ablation discussion notes that removing Vowel Harmony Recovery sometimes yields higher COMET (e.g., Qwen-3 4B English: 0.633 without VHR vs. 0.628 with). The paper attributes this to backbone-dependent behavior, but the interaction is not analyzed further. A brief discussion of why VHR can hurt would strengthen the analysis.","section":null},{"comment":"Appendix A.2, Table 5: Fluency inter-annotator agreement for English is α=0.263, which is below the conventional threshold for reliable annotation. The paper discusses this but could note more explicitly that fluency conclusions for English should be treated with caution.","section":null},{"comment":"§3.2, Morphological Segmentation: The suffix dictionary is mentioned but its size and coverage are not specified. Providing the number of suffixes would help readers assess the generality of this approach.","section":null},{"comment":"Figure 2: The fire emoji symbols are unconventional for a system architecture diagram. Consider replacing with standard notation.","section":null},{"comment":"References: The citation 'Tumur-Ochir et al.' (in §2.2) is missing a year in the reference list.","section":null}],"recommendation":"major_revision","confidential_remarks":"The structural evaluation bias concern is the most important issue. It is not necessarily fatal—the reference-free results (Table 2) partially address it—but the paper currently over-claims based on Table 1 without acknowledging the bias. If the authors can (a) explicitly discuss the bias, (b) emphasize reference-free and human evaluation as the more reliable evidence, and (c) provide the few-shot GPT-4.1 comparison, the paper should be publishable. The synthetic data contribution is solid and does not depend on the biased evaluation."},"author_rebuttal":{"model":"glm-5.2","summary":"The referee raises a valid concern about structural bias in reference-based evaluation: because reference translations are derived from the Cyrillic side of parallel pairs, and CoPiT routes through Cyrillic, CoPiT's outputs may be systematically favored over direct translation outputs that could be semantically valid but diverge from the Cyrillic-grounded reference. We acknowledge this bias and will revise the manuscript accordingly, while also noting that our reference-free evaluation (Table 2) and human evaluation provide independent evidence that is not subject to this concern.","responses":[{"response":"The referee is correct that there is a structural affinity between CoPiT's intermediate representation and the reference translations, since both are derived from the Cyrillic side of the parallel pairs. We acknowledge this bias and will explicitly state it in the revised manuscript, adding a caveat to the description of the reference-based evaluation set in §4.1 and a note in the discussion of Table 1. Specifically, we will add language clarifying that because English references are translated from the Cyrillic side, CoPiT's Cyrillic pivoting may align more closely with reference translations by construction, and that direct translation outputs that are semantically valid but diverge from the Cyrillic-grounded reading would be penalized under reference-based metrics. We will recommend that readers interpret Table 1 results in conjunction with the reference-free evaluation (Table 2), which uses COMETKiwi and human adequacy/fluency ratings and does not rely on Cyrillic-derived references. The reference-free results show consistent improvements for CoPiT across all backbones and target languages, providing evidence that the gains are not solely an artifact of the evaluation bias. That said, we agree the bias should be transparently acknowledged, and the manuscript will be revised accordingly.","revision_made":"yes","referee_comment":"§4.1, Datasets: The reference translations for the reference-based evaluation set (1,031 sentences) are derived from the Cyrillic side of parallel pairs. Since CoPiT explicitly routes Traditional→Cyrillic→Target, its outputs are derived from the same Cyrillic representation that generated the references. Direct translation interprets the Traditional script independently and may produce semantically valid translations that diverge from the Cyrillic-grounded reference, particularly for orthographically ambiguous forms. This creates a systematic structural bias in reference-based evaluation (Table 1) that favors CoPiT by construction. The paper should explicitly acknowledge this bias and clarify that the COMET/BLEU scores in Table 1 should be interpreted with this caveat."}],"tokens_in":20600,"tokens_out":524,"duration_ms":60829,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"Two things to know upfront: CoPiT is a genuinely useful pipeline for low-resource Mongolian MT that earns its keep through linguistic insight and released data, and the stress-test concern about reference bias is partially valid but doesn't sink the paper — the reference-free results carry enough weight on their own.","headline":"CoPiT is a solid, well-engineered pipeline for Traditional Mongolian MT with a real linguistic contribution, but the headline comparison to GPT-4.1 is apples-to-oranges and the reference-based evaluation has a structural bias the paper doesn't acknowledge.","tokens_in":21824,"tokens_out":170,"would_cite":true,"duration_ms":33250,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Pivoting through Cyrillic lifts Traditional Mongolian translation","keywords":[],"falsifier":"If the Traditional-to-Cyrillic conversion step introduced systematic errors that the downstream translation could not recover from, the pivot would degrade rather than improve translation quality. A direct test would measure conversion error rates and correlate them with end-to-end translation quality degradation.","tokens_in":21015,"feed_emoji":"🔤","tokens_out":1089,"duration_ms":48220,"temperature":0.7,"pith_summary":"The paper proposes CoPiT, a translation pipeline that exploits an internal resource hierarchy within Mongolian itself. Mongolian is digraphic: the Cyrillic script is comparatively well-resourced and phonemically transparent, while the Traditional script is data-scarce and orthographically ambiguous — a single written form can correspond to multiple plausible interpretations. Rather than translating directly from the Traditional script, CoPiT first converts Traditional Mongolian into Cyrillic through a structured multi-step disambiguation process — vowel harmony recovery, Latin-assisted phonological normalization, Cyrillic normalization, and sentence-level self-reflection — and then translates from the disambiguated Cyrillic into the target language. This decomposition isolates script-level ambiguity from semantic transfer, mirroring how fluent Mongolian readers reportedly process the Traditional script by implicitly mapping it to Cyrillic. The authors show that CoPiT consistently outperforms direct translation across multiple backbone models (Qwen-3 4B/30B, Ministral-3 3B/14B, GPT-4.1) and target languages (English, Korean, Russian), with fine-tuned open-source models matching or exceeding GPT-4.1's zero-shot direct-translation performance. The pipeline also generates synthetic parallel data from real Traditional-script sources, enabling both forward and reverse-direction translation in a setting where parallel corpora are virtually nonexistent.","feed_headline":"Pivoting through Cyrillic lifts Traditional Mongolian translation","feed_subtitle":"A multi-stage pipeline that converts the ambiguous Traditional script to better-resourced Cyrillic before translation lets small open models","key_machinery":"CoPiT pipeline: morphological segmentation → vowel harmony recovery → Latin-assisted normalization → Cyrillic normalization → sentence reconstruction with self-reflection → Cyrillic-to-target translation. Trained component-wise on 14,125 word-level lexical pairs and 2,061 sentence-level revision pairs, requiring no sentence-level Traditional-to-target parallel data.","core_discovery":"The central mechanism is script-level pivoting: routing translation through a better-resourced orthographic representation of the same language to resolve ambiguity before semantic transfer. The Traditional-to-Cyrillic conversion is factorized into linguistically motivated sub-steps — vowel harmony recovery narrows phonological interpretations, Latin-assisted normalization makes implicit phonological distinctions explicit, Cyrillic normalization produces a canonical intermediate form, and self-reflection enforces sentence-level coherence. The ablation shows that self-reflection is the most load-bearing component: removing it causes the largest performance drop (e.g., English COMET falls from","pith_inferences":[],"forward_implications":["Script-level pivoting could generalize to other digraphic languages where one script is better-resourced than another (e.g., languages with both Latin and non-Latin orthographies, or classical/modern script pairs).","The synthetic data generation loop — using the pipeline to create parallel corpora from monolingual Traditional-script sources — offers a self-bootstrapping path for languages where parallel data collection is prohibitively expensive.","The finding that self-reflection is the most critical component suggests that global sentence-level coherence, not local disambiguation, is the bottleneck in translating from orthographically ambiguous scripts.","Fine-tuned open-source models matching GPT-4.1 suggests that structured linguistic decomposition can compensate for raw model scale in low-resource settings."],"fun_headline_variants":["Routing Mongolian translation through Cyrillic closes the resource gap","Script pivoting lets small models match GPT-4 on Traditional Mongolian","Factorized Traditional-to-Cyrillic conversion stabilizes Mongolian translation","Self-reflection step drives gains in low-resource Mongolian translation","Open models match GPT-4.1 on Traditional Mongolian via script pivoting"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The pipeline depends on the Traditional-to-Cyrillic conversion being accurate enough that conversion errors do not propagate into downstream translation, and the evaluation rests on small datasets (1,031 sentences for reference-based evaluation, 12 sentences for human evaluation) with sometimes low inter-annotator agreement. The comparison that open-source models 'match or outperform GPT-4.1' pits fine-tuned open-source models against zero-shot GPT-4.1, an asymmetry in how","fun_headline_variants_meta":{"raw":{"variants":["Routing Mongolian translation through Cyrillic closes the resource gap","Script pivoting lets small models match GPT-4 on Traditional Mongolian","Factorized Traditional-to-Cyrillic conversion stabilizes Mongolian translation","Self-reflection step drives gains in low-resource Mongolian translation","Open models match GPT-4.1 on Traditional Mongolian via script pivoting","Cyrillic pivot lifts BLEU and COMET for Traditional Mongolian translation","Disambiguating Traditional script before translation yields stable meaning transfer","CoPiT pipeline turns Cyrillic abundance into Traditional-script gains","Script-level pivot outperforms direct translation for data-scarce Mongolian","Multi-stage Cyrillic routing enables synthetic parallel data from Traditional text"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":779,"prompt_tokens":552,"completion_tokens":227,"prompt_tokens_details":null},"tokens_in":552,"tokens_out":227,"duration_ms":13421,"temperature":1.0,"reasoning_tokens":82,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T22:32:12.338072+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If the Traditional-to-Cyrillic conversion step introduced systematic errors that the downstream translation could not recover from, the pivot would degrade rather than improve translation quality. A direct test would measure conversion error rates and correlate them with end-to-end translation quality degradation.","supporting_citations":[],"review_version":1}