{"id":"64ed73f9-04fe-4f79-9ec5-5d6f99046daa","arxiv_id":"2412.00218","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A 35-example few-shot GPT-4-Turbo pipeline reached 48.69% exact-match translation accuracy on held-out Nushu sentences and produced a 98-sentence silver corpus, alongside the first public Nushu-Chinese dataset.","lead":"This paper introduces NushuRescue, a pipeline that uses a commercial large language model to translate modern Chinese sentences into the endangered Nushu script after seeing only 35 examples. It also releases NCGold, a 500-sentence Nushu-Chinese parallel corpus, as the first public dataset for this script.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"NCGold lacks native-expert validation, so the 48.69% headline is measured against an unverified reference; the reported percentage also cannot be an exact count on 50 sentences.","rationale":"I agree with the reader's weakest assumption: NCGold's accuracy is the load-bearing premise. The paper's own limitations concede that no expert validation was performed on the out-of-dictionary mappings, and the annotators are not Nüshu experts. Since every headline number is computed against NCGold, a corrupted reference would invalidate the central claim. I do not escalate to REJECT because the paper is transparent about many of these limitations, the framework and dataset are real artifacts, and the concern can be resolved by an external validation study. I add one further observation not highlighted by the reader: 48.69% of 50 sentences cannot be an integer count under a strict perfect-match metric, and the manual correction step makes the protocol ambiguous. This reinforces CONDITIONAL rather than ACCEPT, but does not change the verdict because the primary fix—independent gold-standard validation and raw-count reporting—is the same. Credit is due for releasing the dataset, documenting prompts, and reporting negative results for longer sentences; those are independent strengths.","tokens_in":11098,"tokens_out":5323,"duration_ms":48091,"concrete_test":"Have at least two independent Nüshu scholars validate a stratified random sample of 100 NCGold sentence pairs, oversampling the 373 sentences whose mappings were manually matched outside the official dictionary. For each pair, compare the Nüshu transcription against the scanned Compendium source and the official Unicode dictionary, and record character-level agreement. Then recompute the headline accuracy on the 50 withheld examples using only sentence pairs that pass expert validation, and ask the authors to provide raw per-sentence outputs and exact counts to verify whether 48.69% corresponds to an integer count. If the expert-validated subset shows >5% reference error, or if the corrected accuracy drops below the reported value, the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that GPT-4-Turbo reaches 48.69% strict character-level accuracy on 50 withheld NCGold sentences from 35 seeds—assumes NCGold itself is correct. That assumption is insecure. Section 4.1 says the annotators are bilingual with computational linguistics experience, not Nüshu experts, and 373/500 sentences required manual visual matching outside the official dictionary. Section 4.1 calls NCGold 'expert-validated,' but the Limitations section concedes that the absence of expert validation prevents releasing generated out-of-dictionary data. If those 373 manual mappings contain systematic errors, the gold reference is corrupted and the headline accuracy, plus all downstream NCSilver and Seq2Seq results, is measured against a flawed standard. A second, independent problem: 48.69% of exactly 50 sentences is 24.345 sentences. Under a strict sentence-level perfect-match count this is impossible; either the metric is aggregated differently, the evaluation set is not exactly 50, or the number is misreported. The paper also says human annotators manually reviewed and 'correcting minor character choice errors,' which blurs whether 48.69% is the model's raw output or output after human correction. Without raw per-round counts, the central quantitative claim is not reproducible as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces NüshuRescue, an LLM-in-the-loop framework for scaling up corpora of endangered languages with minimal human effort, and contributes two new resources: NCGold, a 500-sentence Nüshu–Chinese parallel corpus, and NCSilver, 98 LLM-generated Nüshu–Chinese sentence pairs. The central empirical claim is that GPT-4-Turbo, given only 35 seed pairs from NCGold and a character-count length validator, achieves 48.69% strict character-level translation accuracy on 50 withheld NCGold sentences. The paper also trains a FastText language model and a Seq2Seq Nüshu-to-Chinese translation model on the resulting corpora, reporting that translation metrics improve as the training set grows. The authors include a Limitations section that acknowledges the absence of expert validation and explains why out-of-dictionary generated data was not released.","tokens_in":11400,"tokens_out":5389,"duration_ms":49176,"significance":"If the reported numbers hold, the paper provides a valuable public dataset for a truly endangered language, a simple and reproducible data-augmentation pipeline, and transfer evidence on Cherokee, all of which would be useful for the low-resource NLP community. The release of code and data, the documentation of prompts, and the unusually candid Limitations section are strengths. However, the headline accuracy is measured against a gold standard whose expert validation is asserted in some places and denied in others, and the reported percentage is inconsistent with the stated evaluation size. The significance is therefore conditional: the resource and framework are promising, but the central quantitative claim needs to be either corrected or substantially re-framed before the paper can be accepted.","major_comments":[{"comment":"The abstract and §4.2.3 report 48.69% accuracy on 50 withheld sentences under a strict sentence-level character-by-character match. Since 48.69% of 50 is 24.345, this number cannot be a count of perfectly translated sentences. Either the metric is an average over a different unit (for example, per-character accuracy), the evaluation set does not actually contain 50 sentences, or the number is misreported. Please give the raw counts and define the metric unambiguously; the headline result must be reproducible from the reported numbers.","section":"Abstract; §4.2.3"},{"comment":"The manuscript repeatedly calls NCGold 'expert-validated' (Abstract, §1, §4.1), but §4.1 states that the annotators are bilingual with computational linguistics experience, not Nüshu experts, and that 373 of the 500 sentences required manual visual matching outside the official dictionary. The Limitations section then states that 'the absence of expert validation prevents us from safely releasing the generated data.' These statements are in direct tension. If the 373 manually mapped sentences contain systematic character-selection errors, the 48.69% accuracy, the NCSilver examples, and all Seq2Seq evaluations are measured against a flawed reference. Please state clearly which portion of NCGold is expert-validated, describe the validation procedure actually used, and report agreement or an error audit.","section":"§4.1; Limitations"},{"comment":"The evaluation protocol is ambiguous about whether the reported accuracy is computed on raw GPT-4-Turbo outputs or on outputs after the authors' manual review that 'corrected minor character choice errors.' This distinction is load-bearing because the central claim is about the model's capability, not about human post-editing. Please report raw and post-correction accuracy separately, and specify how disagreements were adjudicated.","section":"§4.2.1; §4.2.3"},{"comment":"The length validator is described as ensuring that 'each Nüshu character maps to a single Chinese character,' yet §3 explicitly describes one-to-many and many-to-one mappings between Nüshu and Chinese, with one Nüshu word mapping to up to 35 Chinese words. The equal-length constraint is therefore not generally guaranteed for correct translations. Using this validator to filter generations could bias the evaluation by rejecting valid translations or accepting incorrect ones solely on length. Please justify the one-to-one length assumption on the actual NCGold data and quantify how many seed and test sentences satisfy it.","section":"§3; §4.2.2"},{"comment":"The number of seed examples (35) and the length-validation rule are said to be informed by 'preliminary tests,' but the paper does not state whether those tests involved the 50 withheld sentences or otherwise influenced design choices on the evaluation set. If any tuning used the evaluation set, the 48.69% figure is not a clean held-out estimate. In addition, a 50-sentence test set at roughly 49% accuracy has a binomial standard error of about 7 percentage points; please report bootstrap or exact binomial confidence intervals and state the precise train/test split.","section":"§4.2.1; §4.2.2"}],"minor_comments":[{"comment":"The spelling of the language name is inconsistent: the title and most sections use 'Nüshu,' but the abstract and several places use 'Nushu.' Please standardize.","section":"Throughout"},{"comment":"The stacked bar chart in Figure 10 is difficult to read because the count labels are small and partially overlapping; please add numeric labels in a table or use a clearer chart format so that the per-round counts and control-group counts can be verified.","section":"§4.2.2; Figure 10"},{"comment":"Table 2 does not state whether the fixed 100-sentence test set is drawn from NCGold only or from the combined NCGold+NCSilver corpus, nor how the test set was selected after shuffling. Please clarify this in the text and table caption.","section":"§5.2; Table 2"},{"comment":"The claim that 'the maximum sentence length in the original 500-sentence dataset is only 18' needs verification and a supporting length distribution, especially since Table 1 reports average lengths only for the generated rounds and the original NCGold corpus is described as containing poetry and prose of varying lengths.","section":"§6"},{"comment":"The statement that GPT-4-Turbo successfully translated '70.00% of sentences with lengths below 16 characters' should include the raw numerator and denominator, as well as the exact length bin definition, so that the reader can assess the reliability of this secondary result.","section":"§4.2.3"},{"comment":"The factual claim that the last known native speaker of Nüshu passed in 2004 is given without a citation; please provide a reference for this important contextual statement.","section":"Limitations; footnote 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is best viewed as a promising pilot and dataset-resource contribution rather than as a benchmark result. The authors' Limitations section is unusually candid, which suggests the framing problems are fixable, but the abstract and introduction currently overstate both the expert validation of NCGold and the precision of the headline accuracy. I would encourage the editor to ask for the raw per-round counts, a correction of the 48.69% arithmetic, a clearer separation between raw and post-edited outputs, and a more careful distinction between expert-validated and manually matched portions of the dataset. If the authors recalibrate their claims, the paper could make a useful contribution to low-resource and endangered-language NLP."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the good news: NCGold and NCSilver are genuinely new and useful artifacts. There was no public Nushu-Chinese parallel corpus before, and this one comes with code and a documented pipeline. The few-shot prompting setup with the length validator is a sensible adaptation of in-context learning to a nearly-zero-resource script, and the Cherokee experiments are a nice sanity check that the approach transfers. The paper also deserves credit for being upfront about the biggest limitation: without a native-expert check, they won't release out-of-dictionary generations.\n\nBut the headline number does not survive contact with arithmetic. 48.69% accuracy on exactly 50 withheld sentences means 24.345 perfect matches. That is impossible if the metric is a sentence-level perfect-match count, which is exactly how Section 4.2.3 describes it. Either the metric is actually a character-level average, the test set is not 50, or the number is misreported. On top of that, the paper says the authors manually reviewed and 'corrected minor character choice errors' before reporting that figure, so it is unclear whether 48.69% is the raw model output or output after human correction. As stated, the central quantitative claim is not reproducible.\n\nThe second soft spot is the gold standard itself. The annotators are bilingual and computationally literate, but they are not Nushu experts, and 373 of 500 sentences required visual matching outside the official dictionary. The introduction and Section 4.1 call NCGold 'expert-validated,' but the Limitations section says the lack of expert validation is exactly why generated data could not be released. That is an internal contradiction, and it matters because every downstream number—including the 48.69%—is measured against this reference. If those 373 manual mappings contain systematic errors, the headline is measured against a corrupted standard.\n\nTo be clear, I am not saying the framework is a failure. The qualitative result—that an off-the-shelf LLM can learn a syllabic script from 35 examples—is plausible and worth testing properly. The dataset alone is a contribution. But the paper as written overstates certainty: the abstract and conclusion call the translations 'high-quality' and 'expert-validated' without support. The fix is straightforward: report the exact evaluation protocol (sentence-level vs. character-level), give per-round counts and confidence intervals, clarify what the human correction step did, and either get partial expert validation for NCGold or soften the 'expert-validated' language.\n\nThis is a legitimate paper for a serious venue, but it needs a major revision focused on the evaluation. I would send it to peer review with a clear request to resolve the arithmetic and the gold-standard validation. The stress-test note is right on both counts.","headline":"The dataset is a real contribution, but the headline 48.69% accuracy cannot be a perfect-match count on 50 sentences, so the central number needs checking before this is accepted.","tokens_in":11904,"tokens_out":2849,"would_cite":true,"duration_ms":25465,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM with no prior exposure to Nüshu learned it from 35 examples and translated 48.69% of held-out sentences exactly.","keywords":["Nüshu","endangered language preservation","low-resource NLP","few-shot machine translation","large language models","parallel corpus construction","data augmentation","women's script"],"falsifier":"Re-run the 35-example few-shot prompt with the Nüshu symbols in the seed pairs randomly permuted, keeping the Chinese sides fixed, then score the 50 withheld sentences on exact character match; if accuracy stays near 48.69%, the result is carried by the length validator and restricted vocabulary rather than learned Nüshu-to-Chinese mappings, while a collapse toward chance would support genuine script acquisition.","tokens_in":10876,"feed_emoji":"📜","tokens_out":8486,"duration_ms":76002,"temperature":0.7,"pith_summary":"This paper claims that a general-purpose large language model with no prior knowledge of Nüshu can learn to translate into this endangered syllabic script from just 35 example sentences, reaching 48.69% character-exact accuracy on 50 held-out sentences. That result matters because Nüshu has almost no digital resources: the only expert-validated source is a print compendium, and manual transcription is slow and costly. The paper's NüshuRescue framework pairs few-shot prompting with an automated length validator and iterative corpus expansion, producing a 500-sentence gold corpus (NCGold) and a 98-sentence machine-generated extension (NCSilver). If the claim holds, AI-assisted data generation can lower the human cost of revitalizing other low-resource languages.","feed_headline":"35 seed sentences let GPT-4 translate Nüshu at 48.69 percent","feed_subtitle":"With 35 translation pairs, an off-the-shelf LLM starts reviving a lost women's script and grows its corpus.","key_machinery":"The load-bearing mechanism is the NüshuRescue few-shot generation loop: the LLM receives uploaded dictionary data and 35 seed Nüshu–Chinese pairs, then translates new Chinese sentences, while a rule-based validator rejects any output whose Nüshu character count does not match the Chinese source and prompts up to seven retries. Across six rounds, the top five newly generated translations replace the bottom five seed examples, gradually extending to longer sentences. The NCGold corpus—500 manually transcribed pairs from a printed Nüshu compendium, including 373 sentences that required manual visual mapping beyond the official dictionary—supplies the seed data and the evaluation set that anchors the claimed accuracy.","core_discovery":"The central discovery is that the knowledge needed to translate into a truly low-resource script can be supplied almost entirely at inference time: GPT-4-Turbo, shown to be unable to recognize Nüshu before prompting, achieved 48.69% exact character-match accuracy on 50 withheld NCGold sentences after seeing only 35 seed pairs plus dictionary files. The paper attributes most errors to Nüshu's one-to-many and many-to-one character mappings, where multiple Chinese homophones share a syllable. Enforcing a one-to-one character-length rule raised accuracy from 31.37% to 48.69%, showing that language-specific structural constraints can partly compensate for missing data. The same pipeline generated 98 new Chinese-to-Nüshu translations (NCSilver) and, applied to Cherokee with larger seed sets, produced 28%, 40%, and 42% exact accuracy across seed sizes of 35, 150, and 300, suggesting the method transfers beyond Nüshu.","pith_inferences":["The paper leaves an expert validation pass as future work; if such a pass corrected even a modest share of the 50 withheld NCGold sentences, the 48.69% figure would need recomputation, so the headline number is only as solid as the annotators' manual visual matching.","Because the last native Nüshu speaker died in 2004, any machine-generated revival text enters circulation without native-speaker checks; a deployment pipeline would need community or expert sign-off before out-of-dictionary generated forms are used pedagogically.","The no-prior-exposure property suggests the pipeline could be pointed at orally attested but unwritten languages, shifting the bottleneck from data volume to trusted seed annotation by community partners."],"forward_implications":["Human effort for endangered-language corpus building drops from full manual transcription to producing a few dozen validated seed pairs.","The resulting NCGold and NCSilver datasets give the research community the first public Nüshu parallel corpus, covering 80% of the only expert-validated Nüshu source.","Because the LLM needs no prior exposure to the target script, the framework can be pointed at other undocumented or underdocumented languages immediately, limited mainly by seed data.","Enforcing simple script-specific constraints, such as the one-to-one character-length rule, can substitute for large training sets in low-resource translation.","The Seq2Seq experiments imply that LLM-generated data continues to improve downstream translation models, up to a length and complexity threshold beyond which performance declines."],"supporting_citations":[{"why":"Supplies the only expert-validated Nüshu–Chinese translations from which NCGold is manually transcribed.","marker":"Zhao, 1992"},{"why":"Provides the online character-mapping dictionary used to generate candidate Nüshu words for each Chinese word.","marker":"Converter"},{"why":"Provides the official Nüshu–Chinese Unicode dictionary used to filter in-dictionary sentences and validate mappings.","marker":"Contributors"},{"why":"Provides the Chinese sentences used to construct the out-of-domain NCSilver translations.","marker":"Wang et al., 2021"},{"why":"Supplies the Cherokee–English corpus used to test the framework's transfer with seed sizes of 35, 150, and 300.","marker":"Zhang et al., 2020"}],"fun_headline_variants":["48.69% Nushu accuracy from 35 seeds, plus a length rule","GPT-4 revives Nüshu script with just 35 sentence pairs","A length rule boosts GPT-4 Nüshu translation to 48.69%","Cherokee also improves: 300 seeds give 42% translation","NushuRescue: LLMs breathe life into lost women's script"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on NCGold being accurate: its annotators were bilingual and computationally trained but not Nüshu experts, and for 373 of 500 sentences they had to visually match characters outside the official dictionary, so a systematic mapping error would make the claimed accuracy a comparison against a flawed benchmark.","fun_headline_variants_meta":{"raw":{"variants":["48.69% Nushu accuracy from 35 seeds, plus a length rule","GPT-4 revives Nüshu script with just 35 sentence pairs","A length rule boosts GPT-4 Nüshu translation to 48.69%","Cherokee also improves: 300 seeds give 42% translation","NushuRescue: LLMs breathe life into lost women's script"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000689,"raw_usage":{"total_tokens":3159,"prompt_tokens":1019,"completion_tokens":2140,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":2045}},"tokens_in":635,"tokens_out":2140,"duration_ms":13332,"temperature":1.0,"reasoning_tokens":2045,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:36:29.276942+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 35-example few-shot prompt with the Nüshu symbols in the seed pairs randomly permuted, keeping the Chinese sides fixed, then score the 50 withheld sentences on exact character match; if accuracy stays near 48.69%, the result is carried by the length validator and restricted vocabulary rather than learned Nüshu-to-Chinese mappings, while a collapse toward chance would support genuine script acquisition.","supporting_citations":[],"review_version":1}