{"id":"07e6456d-990f-4647-aa0e-c7447f4f61ab","arxiv_id":"2501.04927","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLMs frequently mistranslate large numeric units between Chinese and English, but a post-editing step with deterministic unit converters fixes most errors.","lead":"This paper tests how well open-source large language models translate numbers between Chinese and English, using a new dataset of ten numeric patterns from business text. It finds that errors are common, especially for large units like billion and 亿, and shows that a post-editing fix outperforms prompt-based methods.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reference lists in Table II omit admissible numerical renderings (word forms, '…之间' range phrases), so reported ~20% error rates may overstate actual numerical translation failures.","rationale":"The paper's entire quantitative claim is a set of pass rates. A pass rate is only as good as the reference list against which outputs are matched. The paper says the lists were manually generated to contain all acceptable renderings, but Table II itself contains counterexamples: Decimal ZH-EN reference for source '3.525' is just '[3.525]', excluding '三点五二五'; Large-unit ZH-EN reference for '1.43亿' omits word-form 'one hundred forty-three million'; Range EN-ZH reference for 'between 300 and 500' omits '在300到500之间'. If matching uses exact string comparison, and no normalization is described, each omission misclassifies a correct translation as an error. This directly inflates the 20% large-unit error rate that is the paper's central finding. The Decimal row's typo ('1.85' with reference '[2.5, 二点五]') further signals that manual label construction is not reliable enough to carry the conclusion. Other weaknesses (no released dataset, no variance estimates, PE's use of external deterministic tools) matter for reproducibility and interpretation, but they do not threaten the internal correctness of the error-rate numbers as directly as the reference lists do. If human re-annotation shows only a small pass-rate shift, the claim survives; if it shows a large shift, the headline needs substantial qualification. Therefore the conditional verdict remains appropriate: the paper should be accepted only with the dataset and label-generation protocol released and with the sensitivity check reported.","tokens_in":9218,"tokens_out":5684,"duration_ms":55308,"concrete_test":"Release the dataset and take a random sample of 100 large-unit outputs per direction (e.g., Llama3.1-8B outputs marked as failures). Have three independent bilingual annotators judge, without seeing the reference lists, whether each output correctly conveys the numerical value in natural English/Chinese, allowing digits, words, and standard phrasings. Compare the human-validated pass rate to the reported pass rate; if the gap exceeds 5 percentage points, report corrected rates and qualify the headline claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (LLMs make frequent numerical translation errors, ~20% on large units) rests entirely on the pass/fail labels in Table II. The paper states the references were 'manually generated' to include 'all possible translation results,' but the printed lists are visibly incomplete. For '3.525' (Decimal, ZH-EN) the only reference is the digit string, omitting the natural Chinese reading '三点五二五'; for '1.43亿' the EN reference omits the word form 'one hundred forty-three million'; for EN-ZH Range 'between 300 and 500' the list omits '在300到500之间' and other standard forms. If the matcher is exact string equality (the paper does not state any normalization), any model emitting such a valid rendering is counted as an error, directly overstating the error rates that motivate the paper's conclusions. The Table II row for Decimal also lists source '1.85' with reference '[2.5, 二点五]', a numeric mismatch suggesting the manual label construction is error-prone. Because the 20% figure and the 'common issue' conclusion are simple functions of these labels, an incomplete reference set is the most load-bearing vulnerability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript studies numerical translation between Chinese and English using a new dataset of ten numerical types. It evaluates nine open LLMs plus Google Translate, reporting pass rates per type and direction, and identifies large-unit translation (e.g., 亿, million, billion) as the hardest category. It then compares three mitigation strategies—in-context learning, chain-of-thought prompting, and an extraction-based post-editing pipeline that uses cn2an/en2an converters—and reports large gains for post-editing, with pass rates above 0.93 for large units. The paper's central claims are that numerical translation errors are common in open LLMs, that large units are the most error-prone, and that post-editing can largely fix them.","tokens_in":9441,"tokens_out":7803,"duration_ms":77204,"significance":"Conditional on the validity of the manual reference labels, the paper addresses an underexplored and practically important capability of LLM-based MT. Strengths include the use of real-business data, a ten-type taxonomy that goes beyond prior behavioral tests, the inclusion of several recent open LLMs, and a post-editing strategy that leans on deterministic external converters rather than fitted parameters. However, the benchmark's usefulness depends entirely on the completeness and correctness of the reference lists used for pass-rate computation, and the paper provides no confidence intervals, no decoding details, and no dataset/code release; the displayed reference lists already contain likely omissions and one numeric inconsistency. The headline error-rate claims are therefore plausible but not yet established at the reported precision.","major_comments":[{"comment":"Table II shows that the manually generated 'all possible translation results' are incomplete, which directly threatens the pass-rate numbers that motivate the paper. For example, the Decimal row for ZH-EN gives source '3.525' with reference '[3.525]' only, omitting the fully acceptable word-form rendering 'three point five two five'; the Large-unit row for ZH-EN gives '1.43 亿' with references that include '143 million' but omit 'one hundred forty-three million'; and the Range row for EN-ZH gives 'between 300 and 500' with references that omit the natural '在300到500之间'. Section III defines a pass as matching any reference-list item, and no normalization step is described, so exact-string matching against incomplete lists will systematically undercount valid translations. The Decimal row also lists source '1.85' with reference '[2.5, 二点五]', a numeric mismatch that indicates the manual label construction itself is error-prone. These issues must be fixed (or shown not to affect the reported error rates) before the ~20% large-unit error-rate claim can be accepted.","section":"Table II / §III"},{"comment":"Table III reports pass rates without any measure of uncertainty. Sample sizes differ widely across types (e.g., Formula has 19 items in EN-ZH, Number String has 1174 in ZH-EN), so point differences of a few percentage points in small cells are not interpretable. The paper also does not state the decoding configuration (temperature, number of runs, random seeds) used to generate the translations, which matters for small open models with sampling-based decoding. Please provide confidence intervals or bootstrap estimates and specify decoding settings.","section":"Table III"},{"comment":"The PE comparison in Table V is partially metric-incentivized, and the extraction step is not evaluated. PE replaces the detected numerical expression with a digit string, and the reference lists in Table II include digit strings; therefore, if the LLM extraction succeeds, the pass is achieved by construction, regardless of whether the replacement is stylistically or contextually natural. The paper should report extraction precision/recall on the large-unit test set and, ideally, a human or COMET-based acceptability check of the post-edited translations, to confirm the reported gains are not an artifact of the exact-match reference lists.","section":"§III-B, Table V"},{"comment":"The proposed dataset is the core artifact, but no public release link, annotation guidelines, or annotator-agreement statistics are provided. Since the central claims are functions of the manual labels, the dataset and the exact matching script must be made available so that the completeness of the reference lists can be audited; without this, the benchmark is not reproducible.","section":"§II"}],"minor_comments":[{"comment":"In the first paragraph, the clause 'such as a decimal point, an extra zero, or a missing zero' is syntactically incomplete; it should be attached to a noun such as 'errors'.","section":"Introduction"},{"comment":"In the ZH-EN block, after Tower-7B the label 'T-LLMs' is repeated, but the following rows (Mistral, Llama3.1, Qwen, GLM) are G-LLMs; rename the block heading.","section":"Table III"},{"comment":"The URL points to 'Cn2An.jl', which appears to be a Julia package, while the text says 'Python packages'; verify and correct the link, since the Python package is typically 'cn2an'.","section":"Footnote 1"},{"comment":"Reference [21] prints the author name as 'Dragos , Ciobanu' with an odd comma; fix this and make the citation formatting consistent across the reference list.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The recommendation is driven by the reference-label issue and the lack of uncertainty reporting. The paper's scope is modest but the topic is suitable for the readership. I would ask the authors to release the dataset and matching code during revision, and to audit the reference lists thoroughly, particularly the Decimal and Range categories. The overlap with reference [17] through authors Hao Yang and Min Zhang is not load-bearing for the paper's conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a legitimate extension of Wang et al.'s behavioral testing framework from NMT to LLMs, with a broader ten-category taxonomy and a new (though unreleased) Chinese–English dataset. The central finding—open-source LLMs, including Llama-3.1-8B, are unreliable at large-unit conversions like 亿/million/billion—is credible and consistent with the illustrative example in Table I. The post-editing strategy is also convincingly better than ICL or CoT across three weak base models and both translation directions, so the authors earn real credit there.\n\nThe stress-test note is on target. Table II's reference lists are visibly incomplete: '3.525' has only the digit form (no natural Chinese reading), the Range row omits standard phrasings like '在300到500之间', and the EN-ZH Decimal row lists source '1.85' with reference '[2.5, 二点五]'—a numeric mismatch that suggests the manual label construction was not carefully checked. The paper never specifies the matching procedure; if it is exact string equality, any valid alternative rendering would be counted as a failure, inflating the reported error rates. So the precise ~20% figure for large units is unreliable.\n\nThat said, the qualitative conclusion is robust: BLOOMZ at 11% on ZH-EN large units is so weak that relabeling a fraction of failures wouldn't change the picture, and the Table I example is a clear conversion error independent of any reference list. The main claim survives, but the exact rates should be treated as upper bounds.\n\nOther gaps: no confidence intervals, no decoding randomness details, no dataset release. The PE extraction step relies on the same LLM and is not analyzed for its own failure modes, though the external cn2an/en2an converters reduce the circularity. The overlapping author citation ([17]) is not load-bearing.\n\nThis paper is for researchers working on MT evaluation or numeracy in LLMs. It deserves a serious referee, not a desk reject. The authors should be asked to release the data, document normalization and matching, correct the reference lists, and add variance estimates. I'd bring it to our reading group as a cautionary example of how label incompleteness can quietly undermine otherwise plausible accuracy numbers.","headline":"A useful but sloppy empirical snapshot: large-unit numeracy errors are real, but the exact rates are unreliable because the reference lists are incomplete and unnormalized.","tokens_in":9945,"tokens_out":3731,"would_cite":true,"duration_ms":36085,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Open-source LLMs mistranslate large number units in Chinese–English translation about 20 percent of the time, and a post-editing step lifts accuracy to roughly 95 percent.","keywords":["LLM","Numerical Translation","Chinese-English translation","large units","unit conversion","post-editing","pass rate","machine translation evaluation"],"falsifier":"Have independent native speakers translate a random sample of the large-unit test sentences without seeing the reference lists, then check whether any accepted translations are absent from the paper's lists; if a substantial fraction of the sample yields valid renderings the lists omit, the reported error rates overstate how often LLMs actually get numbers wrong.","tokens_in":9050,"feed_emoji":"🔢","tokens_out":7210,"duration_ms":60462,"temperature":0.7,"pith_summary":"The paper sets out to test whether large language models can be trusted to translate numbers correctly between Chinese and English, a question prior work had left largely open. It builds a bilingual dataset from real business text covering ten numerical formats—large units, ranges, decimals, fractions, ratios, ordinals, and others—and manually lists every acceptable rendering of each number so that a translation counts as correct if it matches any of them. Across nine open-source general and translation-specialized LLMs, the paper finds that numerical errors are common, with large-unit conversions such as 亿 to 'billion' the worst category: even the strongest small model, Llama3.1-8B, errs roughly 20 percent of the time. The paper then shows that a post-editing strategy—extracting the translated number pairs, converting them with deterministic tools, and replacing the wrong ones—consistently outperforms in-context learning and chain-of-thought prompting, raising large-unit pass rates to about 95 percent. The stakes are practical: in finance and medicine, a single misplaced decimal point or zero changes the meaning of the text.","feed_headline":"LLMs botch big-number translation 20% of the time","feed_subtitle":"A post-editing step lifts large-unit accuracy to about 95%, the paper shows.","key_machinery":"The load-bearing machinery is the dataset plus the pass-rate evaluation protocol. The dataset organizes 10 numerical translation types, with each test sentence carrying a manually curated reference list of every admissible rendering (for example, '2.82 billion' accepted as 2820000000, 2,820,000,000, 28.2亿, 二十八点二亿, etc.), and the Pass Rate (PR) metric counts a translation as correct only if its numerical component matches at least one reference. Against this measuring stick, the paper pits nine open-source models. The second mechanism is the post-editing (PE) framework: an LLM extracts numerical translation pairs from its own output, Python packages cn2an/en2an convert both sides to digit form, mismatched pairs are flagged, and the wrong numeral is replaced with the correctly converted digit form. PE succeeds because extraction is a task LLMs can do reliably, while the arithmetic is handed to deterministic code.","core_discovery":"On the paper's own terms, the central claim is that current open-source LLMs do not reliably perform numerical translation, and that the failure is systematic rather than random: across all ten numerical types, no tested model is uniformly accurate, and the weakest type is large-unit conversion (e.g., rendering 1000亿 as 10 billion instead of 100 billion, or 134亿 as 3.4 billion instead of 13.4 billion). This weakness holds for both general-purpose LLMs and translation-specialized ones, and it is not simply a matter of model scale—the 70B version of Llama3.1 improves some categories but loses ground on ratios and formulas. The paper further claims that prompting strategies which ask the model to reason about units (in-context learning and chain-of-thought) yield only modest gains, whereas a post-editing pipeline that delegates the arithmetic to deterministic converters (cn2an and en2an) lifts large-unit pass rates to roughly 0.95 across all three weak base models tested. The authors attribute the core difficulty to LLMs' lack of a genuine unit-conversion capability and to floating-point imprecision in their internal arithmetic.","pith_inferences":["The post-editing success suggests a general division of labor for calculation-heavy translation: LLMs should extract and verbalize numbers while deterministic tools perform conversions; the same recipe could extend to fractions, ranges, and formulas, which also underperform.","The failure pattern likely carries over to other language pairs with similar unit systems, such as Japanese 万/億 or Korean 만/억, though the paper does not test them; a cross-lingual replication would be a direct test.","Because the dataset is drawn from real business data, aggregate pass rates may vary by domain; finance-heavy text with more large units would show lower accuracy than text dominated by easier types like decimals.","The paper's error examples (e.g., 'ranked 4th' becoming 'won bronze medal') suggest that numerical mistranslation is often bound up with world-knowledge reasoning, so fixes aimed purely at arithmetic may miss a second error class."],"forward_implications":["Large-unit numerical translation by open-source LLMs cannot be treated as reliable in production without a verification step, since error rates around 20 percent appear even in the newest 8B model.","The post-editing approach gives a model-agnostic recipe: let the LLM do the lexical work, then check every number with deterministic converters, which lifts pass rates to about 95 percent.","Evaluation of numerical translation should use exhaustive manual reference lists, because single-reference scoring would misclassify many valid renderings as errors.","Model size is not a reliable predictor of numerical accuracy; Llama3.1-70B beats the 8B on large units and fractions but falls behind on ratios and formulas.","The ten-type taxonomy from real business data can serve as a reusable benchmark for future numerical translation research."],"supporting_citations":[{"why":"Supplies the Pass Rate metric and the earlier finding that NMT systems err on numerical translation, which this paper extends to LLMs.","marker":"[3]"},{"why":"The Llama 3.1 model family is a central test subject; its 8B and 70B results drive the large-unit error-rate claims.","marker":"[9]"},{"why":"TowerInstruct-7B is the translation-specialized LLM benchmarked and compared across all ten types.","marker":"[10]"},{"why":"The Qwen2 models are benchmarked, including the 72B variant used as a strong open-source comparison.","marker":"[11]"},{"why":"Mistral-7B is both benchmarked and used as a weak base model in the post-editing experiments.","marker":"[12]"},{"why":"BLOOMZ-7B-MT is benchmarked and serves as another weak base model in the strategy comparison.","marker":"[13]"},{"why":"NLLB-200-3.3B is the supervised translation model included as a T-LLM baseline.","marker":"[14]"}],"fun_headline_variants":["LLMs falter on large-number translation, error hits 20%","Post-editing slashes LLM big-unit mistakes to 5%","Big-number mistakes plague LLMs; deterministic fix works","LLM numerical translation: 20% error on huge units"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the manually generated reference lists in Table II contain every admissible numerical rendering for each sentence, so any missing valid translation would be counted as a model error and push the reported pass rates down.","fun_headline_variants_meta":{"raw":{"variants":["LLMs falter on large-number translation, error hits 20%","Post-editing slashes LLM big-unit mistakes to 5%","Big-number mistakes plague LLMs; deterministic fix works","LLM numerical translation: 20% error on huge units"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000233,"raw_usage":{"total_tokens":1492,"prompt_tokens":945,"completion_tokens":547,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":474}},"tokens_in":561,"tokens_out":547,"duration_ms":5580,"temperature":1.0,"reasoning_tokens":474,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:21:14.898142+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent native speakers translate a random sample of the large-unit test sentences without seeing the reference lists, then check whether any accepted translations are absent from the paper's lists; if a substantial fraction of the sample yields valid renderings the lists omit, the reported error rates overstate how often LLMs actually get numbers wrong.","supporting_citations":[{"cited_title":"As easy as 1, 2, 3: Behavioural testing of NMT systems for numerical translation,","cited_arxiv_id":null,"evidence_quote":"Supplies the Pass Rate metric and the earlier finding that NMT systems err on numerical translation, which this paper extends to LLMs."},{"cited_title":"The llama 3 herd of models,","cited_arxiv_id":null,"evidence_quote":"The Llama 3.1 model family is a central test subject; its 8B and 70B results drive the large-unit error-rate claims."},{"cited_title":"Tower: An open multilingual large language model for translation-related tasks,","cited_arxiv_id":null,"evidence_quote":"TowerInstruct-7B is the translation-specialized LLM benchmarked and compared across all ten types."},{"cited_title":"Mistral 7b,","cited_arxiv_id":null,"evidence_quote":"Mistral-7B is both benchmarked and used as a weak base model in the post-editing experiments."},{"cited_title":"Scaling neural machine translation to 200 languages,","cited_arxiv_id":null,"evidence_quote":"NLLB-200-3.3B is the supervised translation model included as a T-LLM baseline."}],"review_version":1}