{"id":"3f2bebb4-5ed0-486b-ab48-225f9fb68eb9","arxiv_id":"2501.07721","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"LLMic, a 3B Romanian-English model with a diacritic-free tokenizer, reports higher WMT English-to-Romanian translation scores than larger open models, though the evaluation setup is biased.","lead":"The authors trained a 3-billion-parameter AI model, LLMic, on Romanian and English text and then fine-tuned it for translating English into Romanian. On the WMT benchmark, it scored higher than several much larger open models, but the comparison used a modified test that may favor LLMic.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The WMT evaluation in §6 is modified to strip diacritics and lowercase, exactly matching LLMic's tokenizer; the headline claim of outperforming existing solutions for English-to-Romanian translation is therefore not established for standard Romanian.","rationale":"The paper is a model-development report, and its central quantitative claim is the translation result. The least secure condition needed to support that claim is the validity of the WMT evaluation, and §6 reveals a load-bearing modification: the test data are stripped of diacritics and lowercased, precisely matching LLMic's intentionally uncased, diacritic-free tokenizer. This changes the task from standard English-to-Romanian translation to a normalized-form translation task. The concern is not merely a stylistic preference: Romanian diacritics distinguish words, and a model that cannot produce them cannot produce standard Romanian text. The reported gap between LLMic and the baselines therefore does not establish superiority for the usual meaning of the abstract's claim. The paper may still offer useful documentation of a specialized pretraining pipeline, but the headline comparison is unsupported absent an evaluation on the unmodified benchmark. No additional concerns are needed; this single issue is sufficient to keep the reader's REJECT verdict.","tokens_in":7105,"tokens_out":6371,"duration_ms":67459,"concrete_test":"Rerun the fine-tuned LLMic and the Table 3b baselines on the original WMT en-ro test set with diacritics and case preserved, scoring with standard sacreBLEU and no normalization of references or candidates. If LLMic cannot emit diacritics or its score drops below the reported baselines, the claimed advantage is an artifact of the test-set modification. As a matched control, fine-tune a comparable baseline (e.g., mBART or a 3B Llama-style model) on the same parallel data and evaluate under both normalized and unnormalized conditions; if the relative ranking flips or the gap shrinks, the headline result is benchmark-dependent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that fine-tuning LLMic for translation outperforms existing solutions in English-to-Romanian translation. In §6 the authors state: 'we modified the WMT data by removing diacritics and converting to lowercase, while leaving everything else unchanged.' This is not a neutral preprocessing step. Section 3 says the tokenizer was deliberately built as uncased and 'intentionally omits the Romanian characters: ă, â, î, ș, ț, replacing them with a, i, s, t respectively.' Romanian diacritics are semantically load-bearing; for example, 'vesela' versus 'veselă' and 'cina' versus 'cină' are different words that become identical after stripping diacritics. A translation that omits diacritics and case is not standard Romanian, and BLEU computed on normalized text cannot distinguish correct inflected forms from incorrect homographs. The reported scores, LLMic 41.01 vs mBART 38.50 and vs 7B-8B instruct models in the 25-29 range, therefore validate LLMic only on the modified, diacritic-free task. The abstract's unqualified claim of outperforming existing solutions in English-to-Romanian translation is not supported without evidence that the advantage survives on the unmodified WMT benchmark with standard scoring.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper describes the pretraining, tokenization, and fine-tuning of LLMic, a 3B-parameter bilingual Romanian-English decoder-only transformer. The authors construct a 300B-token Romanian and 700B-token English corpus, build a 128k-token uncased BPE tokenizer that intentionally omits Romanian diacritics, and fine-tune the model for English-to-Romanian translation. On a modified WMT benchmark with diacritics stripped and text lowercased, LLMic is reported to reach BLEU 41.01, ahead of mBART (38.50) and off-the-shelf instruct models (25-29), with additional quantization experiments. The central claim of the paper is that fine-tuned LLMic outperforms existing solutions for English-to-Romanian translation.","tokens_in":7388,"tokens_out":6121,"duration_ms":59156,"significance":"The release of an open 3B Romanian-focused model and the detailed documentation of corpus construction and tokenizer design would be genuinely useful for low-resource NLP, especially the reported tokenizer fertility improvements and the quantization results. However, the headline translation claim is not currently established: the evaluation protocol normalizes the test data to exactly match LLMic's tokenizer, the baselines are not matched on fine-tuning data or model scale, and no statistical uncertainty is reported. The paper's value would be materially increased by a fair comparison on standard orthographic Romanian.","major_comments":[{"comment":"The central comparison is not a valid test of the abstract's claim. The authors state that they 'modified the WMT data by removing diacritics and converting to lowercase,' while §3 says the tokenizer 'intentionally omits the Romanian characters: ă, â, î, ș, ț, replacing them with a, i, s, t respectively.' This aligns the evaluation with LLMic's design but prevents a direct comparison on standard Romanian, where diacritics distinguish words such as 'vesela'/'veselă' and 'cina'/'cină'. BLEU computed on normalized text cannot detect these errors, so the 41.01 score may simply reflect the fact that LLMic does not need to produce diacritics. Please report scores on the unmodified WMT Romanian references using standard tokenization and casing, and, if diacritic-free output is intended, provide a diacritic-restoration step or clearly reframe the claim as applying only to normalized text.","section":"§6, Table 3b"},{"comment":"Baselines are not comparable. LLMic is fine-tuned on parallel Romanian-English data (including ParaCrawl and EU documents, §2), while the listed 7B/8B models are off-the-shelf instruct models; RoMistral, RoLlama, and RoGemma are Romanian instruction-tuned, not necessarily translation models, and mBART is much smaller than LLMic. The table should include size-matched translation models fine-tuned on the same parallel data (e.g., NLLB-200, fine-tuned mBART or mT5), or should clearly label the comparison as one of a specialized compact NMT system versus general-purpose instruct models. At minimum, report the fine-tuning setup for all baselines.","section":"§6, Table 3b"},{"comment":"No error bars, significance tests, or details of the evaluation setup are given. The table does not report the number of WMT sentences, the BLEU implementation and tokenization (e.g., sacreBLEU), decoding parameters, number of references, or variance across runs. A 2.5-point gap over mBART and the 11-15-point gaps over instruct models need confidence intervals or bootstrap resampling before they support the comparative claim. The statement that 'qualitative analysis indicates that LLMic achieves comparable translation quality to closed models such as ChatGPT' should either be removed or supported by a described human evaluation protocol.","section":"§6"},{"comment":"The fine-tuning procedure for translation is not described: the text says only that LLMic is 'fine-tuned' for translation, with no parallel-data split, number of epochs, learning rate, or indication of whether the LoRA result in Table 3a refers to the same model used in Table 3b. The LoRA (r=8) row reaches only 37.05, which is below mBART, so the reader cannot tell which configuration produced the headline 41.01. Please document the fine-tuning recipe fully and release the training configurations.","section":"§6"}],"minor_comments":[{"comment":"The text says the maximum learning rate is 4×10−3 for both the constant and cosine schedules, while Table 2b lists 4×10−4; please reconcile the two values.","section":"§5, Table 2b"},{"comment":"The sentence listing omitted Romanian characters is garbled ('ă, â, î, s, , t,') and should be corrected to include ș and ț properly.","section":"§3"},{"comment":"The phrase 'lower lower accuracy' and the typos 'usefulnes', 'curates', and 'whith' should be corrected.","section":"§6"},{"comment":"References [32] and [33] appear to cite the same Gemma technical report twice; please consolidate or cite distinct sources.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The main risk is that the headline comparison is tailored to the model's tokenizer and unmatched baselines. I do not see grounds for rejection if the authors rerun or carefully reframe the evaluation, but the current abstract overclaims. The heavy reliance on the authors' own FuLG corpus is also worth an independent quality check in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Key takeaway: this is a real artifact paper—a newly trained 3B bilingual model, tokenizer, and documented recipe—but the central translation claim does not survive contact with the evaluation setup. The authors stripped diacritics and lowercased the WMT test set to match LLMic's tokenizer, then declared victory over models designed for standard Romanian orthography. That is not a neutral preprocessing step; it changes the task. So the headline result, as stated, is unsupported. This is a load-bearing flaw, not a footnote.\n\nWhat's genuinely useful: the paper ships a model under Apache 2.0, a custom BPE tokenizer with good fertility for Romanian, a 300B-token Romanian corpus mixture, and a clear training recipe. The quantization experiments are a nice engineering addition. The choice to drop diacritics is presented as deliberate, but the preliminary experiments are not shown, and there is no controlled comparison against a diacritic-aware tokenizer. Still, the artifact itself is real and checkable.\n\nThe soft spots beyond the test-set modification: baselines are off-the-shelf instruct models rather than models fine-tuned on the same parallel data, and mBART is much smaller than LLMic, so no comparison is matched on all axes. There are no error bars or significance tests, and the qualitative claim of parity with ChatGPT rests on unspecified analysis. None of this fatally undermines the model's potential, but it means the paper's central claim about state-of-the-art translation is not established on the evidence provided.\n\nWho should read it: anyone working on low-resource language modeling, especially for Romanian, and people interested in small specialized foundation models. The corpus composition and tokenizer design are valuable. But the evaluation section needs a serious rewrite with standard Romanian orthography, matched fine-tuned baselines, and ideally human evaluation.\n\nMy recommendation: send it to peer review anyway. The artifact is substantial, the authors are transparent about many choices, and the main flaw is fixable. A good referee should push them to redo the translation evaluation on unmodified WMT with properly matched models. That is exactly what the review process is for.","headline":"Real model and recipe, but the translation claim is built on a modified test set that favors LLMic's diacritic-free design.","tokens_in":7941,"tokens_out":2353,"would_cite":false,"duration_ms":22738,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A compact 3B-parameter bilingual Romanian-English model, fine-tuned for translation, outperforms much larger open models on English-to-Romanian translation.","keywords":["Romanian language model","bilingual pretraining","low-resource language","machine translation","BPE tokenizer","diacritic-free text","model quantization","foundation model"],"falsifier":"Use the original WMT English-to-Romanian test set with diacritics and casing preserved, evaluate LLMic and the baselines with their native tokenizers, and check whether LLMic's 41.01 BLEU score still beats mBART's 38.50; if the gap vanishes or reverses, the claimed superiority is conditional on that preprocessing.","tokens_in":6910,"feed_emoji":"🤖","tokens_out":12621,"duration_ms":104960,"temperature":0.7,"pith_summary":"LLMic is a 3-billion-parameter bilingual foundation model pretrained from scratch on roughly one trillion tokens, about 300 billion in Romanian and 700 billion in English. The central claim is that fine-tuning this compact model for English-to-Romanian translation yields a WMT score of 41.01, ahead of mBART's 38.50 and far above several 7B-8B instruction-tuned models that score in the 25-29 range. The paper also reports qualitative parity with closed commercial models and shows that INT4 quantization costs less than one BLEU point. If these results hold, a small dedicated model is enough to serve a low-resource language effectively, at a fraction of the compute of larger general-purpose systems.","feed_headline":"A 3B Romanian model beats 7B and 8B rivals on En-Ro translation","feed_subtitle":"Fine-tuned for translation, the 3B model scores 41.01 BLEU, ahead of mBART and 7B-8B rivals","key_machinery":"The load-bearing component is the pretraining recipe built around the model and its tokenizer. LLMic is a 24-layer decoder-only transformer with grouped-query attention, rotary position embeddings, and SiLU activations, pretrained with a cosine learning-rate schedule. Its 128,000-token BPE vocabulary is uncased and deliberately maps the Romanian diacritics ă, â, î, ș, ț to plain a, i, s, t, yielding a fertility rate of 1.38 tokens per word. The data recipe gives the model roughly 300B Romanian and 700B English tokens, with the first 50B tokens seen in a 50/50 split and the rest in the 30/70 proportion; fine-tuning this checkpoint for the translation task produces the reported result.","core_discovery":"The paper's central discovery is that a 3B-parameter model pretrained on a 30/70 Romanian-English mix, then fine-tuned for translation, reaches 41.01 BLEU on an English-to-Romanian WMT test set that has been lowercased and stripped of diacritics. This beats mBART at 38.50 and several 7B-8B instruction-tuned models scoring between 25 and 29. The paper further shows that quantizing LLMic to INT4 costs only 0.72 BLEU points, and that its translations are, in the authors' qualitative assessment, comparable to those of closed commercial models. The conclusion the authors draw is that prioritizing pretraining tokens for the target language matters more than scaling up parameter count.","pith_inferences":["A testable extension is to run the same translation comparison on the original, diacritic-preserving WMT test set, which would isolate how much of LLMic's lead comes from matching its uncased tokenizer.","The 30/70 Romanian-English pretraining mix, with high-quality Romanian documents repeated in later training, is a transferable recipe for other low-resource languages with small curated web corpora.","A blinded human rating study would turn the stated qualitative parity with closed commercial models into a quantifiable, comparable score.","The small INT4 quantization penalty suggests that 4-bit fine-tuned versions for downstream Romanian tasks are a plausible next step, though the paper does not test them."],"forward_implications":["A specialized 3B model can beat much larger open models on a low-resource language task, making language-specific pretraining a practical alternative to model-scale scaling.","The quantized INT4 variant loses less than one BLEU point, so edge-device and large-scale Romanian translation deployments are a direct consequence of the reported results.","Other low-resource languages can use the same recipe: build a high-quality monolingual corpus, pair it with a larger English corpus, and pretrain a small bilingual model instead of adapting an English-centric one.","Because LLMic's tokenizer has a fertility rate of 1.38 tokens per word on Romanian text, the model is cheaper and faster per word than models with less efficient tokenizers."],"supporting_citations":[{"why":"Supplies the 220B-token Romanian web corpus that forms the bulk of the Romanian pretraining data.","marker":"[3]"},{"why":"Supplies the English pretraining data used for the larger share of the token mix.","marker":"[25]"},{"why":"Contributes curated English books, Wikipedia, research papers, and forum data to pretraining.","marker":"[29]"},{"why":"Defines the WMT English-to-Romanian benchmark and test set used for the model comparison.","marker":"[8]"},{"why":"Provides the mBART baseline whose 38.50 score anchors the claimed outperformance.","marker":"[22]"},{"why":"Provides the earlier Romanian fine-tuning approach that LLMic's translation result is compared with.","marker":"[23]"},{"why":"Supplies the underlying transformer architecture that LLMic's design is based on.","marker":"[34]"},{"why":"Supplies the tokenizer design on which the custom BPE tokenizer is based.","marker":"[7]"}],"fun_headline_variants":["3B Romanian model tops 7B and 8B rivals in translation","Small Romanian LLM outdoes larger models on En-Ro","Tiny 3B model beats big rivals on English-Romanian","Romanian 3B LLM edges out 7B and 8B on BLEU","3B model beats big rivals for Romanian translation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison rests on treating the removal of diacritics and lowercasing of the WMT test set as a neutral preprocessing step that does not favor LLMic's deliberately uncased, diacritic-free tokenizer.","fun_headline_variants_meta":{"raw":{"variants":["3B Romanian model tops 7B and 8B rivals in translation","Small Romanian LLM outdoes larger models on En-Ro","Tiny 3B model beats big rivals on English-Romanian","Romanian 3B LLM edges out 7B and 8B on BLEU","3B model beats big rivals for Romanian translation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000662,"raw_usage":{"total_tokens":2995,"prompt_tokens":887,"completion_tokens":2108,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":2012}},"tokens_in":503,"tokens_out":2108,"duration_ms":15601,"temperature":1.0,"reasoning_tokens":2012,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:36:45.178350+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use the original WMT English-to-Romanian test set with diacritics and casing preserved, evaluate LLMic and the baselines with their native tokenizers, and check whether LLMic's 41.01 BLEU score still beats mBART's 38.50; if the gap vanishes or reverses, the claimed superiority is conditional on that preprocessing.","supporting_citations":[{"cited_title":"FuLG: 150B Romanian Corpus for Language Model Pretraining","cited_arxiv_id":"2407.13657","evidence_quote":"Supplies the 220B-token Romanian web corpus that forms the bulk of the Romanian pretraining data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the WMT English-to-Romanian benchmark and test set used for the model comparison."},{"cited_title":"Vorbeşti Româneşte?","cited_arxiv_id":null,"evidence_quote":"Provides the earlier Romanian fine-tuning approach that LLMic's translation result is compared with."}],"review_version":1}