{"id":"00ddb8e1-aa3d-4427-af94-262e42041015","arxiv_id":"2504.15431","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Trillion-7B pairs Korean documents with English documents during pretraining and lets them attend to each other, claiming competitive Korean performance with only about 10% multilingual tokens.","lead":"Trillion-7B is a 7-billion-parameter language model aimed at Korean and other languages, trained with a new attention setup that lets Korean training text read nearby English text in the same batch. The authors claim this cuts the share of non-English training data to about 10% of a 2-trillion-token budget and brings the full training cost to roughly $148,000.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"XLDA is never ablated against standard document masking, so the central causal claim that XLDA drives cross-lingual transfer is unsupported.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern: the paper asserts that XLDA transfers knowledge and drives token efficiency, but never isolates the mask change from concurrent data-quality, annealing, diversity, and tokenizer improvements. My reading of Sections 2, 3, and 6 confirms this. The paper is a real training report with useful ablations of other recipe components, and the authors do not appear to hide or distort their main results, but the strongest claim in the abstract is a causal claim about XLDA, and no experiment in the paper tests it. The proposed controlled comparison directly addresses the gap: it holds everything fixed except the attention mask. If XLDA provides no measurable advantage, the central contribution of the paper collapses to a data-recipe engineering report; if it does provide an advantage, the claim would be substantiated. The internal inconsistency in the multilingual token budget is real and worth correcting, but it is not the primary reason the central claim fails. Since the reader's REJECT verdict is consistent with this analysis and no new evidence changes that, the verdict should remain unchanged.","tokens_in":35219,"tokens_out":2399,"duration_ms":23508,"concrete_test":"Run a controlled pre-training comparison at 1.8B scale on 100B tokens with identical data, tokenizer, annealing schedule, and quality filtering, changing only the attention mask: one run with XLDA (cross-lingual document attention) and one run with the standard per-document causal mask used in typical packed training. Evaluate on KoBEST, HAERAE, KMMLU, GMMLU-ko, GMMLU-ja, and English control benchmarks (e.g., MMLU, HellaSwag). If the standard-mask run matches XLDA within a few points, the central causal claim fails; if XLDA is clearly superior on Korean/Japanese benchmarks without English regression, the claim gains support. If resources allow, repeat the comparison on the 7B model over the final 200B-token annealing phase, since that is where the production model's gains are reported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim, stated in the abstract and Section 2, is that the novel Cross-lingual Document Attention (XLDA) mechanism enables efficient knowledge transfer from English to Korean and Japanese. For that claim to hold, the specific attention-mask change must be causally responsible for the reported gains. The report never provides the comparison that would establish this: a model trained with standard per-document causal masking under otherwise identical data, tokenizer, annealing schedule, and compute. Instead, Section 6 ablates quality filtering (6.1), annealing composition (6.1), data diversity (6.2), and vocabulary size (6.3), but no experiment varies the attention mask. The description in Section 2.2 asserts that 'this attention masking is effective because of the above enforced cross-lingual document packing,' yet this is an assertion, not a measurement. Without a standard-mask baseline, the observed Korean and Japanese performance could be attributable to the concurrent data filtering, annealing upsampling, multi-token prediction, or the Korean-specific tokenizer, any of which are known to affect multilingual performance. The internal inconsistency in the reported multilingual token budget (abstract says 10% of 2T; Section 1 says '<220B tokens, with less than 180B in Korean', while the 8.5:1:0.5 mixture implies ~200B Korean tokens alone) further complicates interpretation but is secondary to the missing causal test. The central scientific claim is therefore not demonstrated by the evidence presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Trillion-7B is a 7B-parameter Korean-centric multilingual language model trained on 2T tokens, with the stated goal of transferring English knowledge to Korean and Japanese using a proposed Cross-lingual Document Attention (XLDA) mechanism. XLDA packs documents from different languages into the same sequence and removes the standard causal-mask document boundary for cross-lingual attention. The report describes the pretraining recipe (data filtering, two-stage annealing, tokenizer, infrastructure, context extension), post-training with a Tulu-3-style SFT/DPO/RLVR pipeline, and evaluations on 27 benchmarks in English, Korean, Japanese, and Chinese, plus cross-lingual consistency and a VLM extension experiment. The central claims are token efficiency (59.4K H100 GPU hours, about $148K), competitive performance, and that XLDA is the mechanism enabling efficient knowledge transfer.","tokens_in":35466,"tokens_out":8281,"duration_ms":72915,"significance":"If established, the result would be significant: it would show that a small architectural change (attention masking) plus data curation can substitute for large amounts of target-language pretraining data, and the cost accounting is a useful contribution. The paper's strengths include a broad evaluation suite, explicit training-cost reporting, and a set of ablations for data quality, annealing composition, data diversity, and vocabulary size. Its main weakness is that the causal contribution of XLDA itself is never isolated, and the headline budget and performance statements are not fully consistent with the reported numbers.","major_comments":[{"comment":"The central causal claim that XLDA enables cross-lingual transfer is not tested. Section 2.2 describes the XLDA mask, but none of the Section 6 ablations vary the attention mask: Section 6.1 varies quality filtering and annealing composition, Section 6.2 varies data diversity, and Section 6.3 varies vocabulary size. A matched comparison with standard per-document causal masking under otherwise identical data, tokenizer, training schedule, and compute is required; without it, the observed Korean and Japanese results could be driven by the simultaneous changes in data filtering, annealing upsampling, multi-token prediction, or the Korean-specific tokenizer. This omission is load-bearing because the abstract and Section 1 attribute the efficiency gains to XLDA.","section":"§2.2 and §6"},{"comment":"The multilingual token budget is internally inconsistent. The abstract and Section 1 say roughly 10% of 2T tokens are multilingual, with \"less than 180B\" Korean tokens; Section 3.1 states an 8.5:1:0.5 English:Korean:Other ratio, which at 2T tokens implies about 200B Korean tokens alone and about 300B total non-English tokens (15%). These statements cannot all be correct, and the 10% figure is a headline efficiency claim that must be reconciled with the actual mixture and token counts.","section":"§3.1 vs Abstract/§1"},{"comment":"The phrase \"competitive performance\" is not supported by the model's own summary table. Trillion-7B macro-averages 57.15 and ranks third of six models, behind Qwen2.5-7B (66.15) and EXAONE-3.5-8B (66.71). The largest deficits are in Coding (47.94 vs 66.35 and 70.33) and Math (45.02 vs 67.29 and 65.82), which are central to general-purpose LLM claims. If the intended claim is \"competitive for the amount of compute or tokens used,\" the paper needs a compute-normalized comparison and confidence intervals; as written, the broad competitive-performance claim overreaches the reported results.","section":"Table 4"},{"comment":"The vision-language generalization claim is not a controlled comparison. Trillion-LLaVA is compared to Llava-1.5-Vicuna-7B and Llava-1.6-Mistral-7B after English-only visual instruction tuning, but the underlying base LLMs differ in architecture, pretraining data, and post-training. The higher Korean VLM scores could reflect properties of the base model rather than the multilingual pretraining; an adequate control would fine-tune the same base architecture with and without the Trillion-7B multilingual pretraining.","section":"§7.2"}],"minor_comments":[{"comment":"The formula for P(l_i) contains an undefined operator \"˝\" and does not specify the normalization or the concrete values of α and β_l used in training; please fix the notation and give the actual mixture parameters.","section":"§2.1, Eq. (2)"},{"comment":"The text refers to \"Qwen-72B-Instruct\" in Section 3.1 and \"Qwen-2.5-72B\" in Section 4; please clarify whether these are the same model and which checkpoint was used for filtering and for LLM-as-a-judge scoring.","section":"§3.1 and §4"},{"comment":"The evaluated model is labeled \"Trillion-7B-preview\" in Table 10, while the abstract and title use \"Trillion-7B\"; please state explicitly whether these refer to the same checkpoint.","section":"Table 10"},{"comment":"SOLAR-10.7B appears in the Appendix C tables but is absent from the headline comparison in Table 4; either include it in the main table or explain the exclusion criterion.","section":"Table 4 and Appendix C"},{"comment":"The relationship between the apparent optimal Korean vocabulary size of 1,500–5,000 tokens in Figure 6, the scaling-law value of roughly 13,000 tokens, and the final choice of 24,552 tokens needs a clearer derivation; the current text jumps between these numbers without a defined selection rule.","section":"§3.5 and §6.3"},{"comment":"Figure 3 is described as showing scaling curves but provides no data points, model versions, or evaluation settings; please state the source or mark the figure as an illustrative sketch.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is closer to an industry technical report than a peer-reviewed methods paper. The decisive issue is the absence of any experiment isolating XLDA; if the authors cannot provide an XLDA-vs-standard-mask control at the proxy scale already used in Section 6, the paper should be rejected. Given that the paper already uses 1.8B-parameter, 100B-token ablations, this experiment appears feasible, so I recommend major revision rather than immediate rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a transparent, useful technical report on a real 7B training run, and the recipe details are worth having. The central claim about XLDA is not supported, because the paper never ablates the attention mask itself.\n\nWhat's new and good: a complete recipe for a Korean-centric model at 2T tokens with an unusual 8.5:1:0.5 English:Korean:Other mix, a tokenizer ablation showing a smaller Korean vocabulary is better, and a careful annealing study showing quality filtering and composition upsampling matter. The cross-lingual consistency analysis is a nice touch, and the vision transfer result is intriguing. The cost breakdown—59.4K H100 hours, $148K—is refreshingly concrete.\n\nThe soft spots: XLDA is the load-bearing mechanism, but no experiment compares it against standard per-document masking. The other ablations vary data quality, diversity, annealing, and vocabulary size, so the observed Korean gains could come from any of those. The paper also has an internal inconsistency in the multilingual token budget: the abstract says 10% non-English, while Section 1 says <220B with <180B Korean, but the stated mixture implies roughly 200B Korean tokens alone. And the model's own Table 4 puts it third of six on average, with large deficits in coding and math. 'Competitive' is doing a lot of work.\n\nThat said, the recipe as a whole may be valuable even if XLDA contributes nothing beyond standard packing. A simple ablation at the 1.8B proxy scale would settle that. The paper deserves a serious referee, because the missing experiment is well-defined and the engineering is substantial. I'd send it to review, but the authors should be required to add the XLDA ablation and correct the token-budget arithmetic before publication.","headline":"A transparent and useful training recipe, but the XLDA mechanism is never ablated, so the paper's central causal claim is unsupported.","tokens_in":36094,"tokens_out":3474,"would_cite":true,"duration_ms":30638,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 7B-parameter model can reach competitive Korean and Japanese performance with only 10% of its 2T training tokens in non-English languages, by letting those tokens attend to packed English documents.","keywords":["Cross-lingual Document Attention","XLDA","multilingual language model","Korean LLM","token-efficient pretraining","document packing","attention masking","cross-lingual transfer"],"falsifier":"Train the identical 7B recipe twice, once with XLDA and once with standard document-boundary masking, holding data mixture, quality filters, annealing schedule, tokenizer, and compute fixed; if the standard-mask run matches or beats Trillion-7B on the Korean benchmarks (KoBEST, HAERAE, KMMLU, HRM8k, KoIFEval), the central XLDA claim is refuted.","tokens_in":34985,"feed_emoji":"🇰🇷","tokens_out":9844,"duration_ms":83559,"temperature":0.7,"pith_summary":"The paper claims that a 7B-parameter language model can reach competitive Korean performance while spending only 10% of its 2 trillion training tokens on non-English data, and that the key ingredient is an attention-mask change. The proposed Cross-lingual Document Attention (XLDA) lets tokens in a Korean, Japanese, or Chinese document attend to a preceding packed English document, instead of applying the standard mask that blocks attention at every document boundary. On top of this, the model uses quality-filtered data, a two-stage annealing schedule, and a tuned tokenizer; together the recipe costs about 59.4K H100 GPU hours, roughly $148K. If the claims hold, multilingual capability can be bought with architecture and data curation rather than with massive multilingual corpora, which matters for languages that do not have enough text on the web.","feed_headline":"Attention trick lets a 7B model learn Korean with 10% tokens","feed_subtitle":"XLDA lets Korean tokens attend to English context, matching costlier multilingual training for $148K.","key_machinery":"Cross-lingual Document Attention (XLDA) is the load-bearing mechanism: a batch-level packing rule that places documents from at least two languages contiguously in one sequence, paired with a selective attention mask that keeps full self-attention across those language blocks rather than masking document boundaries. The paper describes this as a form of in-context pretraining and synthetic code-switching, so that low-resource-language tokens are always trained in the presence of an English context. Supporting machinery includes a controlled language-sampling mixture with a temperature and upsampling factor, a two-stage warmup-stable-decay schedule with quality-filtered annealing data, multi-token prediction, and a byte-level BPE tokenizer with roughly 100K English tokens and 24,552 Korean tokens.","core_discovery":"Trillion-7B's central claim is that cross-lingual knowledge transfer can be engineered into a pretraining run by changing how documents are packed and masked. Standard pretraining masks document boundaries so tokens cannot attend across documents; XLDA instead enforces that each packed sequence contains contiguous spans from at least two languages and leaves the attention between those language blocks unmasked. The paper argues this acts as architectural code-switching, so the model meta-learns non-English tokens inside an English linguistic context and transfers English knowledge to Korean, Japanese, and Chinese. With this mechanism plus stronger quality filtering (top 50% of multilingual data, top 10% during annealing), increased data diversity, and a Korean tokenizer sized at 24,552 tokens, the model reports competitive Korean benchmark scores and the highest English-to-Korean prediction consistency among the compared 7-9B models (77.5% of correct English predictions are also correct in Korean), despite using far fewer multilingual tokens and far less compute than the comparison models.","pith_inferences":["Editorial inference: If XLDA works by exposing low-resource tokens to English context, the same mask could be dropped into any English-centric pretraining run for another low-resource language, provided that language has enough clean, diverse text and a well-sized tokenizer; the paper's ablations suggest the mask alone is not sufficient, so the transferable unit is the whole mask-plus-curation rec","Editorial inference: The consistency result implies an English-anchored shared representation; a testable corollary is that XLDA gains shrink on tasks requiring local cultural knowledge or on languages with very different syntax and little shared vocabulary, where English context cannot supply the missing information.","Editorial inference: The English-only vision-language result suggests XLDA-style pretraining could decouple visual alignment from language-specific data collection; if so, one English visual-instruction run would transfer to Korean and other languages, which is directly testable with the released model."],"forward_implications":["Korean-capable 7B models become reproducible on a small budget: the paper reports full pretraining plus post-training for 59.4K H100 GPU hours, about $148K, with only 10% of tokens in non-English languages.","The transfer mechanism is not Korean-specific: the same 10% multilingual budget lifts Japanese and Chinese xwinograd and Global-MMLU scores, so the recipe should apply to any low-resource language paired with English.","English performance does not have to be sacrificed: the annealed high-quality, composition-shifted run improves Global-MMLU in English even while English data volume is reduced, which the paper attributes to cross-lingual bridging.","A small proxy model (1.8B parameters, about 100B tokens) can select the final training recipe, because downstream emergence is observable at that scale; this makes the full 2T run a scaled-up confirmation rather than a blind bet."],"supporting_citations":[{"why":"Defines the standard document-boundary masking that XLDA replaces; removing this baseline would leave the mechanism without a contrast.","marker":"Zhao et al., 2024a"},{"why":"Shows that document packing design can create in-context knowledge integration, the theoretical basis for XLDA's cross-language attention.","marker":"Levine et al., 2022"},{"why":"Provides the meta-learning framing the paper uses for saying non-English tokens are learned inside an English context.","marker":"Lampinen et al., 2024"},{"why":"Demonstrates that code-switching data improves language alignment in pretraining, which XLDA claims to approximate without explicit code-switched text.","marker":"Wang et al., 2025"},{"why":"Earlier code-switching curriculum result cited to support the claim that mixed-language contexts improve multilingual transfer.","marker":"Yoo et al., 2024a"},{"why":"Supplies the large multilingual model whose quality scores are distilled into the filtering model applied to all pretraining data.","marker":"Qwen et al., 2025"},{"why":"Source of the annealing-stage curriculum and the claim that high-quality data at the end of training reduces gradient noise.","marker":"Hu et al., 2024"},{"why":"Gives the vocabulary scaling law used to size the Korean tokenizer and to justify deviating from the theoretical optimum for inference speed.","marker":"Tao et al., 2024"}],"fun_headline_variants":["XLDA attention lets 7B model learn Korean with 10% multilingual tokens","Korean-centric 7B model uses 10% multilingual tokens via cross-lingual attention","Cross-lingual attention gives Korean LLM knowledge from just 10% of tokens","7B model gets Korean via cross-lingual attention with 10% of tokens","Multilingual 7B model rivals costlier ones using 10% multilingual tokens and $148K"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that letting Korean tokens attend to a preceding packed English document transfers useful knowledge rather than adding noise, and that this attention change—not the concurrent quality filtering, annealing, diversity, or tokenizer changes—is what drives the reported gains; the ablations reported test the other components but never isolate the mask.","fun_headline_variants_meta":{"raw":{"variants":["XLDA attention lets 7B model learn Korean with 10% multilingual tokens","Korean-centric 7B model uses 10% multilingual tokens via cross-lingual attention","Cross-lingual attention gives Korean LLM knowledge from just 10% of tokens","7B model gets Korean via cross-lingual attention with 10% of tokens","Multilingual 7B model rivals costlier ones using 10% multilingual tokens and $148K"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001528,"raw_usage":{"total_tokens":6069,"prompt_tokens":850,"completion_tokens":5219,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":466,"completion_tokens_details":{"reasoning_tokens":5105}},"tokens_in":466,"tokens_out":5219,"duration_ms":35309,"temperature":1.0,"reasoning_tokens":5105,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:26:57.876076+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the identical 7B recipe twice, once with XLDA and once with standard document-boundary masking, holding data mixture, quality filters, annealing schedule, tokenizer, and compute fixed; if the standard-mask run matches or beats Trillion-7B on the Korean benchmarks (KoBEST, HAERAE, KMMLU, HRM8k, KoIFEval), the central XLDA claim is refuted.","supporting_citations":[],"review_version":1}