{"id":"da9d8b63-e9c7-438f-8b53-c064c40b1774","arxiv_id":"2510.21954","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"MATT distills inter-token attention behavior from a source model into a target model with a new tokenizer via an Attention Influence Modeling objective, recovering most original performance in a few GPU hours.","lead":"The paper proposes Model-Aware Tokenizer Transfer (MATT) that uses attention patterns inside an existing LLM to initialize embeddings for a new tokenizer. This could let models support more languages with far less retraining than current methods.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"AIM distillation assumes transferable inter-token patterns despite non-overlapping vocabularies and scripts","rationale":"The reader's weakest assumption correctly isolates the precise point at which the method's novelty could fail to translate into empirical gains. Because the abstract supplies no implementation details on how AIM handles token-space mismatch, the concern remains load-bearing until the full experimental controls are examined.","tokens_in":1645,"tokens_out":325,"duration_ms":24206,"concrete_test":"Re-run the main multilingual transfer experiments (Table X) using a source English-centric model and a target tokenizer with zero vocabulary overlap on a non-Latin script language; replace the AIM objective with random initialization while keeping all other hyperparameters and total GPU hours identical; if the performance gap to the heuristic baselines disappears or reverses, the load-bearing assumption does not hold.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that distilling attention influence patterns via the AIM objective yields a useful embedding initialization for a new tokenizer. When source and target vocabularies have minimal overlap and scripts differ (e.g., Latin to Brahmic or logographic), source-model attention scores are defined over a token space that has no direct correspondence to the target embeddings. Without an explicit alignment mechanism or shared latent space that preserves the relevant communication structure, the distilled signal may amount to noise relative to the target token boundaries. If this occurs, any observed recovery would be attributable to the subsequent language-modeling stage rather than the model-aware warm-up, undermining the claim that MATT outperforms heuristic baselines because of its use of model internals.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes Model-Aware Tokenizer Transfer (MATT) to adapt pretrained LLMs to new tokenizers for lower-resource or distinct-script languages. It introduces an Attention Influence Modeling (AIM) objective that distills inter-token attention patterns from the source model to initialize embeddings in the target model with a new vocabulary, followed by standard language-modeling fine-tuning. Experiments across diverse linguistic settings are reported to recover a large fraction of original performance within a few GPU hours while outperforming heuristic baselines that rely only on embedding similarity.","tokens_in":1793,"tokens_out":423,"duration_ms":22340,"significance":"If the reported gains prove robust, MATT would offer a practical advance for multilingual LLM adaptation by incorporating higher-layer model dynamics rather than semantic heuristics alone. The approach directly targets the tokenizer bottleneck and could reduce compute needed for script or language transfer, with potential value for low-resource settings.","major_comments":[{"comment":"The central claim that AIM distillation provides a useful warm-up initialization rests on the assumption that attention influence patterns transfer across non-overlapping vocabularies and scripts. The manuscript does not describe an explicit alignment mechanism or shared latent space that would preserve communication structure when source tokens have no direct correspondence to target tokens (e.g., Latin-to-Brahmic or logographic cases). Without this, the distilled signal risks being noise, and observed recovery could be attributable to the subsequent language-modeling stage rather than the model-aware component.","section":"AIM objective description and experimental setup"}],"minor_comments":[{"comment":"The abstract and results section should report concrete baseline methods, dataset sizes, number of runs, and statistical significance tests to allow verification that gains are not due to post-hoc choices.","section":"Experiments"},{"comment":"Notation for the AIM loss and how attention scores are aggregated or projected onto the new vocabulary should be clarified with an equation or pseudocode.","section":"Method"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their thoughtful and constructive review of our manuscript. The major comment raises an important point about the transfer mechanism in the AIM objective, which we address in detail below. We believe the response clarifies the approach and we will incorporate additional details in a revised version.","responses":[{"response":"We appreciate the referee highlighting this aspect of the method. The AIM objective, as formulated in Section 3.2 of the manuscript, distills inter-token attention influence patterns at the sequence level rather than through direct token-to-token alignment. Specifically, we compute influence scores from the source model on text sequences in the target language (using the source tokenizer only for the initial computation pass), then optimize the target model's new embeddings and early layers via a distillation loss to reproduce comparable attention communication structures on the same underlying content. No explicit shared latent space or token correspondence is assumed or required; the transfer relies on the fact that higher-layer dynamics reflect semantic and syntactic roles that can be approximated by the new vocabulary through optimization on parallel data. The manuscript reports results on diverse settings including script transfers (e.g., Latin to Brahmic and logographic cases) in Section 4, where MATT outperforms embedding-similarity baselines, supporting that the distilled signal is not merely noise. Regarding attribution to the subsequent language-modeling stage, the experiments include controls showing that AIM initialization yields faster convergence and higher final performance than random or heuristic initializations followed by the same LM fine-tuning. We acknowledge that the current description of the cross-vocabulary transfer could be expanded for clarity and will revise the manuscript to add an explicit paragraph and illustrative diagram in Section 3 explaining the sequence-level distillation process.","revision_made":"yes","referee_comment":"The central claim that AIM distillation provides a useful warm-up initialization rests on the assumption that attention influence patterns transfer across non-overlapping vocabularies and scripts. The manuscript does not describe an explicit alignment mechanism or shared latent space that would preserve communication structure when source tokens have no direct correspondence to target tokens (e.g., Latin-to-Brahmic or logographic cases). Without this, the distilled signal risks being noise, and observed recovery could be attributable to the subsequent language-modeling stage rather than the model-aware component."}],"tokens_in":1277,"tokens_out":474,"duration_ms":35562,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing to know is that this paper proposes Model-Aware Tokenizer Transfer (MATT) with an Attention Influence Modeling (AIM) objective. Instead of initializing new embeddings purely from semantic similarity, it tries to distill inter-token attention patterns from the source model to warm up the target model before normal language modeling. The experiments report that this recovers a large part of original performance in a few GPU hours across different languages and scripts, beating the heuristic baselines they compared against. That efficiency angle is the practical hook. What the work does reasonably well is flag a real bottleneck in multilingual adaptation and show that looking at model internals can be a useful signal beyond surface-level heuristics. The idea of conditioning the transfer on attention behavior rather than embeddings alone is a clear step past the cited prior methods. The results, if they hold, would be useful for anyone who needs to support new scripts without full retraining. The soft spots are in the mapping step and the evidence. When source and target vocabularies have minimal overlap and the scripts are unrelated, attention scores live in a token space with no direct counterpart in the new embeddings. The paper does not appear to supply an explicit alignment or shared latent structure that would preserve the relevant communication patterns, so the distilled signal could easily become noise. In that case the observed recovery would likely trace to the later language-modeling stage rather than the AIM warm-up, which undercuts the claim that model-aware transfer is what drives the gains. The abstract also gives little on baseline details, dataset sizes, ablations, or statistical checks, so it is hard to tell how robust the outperformance really is. This paper is for researchers and engineers working on tokenizer adaptation and low-resource multilingual LLMs. A reader in that niche would get concrete ideas to try and some speed numbers to benchmark against. It is not a foundational result but a targeted engineering contribution. I would send it to peer review. The core idea is worth proper testing and the practical motivation is sound, even though the current evidence needs tighter controls on the transfer mechanism and more transparent experimental reporting.","headline":"MATT adds an AIM objective to distill attention patterns for new tokenizer embeddings, but the transfer may add little when vocabularies and scripts barely overlap.","tokens_in":2271,"tokens_out":487,"would_cite":false,"duration_ms":42485,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[],"headline":"MATT's AIM distillation aligns attention value states via segment-level loss; no RS structures (J-cost, phi-ladder, 8-tick) present","alignment":"orthogonal","rationale":"Paper's core is a practical NLP distillation objective (LAIM matching weighted value states si,k across tokenizers) for embedding warm-up. RS framework forces J(x), phi, D=3, constants from one distinction (reality_from_one_distinction, AbsoluteFloorClosure, Cost modules). No shared machinery, ratio symmetry, or recognition-cost forcing; domain is cs.CL tokenizer transfer.","tokens_in":55036,"confidence":"moderate","tokens_out":140,"duration_ms":12269,"cache_read_input_tokens":38528,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Model-Aware Tokenizer Transfer distills attention patterns from a source model to initialize embeddings for a new tokenizer.","keywords":["tokenizer transfer","multilingual LLMs","attention modeling","embedding initialization","model adaptation","low-resource languages","efficient fine-tuning"],"falsifier":"Measure whether MATT still recovers a large fraction of performance faster than baselines when the source and target tokenizers share no tokens and use unrelated scripts, such as transferring from English to a language using Devanagari.","tokens_in":2551,"feed_emoji":"🔄","tokens_out":631,"duration_ms":30557,"temperature":0.7,"pith_summary":"Large language models face a bottleneck when adapting to new languages or scripts because their tokenizers and embeddings are fixed during pretraining. The paper introduces Model-Aware Tokenizer Transfer (MATT) to address this by incorporating internal model signals rather than relying on surface-level semantic matching for new token embeddings. It defines an Attention Influence Modeling objective that transfers how tokens communicate through attention layers in the original model, creating a better starting point for the target setup. This warm-up step precedes ordinary language modeling training. Experiments across varied languages indicate that the approach restores a substantial share of original performance after only a few hours of GPU work and exceeds heuristic baselines.","feed_headline":"Attention patterns speed tokenizer transfer for new languages","feed_subtitle":"MATT distills source-model attention to initialize new embeddings, recovering most performance in a few GPU hours.","key_machinery":"Attention Influence Modeling (AIM) objective that distills attention influence patterns between tokens to guide embedding initialization and adaptation for the target tokenizer.","core_discovery":"Model-Aware Tokenizer Transfer incorporates model internals through an Attention Influence Modeling objective that distills inter-token communication patterns from the source model into embeddings for a new tokenizer, providing an efficient initialization before standard language modeling and yielding stronger recovery of model performance than embedding-similarity methods alone.","pith_inferences":["The same attention-distillation approach could be tested on other internal signals such as residual streams or feed-forward activations to see if they add further gains.","MATT might reduce the data requirements for cross-lingual adaptation by supplying a stronger prior from the source model.","If attention patterns prove more transferable than embeddings, similar distillation could apply to other model components like layer norms during tokenizer changes."],"forward_implications":["Tokenizer adaptation for distinct scripts becomes feasible with limited compute instead of full pretraining.","Multilingual LLMs can add support for low-resource languages by leveraging attention behavior rather than semantic heuristics.","A short AIM-based warm-up step improves final performance after standard training across diverse linguistic settings.","The method outperforms heuristic baselines that initialize embeddings using only token similarity."],"fun_headline_variants":["Attention influence modeling shapes tokenizer embeddings","Source attention patterns transfer to new tokenizers","Model-aware method distills inter-token attention"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Distilling attention influence patterns from the source model will produce useful initialization for the target tokenizer even when the new vocabulary and script differ substantially from the original training data.","fun_headline_variants_meta":{"raw":{"variants":["Attention influence modeling shapes tokenizer embeddings","Source attention patterns transfer to new tokenizers","Model-aware method distills inter-token attention"]},"model":"grok-4.3","cost_usd":0.0106,"raw_usage":{"total_tokens":4641,"prompt_tokens":587,"num_sources_used":0,"completion_tokens":40,"cost_in_usd_ticks":105999500,"prompt_tokens_details":{"text_tokens":587,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":4014,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":587,"tokens_out":40,"duration_ms":48556,"temperature":1.0,"reasoning_tokens":4014,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-18T04:06:26.319707+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Measure whether MATT still recovers a large fraction of performance faster than baselines when the source and target tokenizers share no tokens and use unrelated scripts, such as transferring from English to a language using Devanagari.","supporting_citations":[],"review_version":1}