{"id":"207415b2-bf52-4999-93a6-6cfd44eb3dde","arxiv_id":"2501.02795","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"InfiFusion fuses multiple large language models into one pivot model using enhanced universal logit distillation, and reports that the fused model outperforms all source models on 11 benchmarks with a fraction of the training cost.","lead":"InfiFusion is a training method that combines several specialized AI language models into one smaller model by teaching the smaller model the output patterns of the larger ones. The authors report the combined model beats each individual model on 11 benchmarks, with only about 160 GPU-hours of training instead of the usual millions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (5) is underspecified: Top-K logits from different vocabularies are not aligned, so the central fusion loss is not a well-defined distance and the empirical claim is irreproducible as written.","rationale":"I read the paper in good faith and considered whether the intended loss could be inferred from context. The paper repeatedly emphasizes cross-vocabulary fusion and cites ULD, which uses sorted order statistics (Eq. 3). One might guess that Top-K selection also returns sorted values, making Eq. (5) an L1 distance between standardized order statistics. However, the text never says this, and the notation in Eq. (5) uses vector subscripts that suggest elementwise alignment without any sorting or mapping step. This is not a minor omission: it is the core training objective. If the alignment is undefined, the reported experimental results cannot be reproduced from the paper, and the theoretical framing in Section 3.3.1 loses its meaning. The reader's weakest_assumption identifies exactly this issue, and I agree. I also note secondary concerns such as model selection on the same benchmarks and lack of error bars, but these would be less decisive if Eq. (5) were well-defined. The proposed concrete test would settle the ambiguity by checking whether the two plausible implementations behave differently. If they behave the same, the omission is stylistic; if they differ, the central claim is undermined. Given the paper as written, the REJECT verdict remains appropriate.","tokens_in":15872,"tokens_out":4613,"duration_ms":44919,"concrete_test":"Obtain the training code or, if unavailable, re-implement Eq. (5) under the two plausible alignments: (A) sort the K selected logits descending before computing the L1 distance; (B) align logits by token ID through a shared vocabulary mapping. Train InfiFusion_u with each interpretation on the same 144k-sample dataset and compare final average scores on the 11 benchmarks. If the two implementations yield materially different results (e.g., average score differs by more than 0.5 points or training diverges), the objective is underspecified and the central claim is not reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central contribution is the fusion loss in Eq. (5): Lpair,s = E[ sum_t |\\hat l_{o,t} - \\hat l_{s,t}| ], where \\hat l are top-K logits after standardization. The pivot and source models use different vocabularies, so the K selected logits are not naturally paired. The paper never states whether the top-K logits are sorted by value before subtraction, mapped through a shared token space, or aligned by token ID. If the logits are not sorted or mapped, element i of \\hat l_o and element i of \\hat l_s refer to different tokens, making the L1 distance meaningless as a distillation objective. If sorting is intended, the loss becomes a comparison of order statistics (similar to the 1-Wasserstein distance in Eq. 3), but this is not stated and would still fail to transfer token-specific knowledge. Section 3.3.1 and the surrounding text provide no resolution. Because Eq. (5) is the training objective for both InfiFusion_p and InfiFusion_u, the method cannot be implemented or reproduced from the paper as written, and the reported 79.92 average score rests on an underspecified loss.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces InfiFusion, a framework for fusing multiple domain-specialized LLMs into a single pivot model using a distillation objective that combines Top-K logit selection, logit standardization, and the Universal Logit Distillation loss. Two fusion strategies are proposed: pairwise fusion, where each source is distilled separately and then the resulting models are merged, and unified fusion, where all sources are distilled simultaneously. The authors report that InfiFusion outperforms the pivot model Phi-4 and several baselines across 11 benchmarks, while requiring only about 160 H800 GPU hours for the unified variant. The paper also includes ablations on the Top-K value, logits standardization, number of source models, and dataset scale, plus an appendix that attempts to relate the two fusion strategies.","tokens_in":16209,"tokens_out":3758,"duration_ms":37341,"significance":"If the method is implementable as described and the reported gains are robust, the work would offer a computationally efficient approach to combining heterogeneous LLMs without requiring identical architectures or vocabularies. The idea of extending Universal Logit Distillation with Top-K selection and standardization is plausible and could be a useful contribution to model fusion. However, the central training objective is underspecified in a way that prevents reproduction, the theoretical analysis in the appendix is not a proof, and the empirical validation lacks statistical rigor and suffers from model selection on the evaluation benchmarks. These issues substantially weaken the significance of the claims as currently presented.","major_comments":[{"comment":"The pairwise fusion loss in Eq. (5) is defined as an L1 distance between normalized top-K logits of the pivot and source models, but the source and pivot models have different vocabularies. The paper never states how the K logits from each model are matched: are they sorted by value, aligned by token ID, or mapped through a shared space? Without this specification, the elementwise subtraction in Eq. (5) is not a well-defined distance, and the training objective cannot be implemented or reproduced. If the intended operation is sorting by value, that should be stated explicitly, since it changes the loss from a token-level alignment to a comparison of order statistics; if some other alignment is used, it must be described. This is a load-bearing issue because Eq. (5) underlies both InfiFusion_p and InfiFusion_u.","section":"Section 3.3.1, Eq. (5)"},{"comment":"The paper claims in Section 3.3.2 to provide 'a detailed theoretical analysis' of the relationship between unified and pairwise fusion, but Appendix A does not prove that relationship. Equation (20) simply assumes that a global loss L exists such that the zero-gradient condition at the individual optima implies the zero-gradient condition at the averaged parameters. This is an unproved assumption, not a derivation. The appendix should either present a rigorous proof under explicit conditions or clearly label this as a conjecture. As written, the theoretical support for pairwise fusion is absent.","section":"Appendix A, Eq. (20)"},{"comment":"The choice of K=10 in Eq. (5) is selected by ablating on the same 11 benchmarks used for the final evaluation (Table 5), and the pivot and source models are selected based on Table 7, which reports performance on the same 11 benchmarks. This constitutes selection on the evaluation metric, which can inflate reported performance. The authors should use a separate validation set for model selection and hyperparameter tuning, or at minimum report the results across multiple random seeds and show that the selected configuration generalizes. Without this, the reported average scores are difficult to interpret.","section":"Section 4.1 and Table 5 / Table 7"},{"comment":"All benchmark numbers in Table 2 are single runs without error bars, standard deviations, or multiple seeds. The reported differences are small: InfiFusion_u achieves 79.92, InfiFusion_p&TA 79.96, FuseChat 78.70, and SFT 77.96. It is unclear whether these gaps are statistically meaningful given the lack of repeated trials. The authors should report variance across at least three seeds and, if possible, significance tests for the main comparisons.","section":"Table 2 and Section 4.2"}],"minor_comments":[{"comment":"Equation (3) defines the 1-Wasserstein distance with a sum over V elements, but if Vo and Vs differ, the sorted sequences have different lengths. The text explains padding with zeros, but this should be explicitly reflected in the equation or its surrounding notation for clarity.","section":"Section 3.1.3, Eq. (3)"},{"comment":"The sentence following Eq. (6) has a grammatical issue: 'Let Mo,s be the optimized pivot model, its parameters are θo,s are obtained via' should be revised to a single coherent statement.","section":"Section 3.3.1, Eq. (6)"},{"comment":"The appendix opens with 'In this session, we analyze the relationship...' — 'session' should be 'section'.","section":"Appendix A, first sentence"},{"comment":"The abstract contains a grammatical error: 'significantly reduces computational costs' should be 'significantly reducing computational costs' for parallel structure with the preceding phrase.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim is potentially interesting, but the underspecification of Eq. (5) and the unproved assumption in Appendix A are serious issues that must be resolved before the work can be trusted. The empirical evaluation also needs more rigorous statistical reporting and a clearer separation between model selection and evaluation. If the authors can clarify the alignment in the loss, reframe the theoretical claims, and provide multi-seed results, the paper could become a useful contribution. As it stands, the manuscript is not reproducible from the text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a plausible incremental contribution that deserves a serious referee, but the paper as written has a central reproducibility gap in the loss definition that needs fixing.\n\nThe genuinely new bit is the specific combination of ULD with Top-K selection and logits standardization for multi-source fusion, plus the systematic comparison of pairwise and unified fusion. The experiments are fairly thorough: they compare against FuseLLM, FuseChat, MiniLogit, SeqKD, report GPU hours, and ablate K, data scale, and number of sources. The 160 H800 GPU hour claim is striking and the evaluation is on held-out benchmarks, so the empirical result is not obviously circular.\n\nThe soft spots are real. Equation (5) writes an L1 distance between normalized top-K logits from different vocabularies without saying how the K entries are matched. If they're matched by token ID, the loss is meaningless; if by sorted order, that's the 1-Wasserstein distance in disguise and should be stated. This is not a cosmetic issue — nobody can implement the method from the paper as written. Second, all numbers are single runs with no error bars, and some comparisons differ by 0.04 points, so the ranking between InfiFusion_u and InfiFusion_p&TA is noise. Third, K=10 is chosen by ablating on the same 11-benchmark average used for the headline result, a mild selection on the evaluation metric. Finally, the appendix's 'theoretical analysis' assumes the existence of a global loss in Eq. (20) without proof; it's a heuristic argument, not a derivation.\n\nNone of these are necessarily fatal. The method is a sensible extension of ULD, and the empirical trend across ablations is consistent. But the paper needs a revision that clarifies the loss, adds multiple seeds, and ideally releases code/data. I'd send it to peer review rather than desk reject — the core idea is worth engaging with, and these issues are addressable. The right venue is a solid NLP/ML conference workshop or a main conference with lenient reproducibility standards.","headline":"A plausible incremental fusion method that deserves review, but the central loss equation is underspecified and needs fixing.","tokens_in":16652,"tokens_out":4323,"would_cite":false,"duration_ms":40868,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fusing three specialized LLMs into one pivot model beats all three on 11 benchmarks, using 160 H800 GPU hours.","keywords":["LLM fusion","knowledge distillation","Universal Logit Distillation","Top-K logits","logits standardization","multi-teacher distillation","cross-vocabulary alignment","model merging"],"falsifier":"One decisive check: take a fixed source model and a fixed pivot, permute the source's vocabulary labels consistently, and recompute the Equation (5) loss; a loss that is a true cross-vocabulary distance should be unchanged under relabeling, whereas an elementwise top-K comparison that does not sort or map tokens will change. Running the full InfiFusionu pipeline with source models that share almost no tokenizer vocabulary with the pivot would settle the same question at the benchmark level.","tokens_in":15723,"feed_emoji":"🧩","tokens_out":10600,"duration_ms":95652,"temperature":0.7,"pith_summary":"This paper tries to show that a single pivot language model can absorb the complementary skills of several domain-specialized models without retraining from scratch or requiring the models to share vocabularies. The recipe is to distill only the top-$K$ logits of each source model into the pivot, z-score normalizing those logits first, and to measure the teacher-student gap with a 1-Wasserstein distance that works across different vocabularies. On 11 benchmarks covering math, code, reasoning, instruction following, and text reasoning, the fused model scores 79.92 on average, above the pivot Phi-4 (75.08) and all three source models, while training in about 160 H800 GPU hours. The practical stakes are large: if this holds, model improvement can come from fusing existing specialized models cheaply rather than pretraining ever-larger models.","feed_headline":"Three LLMs fuse into one that beats all three across 11 benchmarks","feed_subtitle":"The fused model trains in 160 H800 GPU hours, about a ten-thousandth of typical pretraining cost.","key_machinery":"The load-bearing object is an enhanced Universal Logit Distillation loss. Universal Logit Distillation compares two models' next-token probability distributions that live on different vocabularies by sorting each distribution's probabilities and taking the 1-Wasserstein distance between the sorted lists; InfiFusion modifies this by keeping only the top-$K$ logits and standardizing them, subtracting the mean and dividing by the standard deviation, before taking the elementwise $\\ell^1$ difference. This makes the distillation loss insensitive to logit scale and to the long tail of low-probability tokens, and it lets one pivot model be trained against several sources at once. The framework also contributes two fusion pipelines: pairwise fusion, which trains one pivot copy per source and then merges the copies with task arithmetic, TIES, or SCE; and unified fusion, which sums the per-source losses into a single training run.","core_discovery":"The paper's central claim is that the main obstacle to fusing models with different tokenizers and training histories, namely that their next-token distributions live on different vocabularies, can be sidestepped by comparing only the most probable logits after standardizing them. InfiFusion takes a pivot model and distills each source model's knowledge into it by minimizing, at each time step, the elementwise $\\ell^1$ difference between the pivot's and the source's top-$K$ logits, after each has been z-score normalized, plus a supervised fine-tuning loss on the same data. In unified fusion, the losses from all sources are summed and one training run produces the final model; in pairwise fusion, each source is distilled separately and the resulting models are merged by parameter arithmetic. The same 11-benchmark average is reached by both strategies, with unified fusion using about one third of the GPU hours. The paper concludes that the combination of Universal Logit Distillation, top-$K$ selection, and logits standardization is what makes cross-model fusion work.","pith_inferences":["Beyond the reported benchmarks, the same top-$K$ standardized-logit loss should apply to fusing more than three sources or to distilling from a much larger teacher into a small pivot, since the per-source cost is additive and the authors' own ablations show gains from additional sources and data.","A direct test of the method's mechanism would be to fuse source models deliberately chosen to have almost no tokenizer overlap with the pivot; the gains would either persist if the standardized sorted-logit distance is doing the work or shrink if hidden token alignment is responsible.","Because logits are z-score standardized, the loss is invariant to each model's output scale, which suggests InfiFusion could also serve as a calibration-agnostic distiller, transferring ranking knowledge between models with very different confidence distributions, an effect the paper does not measure.","The 160 GPU-hour figure excludes the 1.5 TB of pre-extracted teacher hidden states; at much larger data scales, storage and I/O rather than GPU time may become the true cost of fusion, so the efficiency advantage should be quoted with that preprocessing included."],"forward_implications":["If InfiFusion's result holds, a small pivot model can surpass each of its equal-sized or larger specialists on aggregate benchmarks, so model capability can be pooled without merging parameters directly.","Unified fusion reaches nearly the same average score as the more expensive pairwise-then-merge route while using about $1/S$ of the GPU hours, with $S=3$, suggesting simultaneous multi-teacher distillation is the cheaper route to the same capability.","Because the distance is computed on standardized logits, source models do not need to share vocabularies or architectures with the pivot, which widens the set of models that can be fused.","The ablations tie performance to top-$K$ selection, best at $K=10$, and to logits standardization, indicating that noise from low-probability logits and scale differences are the main things that hurt naive logit distillation.","The same training recipe benefits from more fusion data, with 144k samples beating 50k, so further gains are expected from scaling the distillation dataset rather than from more compute per sample."],"supporting_citations":[{"why":"Defines the Universal Logit Distillation loss, the 1-Wasserstein-based cross-vocabulary objective that InfiFusion enhances.","marker":"(Cui et al., 2024)"},{"why":"Introduced the universal logit distillation framework for cross-tokenizer distillation, which the paper names as the approach it extends.","marker":"(Boizard et al., 2025)"},{"why":"Supplies the logits standardization technique used to normalize logit scales before distillation.","marker":"(Sun et al., 2024)"},{"why":"FuseChat, the pairwise-fusion-plus-merging baseline and the source of the model-merging step used in InfiFusion's pairwise strategy.","marker":"(Wan et al., 2024c)"},{"why":"FuseLLM, the distribution-fusion baseline whose MinCE loss the paper compares against its own fusion loss.","marker":"(Wan et al., 2024b)"},{"why":"Phi-4 technical report; Phi-4 is the pivot model for all reported fusion results.","marker":"(Abdin et al., 2024)"},{"why":"Qwen2.5 technical report; provides two of the three source models fused into the pivot.","marker":"(Qwen et al., 2025)"}],"fun_headline_variants":["Three LLMs, one model: beats all on 11 benchmarks","Logit distillation fuses specialists, tops 11 benchmarks","160 GPU hours: fused LLM outranks each source on 11 tests","Top-K logits standardize, cross-model fusion wins 11 benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the elementwise difference between top-K logits from models with different vocabularies is a meaningful distance, yet the paper never states how the K tokens from each vocabulary are matched before subtraction.","fun_headline_variants_meta":{"raw":{"variants":["Three LLMs, one model: beats all on 11 benchmarks","Logit distillation fuses specialists, tops 11 benchmarks","160 GPU hours: fused LLM outranks each source on 11 tests","Top-K logits standardize, cross-model fusion wins 11 benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000361,"raw_usage":{"total_tokens":1966,"prompt_tokens":980,"completion_tokens":986,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":909}},"tokens_in":596,"tokens_out":986,"duration_ms":9702,"temperature":1.0,"reasoning_tokens":909,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:04:18.878343+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One decisive check: take a fixed source model and a fixed pivot, permute the source's vocabulary labels consistently, and recompute the Equation (5) loss; a loss that is a true cross-vocabulary distance should be unchanged under relabeling, whereas an elementwise top-K comparison that does not sort or map tokens will change. Running the full InfiFusionu pipeline with source models that share almost no tokenizer vocabulary with the pivot would settle the same question at the benchmark level.","supporting_citations":[],"review_version":1}