{"id":"85595177-90bc-425e-a5e8-0bef4ddf90ab","arxiv_id":"2608.02583","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"UEmbed uses 16 special tokens over a partitioned vocabulary to make a decoder-only multimodal model emit dense and sparse embeddings in one forward pass; the 9B model scores 71.8 dense / 71.0 sparse on MMEB-v2.","lead":"A decoder-only multimodal language model now produces both dense and sparse search embeddings in one forward pass, using 16 special tokens that each score a slice of the vocabulary. On the 78-task MMEB-v2 benchmark, its 9B model scores 71.8 dense and 71.0 sparse, within one point of each other.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The dense/sparse parity claim rests on single-seed numbers that the paper's own appendix appears to contradict; multi-seed runs are needed before the 'within one point' claim can be trusted.","rationale":"I read UEmbed in good faith: the architecture is coherent, the evaluation is broad, and the ablations support the design choices. The central claim, however, is quantitative and comparative: dense and sparse scores are within one point at every scale, and sparse retrieval is state of the art. That claim is supported only by single-seed runs. The reader flagged single-seed numbers as a secondary issue and placed more weight on the representational-capacity assumption of N=16 readout tokens. I agree that capacity is a real modeling bet, but the paper provides ablation sweeps over N and partitioning that make the bet empirically reasonable. The more load-bearing concern is the fragility of the exact parity number: the paper's own appendix reports UEmbed-2B dense/sparse numbers that differ from the main table by roughly two points, larger than the claimed one-point gap. Without clarification or multi-seed evidence, a reader cannot distinguish a robust architectural property from a single-run artifact. This concern does not invalidate the method or the qualitative conclusion that sparse and dense modes are competitive, so the conditional verdict stands. The concrete test is a multi-seed rerun plus a clarification of the appendix's data source; that would settle whether the quantitative headline survives.","tokens_in":19202,"tokens_out":13621,"duration_ms":155571,"concrete_test":"Retrain UEmbed-2B and UEmbed-9B from scratch with at least three seeds (e.g., 42, 7, 2024) using the exact A.2 recipe and report MMEB-v2 dense/sparse aggregates per scale and per-category gaps. If any seed yields a dense-sparse gap >1.0 point at any scale, or if the dense/sparse ordering against RzenEmbed flips, the 'within one point' and sparse-SOTA claims are not robust. Also clarify in the paper whether Table 10/B.4 scores are from the full training mixture or the 500k-instance ablation subset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim is that UEmbed closes the dense-sparse gap to within one point at every scale on MMEB-v2 (abstract; §4.2). The supporting evidence is one training run per scale with seed fixed to 42 (A.2), and no variance or confidence estimate is reported. Appendix B.4, headed 'random seed 42', reports UEmbed-2B dense 64.6 / sparse 63.4 on MMEB-v2 (Table 10), whereas Table 1 reports dense 66.5 / sparse 65.5 for the same model configuration. That is a 1.9-point shift in the dense score, which is larger than the claimed dense-sparse gap at any scale (max 1.0). Either the appendix used a different training subset (e.g., the 500k-instance ablation mixture from §5.1) without saying so, or run-to-run variation is large enough to undermine the precise 'within one point' wording. The sparse-mode SOTA claim is also sensitive: UEmbed-9B sparse (71.0) is actually 0.1 points below RzenEmbed-V2-7B (71.1), so the ordering flips depending on whether one compares dense or sparse. This does not make the architecture internally inconsistent, but it does mean the headline parity result is not yet empirically established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes UEmbed, a family of decoder-only multimodal embedding models (2B, 4B, 9B) that produce both dense and learned-sparse representations in a single causal forward pass. Sparse representations are generated by appending N=16 learnable special tokens, each responsible for a disjoint k-means-clustered subset of a compressed vocabulary, and concatenating their per-subset sparse predictions. The dense representation is taken from the EOS hidden state. Training uses a combined InfoNCE objective with FLOPS regularizers on publicly available text, image, video, and visual-document data. On MMEB-v2, UEmbed-9B reaches 71.8 dense and 71.0 sparse, and the paper claims the dense-sparse gap is at most 1.0 point at all scales, establishing a new state of the art in sparse multimodal retrieval. On BEIR, UEmbed remains competitive with dense and sparse baselines. Additional experiments analyze joint-training robustness, vocabulary partitioning, sparse temperature, the number of special tokens, hybrid scoring, serving efficiency, and agentic-search cost.","tokens_in":19567,"tokens_out":3795,"duration_ms":37513,"significance":"If the claims hold, the architecture is a meaningful advance: it is one of the first decoder-only models to natively support both dense and sparse retrieval for multimodal inputs, eliminates auxiliary cross-modal modules, and is compatible with high-throughput causal serving stacks and inverted indices. The paper's strengths include extensive evaluations on MMEB-v2 and BEIR, controlled ablations of each design choice, release of models at three scales, and explicit discussion of limitations (language bias, vocabulary artifacts, video gap). The central quantitative claim—that dense and sparse modes are within one point across scales—is the load-bearing result, and it is currently supported only by single-seed numbers that appear to be internally inconsistent with the paper's own appendix.","major_comments":[{"comment":"Table 1 reports UEmbed-2B dense 66.5 / sparse 65.5 on MMEB-v2, while Table 10 in B.4 reports dense 64.6 / sparse 63.4 for the same model name, and both the main training description (A.2) and the appendix header state random seed 42. The dense discrepancy is 1.9 points—larger than the claimed maximum dense-sparse gap of 1.0 point. The paper must either explicitly state that Table 10 uses the 500k-instance ablation mixture from §5.1 (and therefore is not directly comparable to Table 1), or, if the same training setup was used, the run-to-run variance is large enough to undermine the central 'within one point' parity claim. Please report multi-seed means and variances, or clearly label the appendix models as ablation-subset models.","section":"Appendix B.4, Table 10 vs. Table 1"},{"comment":"The abstract says UEmbed 'outperforming multimodal embedding models trained on publicly available data (e.g., RzenEmbed)'. This is true for dense mode (71.8 vs. 71.1 for RzenEmbed-V2-7B), but false for sparse mode: UEmbed-9B sparse scores 71.0, 0.1 below RzenEmbed-V2-7B (71.1). The body text in §4.2 carefully restricts the claim to dense mode, but the abstract's blanket wording is contradicted by Table 1. Please qualify the claim by mode, or revise the abstract to reflect the dense-only comparison.","section":"Abstract and §4.2, Table 1"},{"comment":"The central parity result relies on single-seed runs: A.2 fixes random seed 42, and no variance or confidence intervals are reported anywhere. In addition, the hyperparameters that most affect the sparse mode—N, sparse temperature tau_s, and the hybrid beta weights—were tuned on MMEB-v1 or held-out splits of the same benchmark family (B.3). This tuning dependence weakens the claim that the MMEB-v2 numbers are unbiased. Please report at least three seeds for the main models, or clearly separate tuned-ablations from final evaluations and justify the absence of variance estimates.","section":"§4.2 and §5.1"}],"minor_comments":[{"comment":"The SPLADE baseline shares the Qwen3.5 backbone but adopts bidirectional attention. Please explain how bidirectional attention is realized in this decoder-only backbone (e.g., full attention masking, weight conversion, or architectural modification), as this is essential for reproducing the +3.2/+2.1 gains.","section":"§5.1, Table 3"},{"comment":"The vocabulary compression reduces size from 248,320 to 184,016, but it is not stated which tokenizer/vocabulary this corresponds to (presumably the Qwen3.5 tokenizer). Please state the vocabulary source explicitly.","section":"§3.2"},{"comment":"The training-dynamics figure has a single y-axis labeled 'Metric Value' for three subplots with very different scales (loss, max-neg gap, mean-neg gap). Please use separate axes or normalize so the stabilization claim is visually interpretable.","section":"Figure 5"},{"comment":"The hybrid beta values are tuned per modality on a held-out split, but it is not stated whether the held-out split is from MMEB-v2, MMEB-v1, or another benchmark. Please specify, since this affects the validity of the hybrid-scoring results in Table 5.","section":"Appendix B.3"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important problem, and the proposed architecture is clean and well motivated. The main risk is the unresolved discrepancy between Table 1 and Appendix B.4, which directly bears on the paper's central quantitative claim. If the appendix numbers are from a different training setup, that must be stated prominently; otherwise, the within-one-point claim is not currently supported. The manuscript is otherwise a solid contribution with strong ablations and practical analysis, and the issues are fixable within a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is the real thing: append N special tokens to a decoder-only MLLM, partition the vocabulary with k-means, and let each token score its slice through a causal hidden state. That cleanly sidesteps the causal bottleneck that has kept learned sparse retrieval tied to encoders. The paper evaluates this at three scales, on MMEB-v2 and BEIR, with ablations over N, temperature, partitioning, and joint vs. single-mode training. The control against a bidirectional SPLADE baseline on the same backbone is exactly the right comparison, and the sparse results are genuinely strong—71.0 at 9B is the first sparse score on MMEB-v2, and it beats several dense open baselines. The hybrid scoring and agentic-search experiments are useful, not filler.\n\nThe problems are mostly about evidence quality, not architecture. The appendix (Table B.4, headed 'random seed 42') reports UEmbed-2B dense 64.6 / sparse 63.4 on MMEB-v2, while Table 1 reports 66.5 / 65.5 for the same configuration. That 1.9-point dense shift is larger than the claimed dense-sparse gap at any scale. Either the appendix used a different training subset without saying so, or run-to-run variation is big enough to make 'within one point at every scale' a statement about one seed, not a robust result. The paper fixes seed 42, but that is not a guarantee of stability. Second, the abstract's 'outperforming RzenEmbed' is true only for the dense mode; the sparse 9B score (71.0) is actually 0.1 below RzenEmbed-V2-7B (71.1). That is a wording fix, but it matters. Third, the sparse head is under-specified: per-term weight vectors over a 184k vocabulary could be hundreds of millions of parameters, and the paper doesn't say whether this is a full matrix, a LoRA, or something else, nor what it costs at serving time. Finally, the promised release has no link to code or checkpoints anywhere in the text.\n\nThe capacity worry about 16 fixed slices is real but not fatal. The ablations show N=16 beats N=2 and N=32, and the SPLADE comparison suggests the partitioning plus causal readout is doing useful work. I'd want a capacity argument or at least a clearer analysis of what each slice learns, but the empirical case is plausible.\n\nOverall this is a solid contribution that deserves a serious referee and, with reasonable revisions, publication. The authors need to resolve the appendix discrepancy, report multi-seed variance or soften the parity wording, clarify the sparse-head parameterization, and point to actual artifacts. Fix those and I'd be happy to cite it.","headline":"Genuinely new idea—decoder-only MLLM emitting dense and sparse embeddings via partitioned special tokens—but the headline parity claim is undercut by a 1.9-point appendix discrepancy and single-seed numbers.","tokens_in":20070,"tokens_out":2398,"would_cite":true,"duration_ms":25148,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a decoder-only multimodal LLM, using 16 special tokens each covering a slice of the vocabulary, can produce sparse lexical and dense embeddings in one forward pass, with sparse performance within about one point of den","keywords":["unified embeddings","learned sparse retrieval","decoder-only","multimodal retrieval","dense retrieval","special tokens","vocabulary partition","MMEB-v2"],"falsifier":"Take the released 9B checkpoint and compute sparse vectors for a set of queries rich in rare or technical terms; compare the number of relevant activated terms against SPLADE-v3 on the same queries. If the fixed 16-slice head consistently misses terms SPLADE finds, the capacity bet is falsified. Alternatively, train a variant with N=64 while keeping the contrastive sequence length constant (e.g., by reusing input-token hidden states) and check whether sparse performance improves; if it does, the paper's N=16 choice is a capacity limit rather than a sweet spot.","tokens_in":19086,"feed_emoji":"🔍","tokens_out":5244,"duration_ms":50126,"temperature":0.7,"pith_summary":"The paper tries to establish that sparse lexical retrieval, historically tied to bidirectional encoder architectures, can be produced natively by a decoder-only multimodal model without auxiliary cross-modal modules. It achieves this by appending 16 learnable special tokens, each assigned a disjoint slice of the vocabulary via k-means clustering, and training the model to output a sparse vector that is nearly as effective as its dense vector. On MMEB-v2, the 9B model scores 71.8 (dense) and 71.0 (sparse), a gap under one point, and the paper reports the first sparse model evaluated on that benchmark. A sympathetic reader would care because, if right, sparse retrieval gains the efficiency and interpretability of lexical search while keeping semantic power, and it works across text, images, video, and documents in one backbone.","feed_headline":"Sparse retrieval matches dense within 1 point in a unified model","feed_subtitle":"A 9B model scores 71.8 dense and 71.0 sparse on MMEB-v2, making sparse retrieval practical for multimodal search.","key_machinery":"The partitioned sparse head: 16 learnable special tokens appended after the input, each assigned a disjoint subset of the (accent-stripped, lowercased, whitespace-collapsed) vocabulary via k-means clustering. Each token computes log(1 + ReLU(W_t^T h_s + b_t)) over its slice, and the slices are concatenated into the full sparse vector; the dense embedding is the hidden state of the EOS token immediately before the special tokens. This replaces SPLADE's max-pooling over all token states, which causal masking forbids, with a fixed set of causal readout positions.","core_discovery":"The central finding is that the information bottleneck of causal attention for sparse retrieval can be circumvented by partitioning the vocabulary into N disjoint subsets and assigning each to a dedicated special token at the end of the input. Each token's causal hidden state produces sparse weights only over its assigned slice, and the N slices are concatenated into the full sparse vector; the dense embedding comes from the EOS hidden state preceding the special tokens. Trained with a joint dense/sparse contrastive objective, UEmbed-9B reaches 71.8 (dense) and 71.0 (sparse) on MMEB-v2, closing the dense-sparse gap to within one point at every scale tested. The paper also shows that joint tr","pith_inferences":["The 16-token partition is a capacity bet: the paper's observed drop at N=32 may reflect inflated contrastive sequence lengths rather than a true optimum, so increasing N while controlling sequence length is a testable extension that could push sparse performance higher.","Because the dense mode is the hidden state of a single EOS token, it is a weaker representation than the ensemble of 16 sparse heads; stronger dense pooling could widen the sparse-dense gap again, so the reported parity may partly reflect an underpowered dense baseline.","The English-Chinese training distribution limits the sparse vocabulary's language coverage; a more language-diverse corpus would test whether the partition mechanism generalizes or collapses to a few dominant languages.","The smaller gain on video suggests a flat sparse vector has capacity limits for temporally dense data; hierarchical or multi-vector sparse representations are a natural next step the paper leaves implicit."],"forward_implications":["If the central claim holds, any causal LLM can be adapted to produce sparse retrieval vectors in a single forward pass, removing the need to re-train backbones as bidirectional encoders.","Hybrid scoring over the model's dense and sparse modes requires no extra encoding pass and measurably improves text and visual-document retrieval.","Multimodal sparse retrieval can be served from one backbone without auxiliary cross-modal modules, simplifying extension to new modalities.","The method scales consistently from 2B to 9B parameters, with sparse performance staying within one point of dense at every scale tested.","The sparse representation is natively compatible with inverted-index search and high-throughput autoregressive serving stacks, lowering deployment cost for lexical retrieval."],"fun_headline_variants":["Unified model nails sparse and dense embeddings within 1 point","Decoder-only model outputs sparse and dense in one pass","Sparse and dense embeddings unified in single causal pass","9B model closes dense-sparse gap to 0.8 points on MMEB-v2","One model, two embeddings: dense and sparse within 1 point"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The argument assumes that 16 causal readout tokens, each restricted to a fixed vocabulary slice, have enough representational capacity to replace the bidirectional max-pooling over all token states used by SPLADE; the paper offers empirical ablations but no capacity argument, so if 16 fixed slices cannot jointly express the needed sparse vector, sparse mode would degrade.","fun_headline_variants_meta":{"raw":{"variants":["Unified model nails sparse and dense embeddings within 1 point","Decoder-only model outputs sparse and dense in one pass","Sparse and dense embeddings unified in single causal pass","9B model closes dense-sparse gap to 0.8 points on MMEB-v2","One model, two embeddings: dense and sparse within 1 point"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000681,"raw_usage":{"total_tokens":2968,"prompt_tokens":818,"completion_tokens":2150,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":2059}},"tokens_in":562,"tokens_out":2150,"duration_ms":12909,"temperature":1.0,"reasoning_tokens":2059,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T04:13:01.417509+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the released 9B checkpoint and compute sparse vectors for a set of queries rich in rare or technical terms; compare the number of relevant activated terms against SPLADE-v3 on the same queries. If the fixed 16-slice head consistently misses terms SPLADE finds, the capacity bet is falsified. Alternatively, train a variant with N=64 while keeping the contrastive sequence length constant (e.g., by reusing input-token hidden states) and check whether sparse performance improves; if it does, the paper's N=16 choice is a capacity limit rather than a sweet spot.","supporting_citations":[],"review_version":1}