Pith. sign in

REVIEW 3 major objections 4 minor 5 cited by

Training Sparse Mixture Of Experts Text Embedding Models

T0 review · 3 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Sparse upcycling yields a text embedder that beats dense models of equal size

desk verdict The MoE-for-embeddings direction is real and the controlled ablations support it, but the headline outperformance claim is contradicted by the paper's own Table 1 and by disclosed in-domain training on BEIR and MIRACL splits — fixable in revision, not a desk-reject. read the letter →

arxiv 2502.07972 v3 pith:236RC4Y3 submitted 2025-02-11 cs.CL cs.AIcs.IR

Zach Nussbaum , Brandon Duderstadt This is my paper · ORCID
classification cs.CLcs.AIcs.IR
keywords mixtureofexpertstextembeddingsinformationretrievalmultilingualembeddingBEIRMIRACLsparseupcyclingcontrastivelearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces Nomic Embed v2, which it calls the first general-purpose Mixture of Experts (MoE) text embedding model, and argues that scaling embeddings with sparse experts is a workable alternative to scaling dense parameters. The model has 475M total parameters but only 305M active during inference, and the paper reports that it beats same-size dense models on the BEIR and MIRACL retrieval benchmarks while staying competitive with models roughly twice its size. The motivation is practical: large dense embedders strain memory and latency budgets, especially for retrieval-augmented generation, where they limit how much text can be indexed and slow down queries. The paper's claim is that MoE upcycling delivers the retrieval quality of a larger model without the serving cost, and it releases the code, models, and evaluation data to back that up.

What carries the argument

The load-bearing object is the upcycled MoE biencoder. Starting from the authors' mNomic-BERT (an XLM-RoBERTa Base whose absolute position encodings are replaced with RoPE for long contexts), every alternate MLP layer is converted into a sparse MoE layer with eight expert networks and top-2 token routing, following the sparse upcycling method. An auxiliary load-balancing loss keeps the experts from collapsing onto a few routes, and the model is trained in two contrastive stages: InfoNCE pretraining on 1.6 billion filtered query-document pairs, then finetuning with positive-aware hard negatives and Matryoshka representation learning so the same model serves 768- and 256-dimension embeddings. This combination is what lets the model keep dense-scale capacity while activating only about 64% of its parameters.

What would settle it

Train a dense control model with the identical architecture minus MoE (or with the MoE layers replaced by dense MLPs) on the same filtered 1.6B pairs and the same BEIR/MIRACL finetuning data, and compare nDCG@10 at equal active parameters; if the dense control matches or beats the MoE model, the claimed MoE advantage collapses. Separately, measure actual GPU memory and query latency of Nomic Embed v2 versus a dense baseline like mGTE Base; if the MoE model is not faster or lighter in practice, the deployment claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a sparse upcycled encoder — built by taking a long-context multilingual XLM-RoBERTa and replacing every other MLP block with a top-2 routed layer of eight experts — produces a text embedder that outperforms dense models in its parameter class and roughly matches models twice its size on both English and multilingual retrieval. The reported numbers are a BEIR average nDCG@10 of 52.86 and a MIRACL average of 65.80 at 305M active parameters, versus 48.88/62.30 for mE5 Base and 51.10/63.40 for mGTE Base, and against larger models like mE5 Large (51.40/66.50) and Arctic Embed v2 Large (55.65/66.00). The paper frames this as the first successful adaptation of MoE to general text embeddings.

Load-bearing premise

The benchmark comparison is fair: the model is finetuned on the training splits of BEIR and MIRACL and then reported on those same benchmarks, so the gains over baselines may partly come from seeing this data rather than from the MoE architecture.

Editorial extensions

If this is right

  • If the central claim holds, MoE upcycling gives embedding practitioners a scaling axis that does not multiply inference cost, so retrieval quality per active parameter can improve without bigger dense models.
  • The released pipeline shows that a competitive multilingual embedder can be built from publicly available data (mC4, CC News, and public finetuning sets), without private web search data.
  • The 256-dimension output (BEIR average 49.63) stays close to the full 768-dimension version, so the model can serve cheap storage-heavy retrieval settings with limited quality loss.
  • The layer ablation — upcycling 6 of 12 layers beats upcycling all 12 — indicates that selective sparsity is a better design point than full MoE conversion for this training regime.
  • The multilingual ablation shows MoE beats its dense base on BEIR and MIRACL at large batch sizes, but the gains shrink at smaller batches, so the recipe's benefits are tied to training scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A careful reader should weigh the benchmark protocol: the paper finetunes on the training splits of BEIR and MIRACL and reports nDCG@10 on those benchmark families, so part of the reported lead over baselines may come from seeing the evaluation tasks in training rather than from the MoE architecture itself.
  • The deployment advantage is asserted rather than measured: the paper does not report wall-clock latency, throughput, or GPU memory for Nomic Embed v2 versus its dense baselines, so the 'cheaper to serve' story needs a direct benchmark.
  • The analysis that MoE approaches a 3x-larger dense model only at large batch sizes suggests the efficiency claim may depend on training data volume; a testable extension is scaling the same upcycling recipe to fewer pretraining pairs and checking where the MoE advantage disappears.
  • Distilling the MoE embedder back into a dense model, which the paper lists as future work, would be the natural stress test of whether the experts contribute learnable knowledge or mostly act as a wider MLP.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces Nomic Embed v2, a multilingual text embedding model built by upcycling alternating MLP layers of a modified XLM-RoBERTa into sparse MoE layers (8 experts, top-2 routing), trained with weakly supervised contrastive pretraining followed by contrastive finetuning with hard negatives and Matryoshka output dimensions. The central claims are that this is the first general-purpose MoE text embedding model, that it outperforms same-parameter-class models on BEIR and MIRACL, and that it remains competitive with models twice its size. The paper also reports a controlled monolingual ablation suggesting MoE upcycling helps retrieval accuracy relative to a dense baseline of similar active parameter count.

Significance. If the headline results were cleanly established, MoE upcycling would be a valuable efficiency/capacity trade-off for embedding models, and the paper's open release of code, models, and evaluation data would support reproducibility. The controlled ablation in Section 7.1 provides some evidence for the architectural benefit. However, the main empirical comparison is compromised by in-domain finetuning on the evaluation benchmarks, and the abstract's performance claim is internally contradicted by the paper's own Table 1. The central 'outperforms' claim is therefore not supported by the current evidence.

major comments (3)
  1. [Appendix B (Table 11) and Section 5.3] The model is finetuned on the training splits of BEIR datasets (MSMARCO, HotpotQA, NQ, FEVER) and on MIRACL training data for 16 of 18 languages, as stated in Appendix B and Table 11. Section 5.3 then reports nDCG@10 on BEIR and MIRACL. This means the evaluation is not zero-shot for those tasks/languages, and the reported gains over baselines that were not trained on these splits may reflect in-domain adaptation rather than any benefit of the MoE architecture. This undermines the abstract's claim of general-purpose outperformance.
  2. [Abstract and Table 1] The abstract states that the model 'outperforms models in the same parameter class on both monolingual and multilingual benchmarks,' but Table 1 shows Arctic Embed v2 Base (305M parameters) achieves a BEIR score of 55.40 versus Nomic Embed v2's 52.86. Section 6.3 acknowledges this exception, but the abstract, introduction, and conclusion (Section 8) restate the unqualified outperformance claim, making it false as written.
  3. [Section 7.1 versus Section 4.3] The controlled ablation in Section 7.1 uses top-1 routing ('Switch Routing') and reports improvements for the MoE variant against dense Nomic BERT, while the final model Nomic Embed v2 uses top-2 routing (Section 4.3). The ablations do not directly test the configuration used in the headline evaluations, so they cannot rescue the benchmark comparison from the contamination issue.
minor comments (4)
  1. [Table 1] The columns 'Pretrain Data', 'Finetune Data', and 'Code' use inconsistent capitalization ('Y es' appears to be a typographical artifact in the extracted text; please ensure 'Yes' is consistently capitalized).
  2. [Figure 1] Figure 1 is referenced in Section 7.1 but the text does not display the figure; please ensure the figure is included in the submission.
  3. [Table 9] Table 9 reports both k=1 and k=2 variants, but the main model description in Section 4.3 specifies top-2 routing; the relationship between the ablation and the final model should be clarified.
  4. [References] Several reference entries have incomplete URLs (e.g., 'URL https://' without the full address); please verify and complete all references.

Circularity Check

1 steps flagged · score 6.0 of 10

BEIR/MIRACL train splits in the finetuning mix make the headline benchmark comparisons partially circular.

  1. fitted input called prediction [Appendix B / Table 11; Section 5.3]
    "Appendix B: "We train on the training sets of BEIR and MIRACL as well as SQuAD and Stackoverflow." Section 5.3: "We evaluate our model on two retrieval benchmarks: (1) BEIR ... and (2) MIRACL ...". Table 11 lists "MSMARCO 485,120", "NQ 57,856", "FEVER 28,672", and "MIRACL Train Datasets" for 16 languages."

    The reported BEIR and MIRACL nDCG@10 scores are not out-of-sample predictions: the contrastive finetuning data in Table 11 includes the training splits of four BEIR tasks (MSMARCO, NQ, HotpotQA, FEVER) and MIRACL train sets for 16 of the 18 evaluated languages, and Section 5.3 then scores the model on those same benchmark families. The headline claim that Nomic Embed v2 outperforms same-parameter-class models on BEIR and MIRACL is therefore partly a consequence of in-domain training rather than an independent test of the MoE architecture, because the comparison baselines were not finetuned on these benchmark train splits. The 'outperforms' result reduces, to a substantial degree, to the fact that the benchmark data were included in the model's own training mix.

full rationale

The central benchmark comparison is partially circular: the paper finetunes on BEIR and MIRACL training splits (Appendix B, Table 11) and then evaluates on those same benchmark families (Section 5.3, Tables 1 and 5), so the reported gains over baselines are not clean evidence for the MoE architecture. The controlled monolingual and multilingual ablations in Sections 7.1 and 7.2 use separate weakly-supervised pretraining data and do provide independent support for the narrower claim that MoE upcycling helps dense embeddings, which prevents a higher circularity score. Separately, the abstract's claim of outperforming all same-class models is internally contradicted by Table 1, where Arctic Embed v2 Base scores 55.40 on BEIR versus Nomic Embed v2's 52.86, but that is a consistency error rather than a circularity. Overall, the headline outperformance claim is partially reduced by construction because the evaluation benchmarks were part of the training input.

Assumptions & free parameters 10 free parameters · 5 assumptions · 0 invented entities

The ledger is dominated by training hyperparameters and benchmark protocol choices rather than mathematical axioms. The key non-standard assumption is that finetuning on the evaluation benchmarks' training splits is an acceptable way to support a general-purpose outperformance claim. The model itself is a trained artifact, not a newly postulated physical or theoretical entity.

free parameters (10)
  • InfoNCE temperature tau = 0.02
    Set in Section 4.3; controls contrastive sharpness and affects learned embedding geometry.
  • Load balancing loss coefficient alpha = 1.0
    Applied with Equation 1 in Section 4.3; strength of the auxiliary MoE balancing loss.
  • Number of experts and routing top-k = 8 experts, top-2
    Section 4.3; routing configuration determines active parameter count and model quality.
  • Number of upcycled layers = 6 of 12 alternating layers
    Section 4.3 with ablation in Table 7; selective upcycling is a hand-chosen design decision that affects performance.
  • Consistency filtering top-k threshold = top 2 documents
    Section 4.2; determines which weakly supervised pairs are retained, affecting the pretraining dataset.
  • Hard negative mining margin = 0.95
    Section 4.4 and Equation 5; margin threshold below which negatives are accepted, tuned in Table 8.
  • Number of hard negatives per query = 10
    Section 4.5; chosen from the ablation in Table 8, with diminishing gains beyond 4-7 negatives.
  • Pretraining sequence lengths = 32 query tokens, 256 document tokens
    Section 4.3; computational constraint choices that affect what the contrastive model learns.
  • Matryoshka output dimensions = [768, 256]
    Section 4.5; training the model to produce effective embeddings at two dimensions affects flexibility and quality.
  • RoPE base parameter = 10,000
    Section 4.1; chosen after experiments showed degraded GLUE and XTREME-R performance with larger values.
assumptions (5)
  • standard math InfoNCE contrastive loss and the auxiliary MoE load-balancing loss are valid differentiable training objectives.
    Used in Equations 1-4; these are standard objectives from the cited literature.
  • domain assumption Weakly supervised contrastive pretraining followed by hard-negative finetuning transfers to retrieval benchmarks.
    This is the standard two-stage embedding training paradigm assumed throughout Sections 3.3 and 4.
  • domain assumption mNomic-BERT, after RoPE replacement and 10,000 MLM steps, preserves enough XLM-RoBERTa multilingual quality for downstream contrastive training.
    Section 4.1 and Tables 3-4 support this with GLUE and XTREME-R results, but the downstream claim relies on it.
  • ad hoc to paper Upcycling alternate MLP layers of a dense checkpoint into 8-expert top-2 MoE layers is an efficient capacity-quality trade-off.
    Section 4.3 and the ablation in Table 7 motivate this choice, but it is not derived from first principles.
  • ad hoc to paper Evaluating on BEIR and MIRACL after finetuning on their training splits yields fair general-purpose comparisons.
    Appendix B and Table 11 show the finetuning data includes BEIR and MIRACL training sets, while Section 5.3 evaluates on those benchmark families; this assumption is the main threat to the headline claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Training Sparse Mixture Of Experts Text Embedding Models." pith.science (2026). https://pith.science/paper/236RC4Y3

@misc{pith2026250207972,
  author       = {Pith},
  title        = {Pith review of: Training Sparse Mixture Of Experts Text Embedding Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/236RC4Y3}},
  note         = {Machine review of arXiv:2502.07972}
}
read the original abstract

Transformer-based text embedding models have improved their performance on benchmarks like MIRACL and BEIR by increasing their parameter counts. However, this scaling approach introduces significant deployment challenges, including increased inference latency and memory usage. These challenges are particularly severe in retrieval-augmented generation (RAG) applications, where large models' increased memory requirements constrain dataset ingestion capacity, and their higher latency directly impacts query-time performance. While causal language models have addressed similar efficiency challenges using Mixture of Experts (MoE) architectures, this approach hasn't been successfully adapted to the general text embedding setting. In this paper, we introduce Nomic Embed v2, the first general purpose MoE text embedding model. Our model outperforms models in the same parameter class on both monolingual and multilingual benchmarks while also maintaining competitive performance with models twice its size. We open-source all code, models, and evaluation data to ensure full reproducibility of our training pipeline at \href{https://github.com/nomic-ai/contrastors}{https://github.com/nomic-ai/contrastors}.

Figures

Figures reproduced from arXiv: 2502.07972 by the authors.

Figure 1
Figure 1. Impact of Model Size and Batch Size on Retrieval Performance. NDCG@10 scores on BEIR benchmark across different batch sizes and model architectures. The upcycled MoE model’s performance approaches that of a model with 3x more active parameters as batch size increases, demonstrating efficient scaling behavior. 7.2. Effectiveness of MoEs for Multilingual Text Embeddings We extend our analysis to the multilingual setti… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech

    cs.CL 2026-03 accept novelty 7.0 of 10

    Progressive LLM-initialized training plus teacher-student distillation yields omnilingual sentence embeddings that halve FLORES error and cut BIBLE error 15× while supporting speech and zero-shot Spectrum transfer.

  2. A Comparative Evaluation of Embeddings and LLMs in a Greek Book Publisher Setting - The CUP Dataset

    cs.CL 2026-07 conditional novelty 6.0 of 10

    CUP, a 104-query Greek book retrieval benchmark, shows hybrid lexical-semantic retrieval outperforms both BM25 and dense-only methods in a real publisher catalog.

  3. Continuous Online Evaluation of Recommendation Strategies in Social Science Academic Search

    cs.IR 2026-07 conditional novelty 5.0 of 10

    In a live test on the GESIS social-science search engine, semantic-embedding recommenders beat term-matching and click-session models, with dataset and variable search favoring simpler matching.

  4. MARS: Multi-hop Adaptive Retrieval and SPARQL Generation for KGQA

    cs.CL 2026-07 conditional novelty 5.0 of 10

    MARS answers multi-hop knowledge-graph questions by iteratively retrieving ranked triple patterns and letting an LLM decide when to emit a SPARQL query, beating agentic baselines on QALD-10 without fine-tuning.

  5. Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generation

    cs.IR 2025-06 conditional novelty 5.0 of 10

    A systematic evaluation shows that recursive 100-token non-overlapping chunks and retrieval-tuned embeddings outperform fixed-size chunks and domain-specific models like SciBERT for chemistry retrieval, and it introdu...

Reference graph

Works this paper leans on

30 extracted references · 3 canonical work pages · cited by 5 Pith papers

  1. [4]

    Gumma, V ., Chitale, P

    URL https://arxiv.org/abs/ 2211.15841. Gumma, V ., Chitale, P. A., and Bali, K. Towards induc- ing document-level abilities in standard multilingual neu- ral machine translation models,

    arXiv

  2. [5]

    Towards Inducing Long-Context Abilities in Multilingual Neural Machine Translation Models

    URL https: //arxiv.org/abs/2408.11382. G¨unther, M., Milliken, L., Geuter, J., Mastrapas, G., Wang, B., and Xiao, H. Jina embeddings: A novel set of high- performance sentence embedding models,

    work page Pith review arXiv

  3. [6]

    Contrastive Learning and Mixture of Experts Enables Precise Vector Embeddings

    URL https://arxiv.org/abs/2401.15713. Hochreiter, S. and Schmidhuber, J. Long Short-Term Mem- ory. Neural Computation , 9(8):1735–1780, 11

    work page Pith review arXiv

  4. [9]

    Kusupati, A., Bhatt, G., Rege, A., Wallingford, M., Sinha, A., Ramanujan, V ., Howard-Snyder, W., Chen, K., Kakade, S., Jain, P., and Farhadi, A

    URL https://arxiv.org/abs/2402.07871. Kusupati, A., Bhatt, G., Rege, A., Wallingford, M., Sinha, A., Ramanujan, V ., Howard-Snyder, W., Chen, K., Kakade, S., Jain, P., and Farhadi, A. Matryoshka represen- tation learning,

    arXiv

  5. [10]

    Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Kel- cey, M., Devlin, J., Lee, K., Toutanova, K

    URL https://arxiv.org/ abs/2205.13147. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Kel- cey, M., Devlin, J., Lee, K., Toutanova, K. N., Jones, L., Chang, M.-W., Dai, A., Uszkoreit, J., Le, Q., and Petrov, S. Natural questions: a benchmark for question answering research. Transactions of t...

    arXiv

  6. [12]

    URL https://arxiv.org/abs/ 2006.16668. Li, Z. and Zhou, T. Your mixture-of-experts llm is secretly an embedding model for free,

    arXiv 2006

  7. [13]

    Li, Z., Zhang, X., Zhang, Y ., Long, D., Xie, P., and Zhang, M

    URL https:// arxiv.org/abs/2410.10814. Li, Z., Zhang, X., Zhang, Y ., Long, D., Xie, P., and Zhang, M. Towards general text embeddings with multi-stage contrastive learning,

    arXiv

  8. [14]

    Liu, Y ., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V

    URL https://arxiv.org/abs/2310.05209. Liu, Y ., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V . Roberta: A robustly optimized bert pretraining approach,

    arXiv

Show all 30 references
  1. [15]

    Moreira, G

    URL https://arxiv.org/abs/ 2405.05374. Moreira, G. d. S. P., Osmulski, R., Xu, M., Ak, R., Schif- ferer, B., and Oldridge, E. Nv-retriever: Improving text embedding models with effective hard-negative mining. arXiv preprint arXiv:2407.15831,

    arXiv

  2. [16]

    Nussbaum, Z., Morris, J

    URL https://arxiv.org/ abs/2402.09906. Nussbaum, Z., Morris, J. X., Duderstadt, B., and Mulyar, A. Nomic embed: Training a reproducible long context text embedder,

    arXiv

  3. [17]

    Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P

    URL https://arxiv.org/abs/ 2402.01613. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. SQuAD: 100,000+ questions for machine comprehension of text. In Su, J., Duh, K., and Carreras, X. (eds.), Proceed- ings of the 2016 Conference on Empirical Methods in Natural Language P...

    2016 arXiv

  4. [19]

    URL https://arxiv.org/abs/2004. 09813. 9 Training Sparse Mixture Of Experts Text Embedding Models Ruder, S., Constant, N., Botha, J., Siddhant, A., Firat, O., Fu, J., Liu, P., Hu, J., Garrette, D., Neubig, G., and Johnson, M. Xtreme-r: Towards more challenging and nuanced mult...

    2004

  5. [22]

    Text embeddings by weakly- supervised contrastive pre-training, 2024a

    Wang, L., Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., Majumder, R., and Wei, F. Text embeddings by weakly- supervised contrastive pre-training, 2024a. URL https: //arxiv.org/abs/2212.03533. Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R., and Wei, F. Multilingual...

    arXiv

  6. [23]

    org/abs/2309.07597

    URL https://arxiv. org/abs/2309.07597. Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., Khabsa, M., Fang, H., Mehdad, Y ., Narang, S., Malik, K., Fan, A., Bhosale, S., Edunov, S., Lewis, M., Wang, S., and...

    arXiv

  7. [24]

    org/abs/2309.16039

    URL https://arxiv. org/abs/2309.16039. Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. mt5: A massively multilingual pre-trained text-to-text transformer,

    arXiv

  8. [25]

    Yu, P., Merrick, L., Nuti, G., and Campos, D

    URL https://arxiv.org/abs/2010.11934. Yu, P., Merrick, L., Nuti, G., and Campos, D. Arctic-embed 2.0: Multilingual retrieval without compromise,

    2010 arXiv

  9. [26]

    Zhang, D., Li, J., Zeng, Z., and Wang, F

    URL https://arxiv.org/abs/2412.04506. Zhang, D., Li, J., Zeng, Z., and Wang, F. Jasper and stella: distillation of sota embedding models,

    arXiv

  10. [28]

    Zhang, X., Zhang, Y ., Long, D., Xie, W., Dai, Z., Tang, J., Lin, H., Yang, B., Xie, P., Huang, F., Zhang, M., Li, W., and Zhang, M

    URL https://arxiv.org/abs/2210.09984. Zhang, X., Zhang, Y ., Long, D., Xie, W., Dai, Z., Tang, J., Lin, H., Yang, B., Xie, P., Huang, F., Zhang, M., Li, W., and Zhang, M. mgte: Generalized long-context text representation and reranking models for multilingual text retrieval,

    arXiv

  11. [29]

    Zoph, B., Bello, I., Kumar, S., Du, N., Huang, Y ., Dean, J., Shazeer, N., and Fedus, W

    URL https://arxiv.org/abs/ 2407.19669. Zoph, B., Bello, I., Kumar, S., Du, N., Huang, Y ., Dean, J., Shazeer, N., and Fedus, W. St-moe: Designing stable and transferable sparse expert models,

    arXiv

  12. [30]

    10 Training Sparse Mixture Of Experts Text Embedding Models A

    URL https: //arxiv.org/abs/2202.08906. 10 Training Sparse Mixture Of Experts Text Embedding Models A. Weakly Supervised Contrastive Pretraining Dataset Distribution The full pretraining dataset distribution can be see in Table A. Table

    arXiv

  13. [1997]

    doi: 10.1162/neco.1997.9.8.1735

    ISSN 0899-7667. doi: 10.1162/neco.1997.9.8.1735. 8 Training Sparse Mixture Of Experts Text Embedding Models URL https://doi.org/10.1162/neco.1997. 9.8.1735. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., La...

    1997 doi

  14. [2016]

    doi: 10.18653/v1/D16-1264

    Association for Computational Linguistics. doi: 10.18653/v1/D16-1264. URL https: //aclanthology.org/D16-1264. Reimers, N. and Gurevych, I. Sentence-bert: Sentence embeddings using siamese bert-networks,

    doi

  15. [2017]

    Su, H., Shi, W., Kasai, J., Wang, Y ., Hu, Y ., Ostendorf, M., tau Yih, W., Smith, N

    URL https://arxiv.org/abs/1701.06538. Su, H., Shi, W., Kasai, J., Wang, Y ., Hu, Y ., Ostendorf, M., tau Yih, W., Smith, N. A., Zettlemoyer, L., and Yu, T. One embedder, any task: Instruction-finetuned text embeddings, 2023a. Su, J., Lu, Y ., Pan, S., Murtadha, A., Wen, B., an...

    arXiv

  16. [2019]

    Nv-embed: Improved techniques for training llms as generalist embedding models

    Lee, C., Roy, R., Xu, M., Raiman, J., Shoeybi, M., Catan- zaro, B., and Ping, W. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint arXiv:2405.17428, 2024a. Lee, C., Roy, R., Xu, M., Raiman, J., Shoeybi, M., Catan- zaro, B., and Ping...

    arXiv

  17. [2020]

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K

    URL https: //arxiv.org/abs/1911.02116. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding,

    1911 arXiv

  18. [2021]

    org/abs/2104.07412

    URL https://arxiv. org/abs/2104.07412. Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural net- works: The sparsely-gated mixture-of-experts layer,

    arXiv

  19. [2022]

    Gale, T., Narayanan, D., Young, C., and Zaharia, M

    URL https://arxiv.org/ abs/2101.03961. Gale, T., Narayanan, D., Young, C., and Zaharia, M. Megablocks: Efficient sparse training with mixture-of- experts,

    arXiv

  20. [2023]

    Krajewski, J., Ludziejewski, J., Adamczewski, K., Pi ´oro, M., Krutul, M., Antoniak, S., Ciebiera, K., Kr ´ol, K., Odrzyg´o´zd´z, T., Sankowski, P., Cygan, M., and Jaszczur, S

    URL https: //arxiv.org/abs/2212.05055. Krajewski, J., Ludziejewski, J., Adamczewski, K., Pi ´oro, M., Krutul, M., Antoniak, S., Ciebiera, K., Kr ´ol, K., Odrzyg´o´zd´z, T., Sankowski, P., Cygan, M., and Jaszczur, S. Scaling laws for fine-grained mixture of experts,

    arXiv

  21. [2024]

    Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V ., Wenzek, G., Guzm ´an, F., Grave, E., Ott, M., Zettle- moyer, L., and Stoyanov, V

    URL https:// arxiv.org/abs/2402.03216. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V ., Wenzek, G., Guzm ´an, F., Grave, E., Ott, M., Zettle- moyer, L., and Stoyanov, V . Unsupervised cross-lingual representation learning at scale,

    arXiv

  22. [2025]

    Zhang, X., Thakur, N., Ogundepo, O., Kamalloo, E., Alfonso-Hermelo, D., Li, X., Liu, Q., Rezagholizadeh, M., and Lin, J

    URL https://arxiv.org/abs/2412.19048. Zhang, X., Thakur, N., Ogundepo, O., Kamalloo, E., Alfonso-Hermelo, D., Li, X., Liu, Q., Rezagholizadeh, M., and Lin, J. Making a miracl: Multilingual infor- mation retrieval across a continuum of languages,

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.