Pith. sign in

REVIEW 17 cited by

Jasper and Stella: distillation of SOTA embedding models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2412.19048 v2 pith:PQINM2ET submitted 2024-12-26 cs.IR

Dun Zhang , Jiacheng Li , Ziyang Zeng , Fulong Wang This is my paper
classification cs.IR
keywords embeddingmodeljaspermodelstexthttpsstudentdatasets
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

A crucial component in many deep learning applications, such as Frequently Asked Questions (FAQ) and Retrieval-Augmented Generation (RAG), is dense retrieval. In this process, embedding models transform raw text into numerical vectors. However, the embedding models that currently excel on text embedding benchmarks, like the Massive Text Embedding Benchmark (MTEB), often have numerous parameters and high vector dimensionality. This poses challenges for their application in real-world scenarios. To address this issue, we propose a novel multi-stage distillation framework that enables a smaller student embedding model to distill multiple larger teacher embedding models through three carefully designed losses. Meanwhile, we utilize Matryoshka Representation Learning (MRL) to reduce the vector dimensionality of the student embedding model effectively. Our student model named Jasper with 2 billion parameters, built upon the Stella embedding model, obtained the No.3 position on the MTEB leaderboard (as of December 24, 2024), achieving an average 71.54 score across 56 datasets. We have released the model and data on the Hugging Face Hub (https://huggingface.co/infgrad/jasper_en_vision_language_v1) (https://huggingface.co/datasets/infgrad/jasper_text_distill_dataset), and the training codes are available in this project repository (https://github.com/NLPJCL/RAG-Retrieval).

Discussion (0). Sign in to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Search

    cs.CL 2026-07 conditional novelty 7.0 of 10

    With matched open data and backbones, ColBERT-style late interaction turns English translate-train into multilingual generalization, while dense retrieval stays mostly inside the translated languages.

  2. SQuTR: A Robustness Benchmark for Spoken Query to Text Retrieval under Acoustic Noise

    cs.IR 2026-02 unverdicted novelty 7.0 of 10

    SQuTR is a large bilingual benchmark of 37,317 synthesized spoken queries under clean/low/medium/high noise, showing that retrieval quality steadily degrades as noise increases.

  3. Bifrost: Empowering Pretrained Language Model with Fallibility Representation for Log-Based Fault Diagnosis

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Bifrost fine-tunes BART-Base on logs with three self-supervised contrastive tasks, beating general PLMs by roughly 10-20% on anomaly detection, root-cause localization, and fault identification.

  4. CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matching

    cs.AI 2026-02 conditional novelty 6.0 of 10

    A new benchmark and training strategy show LLMs trained to internalize causal reasoning steps are less fooled by semantically similar, label-flipped questions than models using explicit chain-of-thought.

  5. THEME: Enhancing Thematic Investing with Semantic Stock Representations and Temporal Dynamics

    q-fin.PM 2025-08 conditional novelty 6.0 of 10

    A hierarchical contrastive learning framework that aligns stocks with theme descriptions and refines embeddings with short-term return signals improves thematic retrieval and backtested portfolio metrics.

  6. Exploiting Leaderboards for Large-Scale Distribution of Malicious Models

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A new attack framework, TrojanClimb, shows that adversaries can place models with embedded backdoors or biases on public leaderboards while retaining competitive rankings, across text embeddings, text generation, spee...

  7. Converting Annotated Clinical Cases into Structured Case Report Forms

    cs.CL 2025-06 conditional novelty 6.0 of 10

    The authors release a new English and Italian CRF slot-filling dataset derived from E3C and show GPT-4o achieves 59.7% and 67.3% micro F1 respectively, with open-weight models behind.

  8. SHIFT: Self-reconstruction Harnesses Implicit Fine-grained Thinking for Retrieval

    cs.IR 2026-07 conditional novelty 5.0 of 10

    SHIFT trains LLM retrievers to reason through latent continuous tokens and reconstruct explicit reasoning traces, improving reasoning-intensive retrieval.

  9. Exploratory and Assimilating Reflection: Reflective Recall Cycle for Long-term Memory

    cs.AI 2026-07 conditional novelty 5.0 of 10

    EAR combines bandit-style memory exploration with experience replay to adapt an LLM agent's reranker, improving retrieval by up to 17.9% over a static retriever on LongMemEval and LoCoMo.

  10. Research Challenges in Relational Database Management Systems for LLM Queries

    cs.DB 2025-08 conditional novelty 5.0 of 10

    Open-source SQL-plus-LLM database integrations fail on most representative LLM queries and run orders of magnitude slower than enterprise APIs, due to missing structured outputs, poor batching, and plan-unaware LLM calls.

  11. Specializing General-purpose LLM Embeddings for Implicit Hate Speech Detection across Datasets

    cs.CL 2025-08 conditional novelty 5.0 of 10

    Fine-tuning large general-purpose text embeddings with a simple instruction yields state-of-the-art implicit hate speech detection, with up to 20.35 point cross-dataset F1 gains.

  12. Do Students Rely on AI? Analysis of Student-ChatGPT Conversations from a Field Study

    cs.AI 2025-08 conditional novelty 5.0 of 10

    In 315 real quiz conversations, college students showed moderate, often ineffective reliance on ChatGPT, and simple behaviors, such as how closely a prompt matched the quiz text and how long the interaction lasted, pr...

  13. T-GRAG: A Dynamic GraphRAG Framework for Resolving Temporal Conflicts and Redundancy in Knowledge Retrieval

    cs.AI 2025-08 conditional novelty 5.0 of 10

    A temporal GraphRAG framework that partitions knowledge graphs by timestamp and retrieves at subgraph, node, and knowledge levels outperforms RAG baselines on a new Audi annual-report QA benchmark.

  14. MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphs

    cs.AI 2025-07 conditional novelty 5.0 of 10

    MMGraphRAG links scene-graph entities from images to text knowledge graph entities via SpecLink, and reports accuracy gains over naive RAG and GraphRAG on multimodal document QA.

  15. SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compression

    cs.CL 2025-07 conditional novelty 5.0 of 10

    SARA combines short natural-language snippets with vector-compressed summaries of the remaining retrieved documents, improving RAG answer quality under 512/1024-token context budgets.

  16. QZhou-Embedding Technical Report

    cs.CL 2025-08 conditional novelty 4.0 of 10

    QZhou-Embedding reports state-of-the-art average scores on MTEB and CMTEB as of August 27, 2025, using a two-stage multi-task pipeline with LLM-based data synthesis.

  17. DS@GT at Touch\'e: Large Language Models for Retrieval-Augmented Debate

    cs.IR 2025-07 conditional novelty 3.0 of 10

    In the Touché 2025 retrieval-augmented debate task, LLM debaters generated verbose but relevant responses, and LLM evaluators were strict and only moderately consistent.

Pith tools