Pith. sign in

REVIEW 17 cited by

Generative Representational Instruction Tuning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.09906 v3 pith:PEHKX6B3 submitted 2024-02-15 cs.CL cs.AIcs.LG

Niklas Muennighoff , Hongjin Su , Liang Wang , Nan Yang , Furu Wei , Tao Yu , Amanpreet Singh , Douwe Kiela This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords modelsgenerativeembeddinggenerationgritgritlmlanguageother
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

All text-based language problems can be reduced to either generation or embedding. Current models only perform well at one or the other. We introduce generative representational instruction tuning (GRIT) whereby a large language model is trained to handle both generative and embedding tasks by distinguishing between them through instructions. Compared to other open models, our resulting GritLM 7B sets a new state of the art on the Massive Text Embedding Benchmark (MTEB) and outperforms all models up to its size on a range of generative tasks. By scaling up further, GritLM 8x7B outperforms all open generative language models that we tried while still being among the best embedding models. Notably, we find that GRIT matches training on only generative or embedding data, thus we can unify both at no performance loss. Among other benefits, the unification via GRIT speeds up Retrieval-Augmented Generation (RAG) by > 60% for long documents, by no longer requiring separate retrieval and generation models. Models, code, etc. are freely available at https://github.com/ContextualAI/gritlm.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. UEmbed: Unified Sparse and Dense Multimodal Embeddings

    cs.CV 2026-08 conditional novelty 6.0 of 10

    UEmbed uses 16 special tokens over a partitioned vocabulary to make a decoder-only multimodal model emit dense and sparse embeddings in one forward pass; the 9B model scores 71.8 dense / 71.0 sparse on MMEB-v2.

  2. FiRE: Enhancing MLLMs with Fine-Grained Context Learning for Complex Image Retrieval

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Disentangled fine-grained context then retrieval fine-tuning on an 87K auto-generated quintuple CIR dataset lifts a 4B MLLM past larger universal retrievers on complex zero-shot image search.

  3. OPERA: Online Data Pruning for Efficient Retrieval Model Adaptation

    cs.IR 2026-03 conditional novelty 6.0 of 10

    Dynamic hierarchical data pruning improves NDCG@10 and Recall@20 for dense retrievers while reaching full performance in half the iterations.

  4. No Thoughts Just AI: Biased LLM Hiring Recommendations Alter Human Decision Making and Limit Human Autonomy

    cs.CY 2025-09 conditional novelty 6.0 of 10

    People largely adopt racially biased AI hiring recommendations, selecting the AI-favored group up to 90% of the time; prior IAT exposure may reduce stereotype-congruent choices by about 13%.

  5. THEME: Enhancing Thematic Investing with Semantic Stock Representations and Temporal Dynamics

    q-fin.PM 2025-08 conditional novelty 6.0 of 10

    A hierarchical contrastive learning framework that aligns stocks with theme descriptions and refines embeddings with short-term return signals improves thematic retrieval and backtested portfolio metrics.

  6. BiXSE: Improving Dense Retrieval via Probabilistic Graded Relevance Distillation

    cs.IR 2025-08 conditional novelty 6.0 of 10

    BiXSE trains dense retrievers with binary cross-entropy on LLM-generated graded relevance probabilities, consistently beating InfoNCE and roughly matching pairwise ranking baselines.

  7. Maximally-Informative Retrieval for State Space Model Generation

    cs.CL 2025-06 conditional novelty 6.0 of 10

    RICO ranks documents by how much they reduce an SSM's question perplexity, using gradient-document inner products, and matches BM25 while often beating E5 on answer quality without finetuning.

  8. GEM: Empowering LLM for both Embedding Generation and Language Understanding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    GEM fine-tunes decoder-only LLMs with bottleneck special tokens and a mixed next-token prediction and contrastive objective, giving them text-embedding ability with only a modest MMLU drop.

  9. Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddings

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Prompt-based text embeddings can be truncated to a small fraction of their dimensions with little performance loss on classification and clustering, but retrieval and STS degrade faster; the difference tracks lower in...

  10. RaDeR: Reasoning-aware Dense Retrieval Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A math-trained dense retriever and reranker, built from MCTS reasoning trajectories and self-reflection, outperforms strong baselines on reasoning-intensive retrieval benchmarks and beats BM25 on chain-of-thought queries.

  11. Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Diffusion-language-model embeddings beat LLM embeddings on long-document and reasoning retrieval benchmarks, but the paper overstates some gains and releases no code or data.

  12. Training Sparse Mixture Of Experts Text Embedding Models

    cs.CL 2025-02 reject novelty 6.0 of 10

    Nomic Embed v2 applies sparse mixture-of-experts upcycling to a multilingual biencoder, reporting competitive BEIR and MIRACL scores with fewer active parameters than dense models of similar size.

  13. SHIFT: Self-reconstruction Harnesses Implicit Fine-grained Thinking for Retrieval

    cs.IR 2026-07 conditional novelty 5.0 of 10

    SHIFT trains LLM retrievers to reason through latent continuous tokens and reconstruct explicit reasoning traces, improving reasoning-intensive retrieval.

  14. LLMs for LLMs: A Structured Prompting Methodology for Long Legal Documents

    cs.AI 2025-09 reject novelty 5.0 of 10

    On CUAD legal contracts, a prompt-engineered QWEN-2 pipeline with chunking and two answer-selection heuristics reportedly outperforms the fine-tuned DeBERTa-large baseline by about 9%, reaching claimed state-of-the-ar...

  15. DeRAG: Black-box Adversarial Attacks on Multiple Retrieval-Augmented Generation Applications via Prompt Injection

    cs.AI 2025-07 conditional novelty 5.0 of 10

    DeRAG shows that five or fewer tokens found by differential evolution can make black-box RAG retrievers rank a chosen wrong document near the top on small BEIR subsets.

  16. UniConv: Unifying Retrieval and Response Generation for Large Language Models in Conversations

    cs.CL 2025-07 reject novelty 5.0 of 10

    A single LLM jointly fine-tuned for conversational dense retrieval and retrieval-augmented generation beats separate retriever-plus-generator pipelines on most test collections, though its headline benchmark was conta...

  17. Exploring Reasoning-Infused Text Embedding with Large Language Models for Zero-Shot Dense Retrieval

    cs.CL 2025-08 conditional novelty 4.0 of 10

    Reasoning-infused text embedding, which prepends LLM-generated reasoning to queries before embedding, improves zero-shot dense retrieval on BRIGHT.

Pith tools