Pith. sign in

REVIEW 43 cited by

LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.05961 v2 pith:NZXYC7PV submitted 2024-04-09 cs.CL cs.AI

Parishad BehnamGhader , Vaibhav Adlakha , Marius Mosbach , Dzmitry Bahdanau , Nicolas Chapados , Siva Reddy This is my paper · ORCID
classification cs.CLcs.AI
keywords modelsllm2vectexttaskslargellmsstate-of-the-artunsupervised
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large decoder-only language models (LLMs) are the state-of-the-art models on most of today's NLP tasks and benchmarks. Yet, the community is only slowly adopting these models for text embedding tasks, which require rich contextualized representations. In this work, we introduce LLM2Vec, a simple unsupervised approach that can transform any decoder-only LLM into a strong text encoder. LLM2Vec consists of three simple steps: 1) enabling bidirectional attention, 2) masked next token prediction, and 3) unsupervised contrastive learning. We demonstrate the effectiveness of LLM2Vec by applying it to 4 popular LLMs ranging from 1.3B to 8B parameters and evaluate the transformed models on English word- and sequence-level tasks. We outperform encoder-only models by a large margin on word-level tasks and reach a new unsupervised state-of-the-art performance on the Massive Text Embeddings Benchmark (MTEB). Moreover, when combining LLM2Vec with supervised contrastive learning, we achieve state-of-the-art performance on MTEB among models that train only on publicly available data (as of May 24, 2024). Our strong empirical results and extensive analysis demonstrate that LLMs can be effectively transformed into universal text encoders in a parameter-efficient manner without the need for expensive adaptation or synthetic GPT-4 generated data.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 43 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Fisher Random Walk: Automatic Debiasing Contextual Preference Inference for Large Language Model Evaluation

    stat.ML 2025-09 conditional novelty 7.0 of 10

    A Fisher random walk weighted residual estimator achieves semiparametric efficient confidence intervals for contextual Bradley-Terry-Luce preference comparisons with flexible score estimators.

  2. A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation

    cs.CV 2025-06 conditional novelty 7.0 of 10

    Layer-normalized averaging of all decoder-only LLM hidden states, rather than last-layer embeddings, improves text-to-image compositional alignment and beats T5 on GenAI-Bench.

  3. UEmbed: Unified Sparse and Dense Multimodal Embeddings

    cs.CV 2026-08 conditional novelty 6.0 of 10

    UEmbed uses 16 special tokens over a partitioned vocabulary to make a decoder-only multimodal model emit dense and sparse embeddings in one forward pass; the 9B model scores 71.8 dense / 71.0 sparse on MMEB-v2.

  4. Illuminating Visual Identity in Universal Multimodal Embeddings

    cs.CV 2026-08 conditional novelty 6.0 of 10

    By adding identity-aware sampling and a contrastive loss on a new 28-dataset benchmark, the authors build multimodal embeddings that are far better at visual identity matching without losing general retrieval accuracy.

  5. IRIS: Reusable Identity Representations from Frozen LLMs for Entity Alignment

    cs.CL 2026-07 conditional novelty 6.0 of 10

    IRIS extracts identity embeddings from frozen LLMs so each entity is encoded once from its own knowledge graph and matched to other graphs by cosine similarity, hitting 97.99-100.00 Hits@1 on four benchmarks.

  6. SkipOPU: An FPGA-based Overlay Processor for Large Language Models with Dynamically Allocated Computation

    cs.AR 2026-03 conditional novelty 6.0 of 10

    An FPGA accelerator for SkipGPT-style pruned LLMs that fuses router/normalization/attention computation to hide nonlinear latency, packs two FP16 products in each DSP, and serves reused KV entries from on-chip URAM to...

  7. Enhancing next token prediction based pre-training for jet foundation models

    hep-ph 2025-12 conditional novelty 6.0 of 10

    Using continuous particle features as input and combining next-token with masked-token pre-training markedly improves classification accuracy of the OmniJet jet foundation model without visibly hurting its generative quality.

  8. Unpacking Hateful Memes: Presupposed Context and False Claims

    cs.CL 2025-10 conditional novelty 6.0 of 10

    A hateful-meme detector that combines presupposed-context fusion, LLM-based social perception, and cross-modal reference graphs outperforms prior models on three benchmarks and transfers to fake news.

  9. Exploring the Capabilities of Large Language Model Encoders for Image-Text Retrieval in Chest X-rays

    cs.CV 2025-09 conditional novelty 6.0 of 10

    Domain-adapted LLM encoders trained with masked token prediction and supervised contrastive learning improve chest X-ray image-text retrieval and external generalization, reaching GREEN scores of 0.308 on MIMIC-CXR an...

  10. Topic-Guided Reinforcement Learning with LLMs for Enhancing Multi-Document Summarization

    cs.CL 2025-09 conditional novelty 6.0 of 10

    A topic-F1 reward measuring alignment between summary and source-document topics, combined with GRPO training, improves multi-document summarization over several baselines.

  11. Negative Matters: Multi-Granularity Hard-Negative Synthesis and Anchor-Token-Aware Pooling for Enhanced Text Embeddings

    cs.CL 2025-08 conditional novelty 6.0 of 10

    A new MTEB state-of-the-art for text embeddings is reported by combining multi-granularity LLM-generated hard negatives with curriculum training and an anchor-token-aware pooling method.

  12. BiXSE: Improving Dense Retrieval via Probabilistic Graded Relevance Distillation

    cs.IR 2025-08 conditional novelty 6.0 of 10

    BiXSE trains dense retrievers with binary cross-entropy on LLM-generated graded relevance probabilities, consistently beating InfoNCE and roughly matching pairwise ranking baselines.

  13. HT-Transformer: Event Sequences Classification by Accumulating Prefix Information with History Tokens

    cs.LG 2025-08 conditional novelty 6.0 of 10

    Introducing history tokens with sparse attention masks during next-token pretraining improves transformer event-sequence classification, but the effect is confounded by the use of an appended token at inference.

  14. Learning Robust Negation Text Representations

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Finetuning text encoders on taxonomy-guided, LLM-generated negation and hedging triples substantially improves negation benchmark performance while keeping general embedding quality roughly intact.

  15. Text-ADBench: Text Anomaly Detection Benchmark based on LLMs Embedding

    cs.CL 2025-07 conditional novelty 6.0 of 10

    LLM embeddings improve text anomaly detection, shallow detectors match deep ones only under oracle embedding selection, and AUROC matrices are low-rank enough to support fast model evaluation.

  16. MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A two-stage training recipe that converts causal VLMs into bidirectional multimodal embedding models, achieving SOTA on MMEB.

  17. AI-Generated Song Detection via Lyrics Transcripts

    cs.SD 2025-06 conditional novelty 6.0 of 10

    Transcribing audio with Whisper and classifying the transcript with LLM2Vec detects AI-generated songs from audio alone, nearly matching clean-lyrics accuracy and beating audio-based detectors under perturbations and ...

  18. Maximally-Informative Retrieval for State Space Model Generation

    cs.CL 2025-06 conditional novelty 6.0 of 10

    RICO ranks documents by how much they reduce an SSM's question perplexity, using gradient-document inner products, and matches BM25 while often beating E5 on answer quality without finetuning.

  19. Build the web for agents, not agents for the web

    cs.LG 2025-06 conditional novelty 6.0 of 10

    The paper proposes a paradigm shift: design a standardized Agentic Web Interface for AI agents, rather than adapting agents to human-facing websites.

  20. Just a Scratch: Enhancing LLM Capabilities for Self-harm Detection through Intent Differentiation and Emoji Interpretation

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Adding an emoji-interpretation matrix and casual/serious intent span labels to multitask LLM fine-tuning lifts self-harm detection F1 to 0.88 on a newly annotated Reddit dataset.

  21. GEM: Empowering LLM for both Embedding Generation and Language Understanding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    GEM fine-tunes decoder-only LLMs with bottleneck special tokens and a mixed next-token prediction and contrastive objective, giving them text-embedding ability with only a modest MMLU drop.

  22. DeepRTL2: A Versatile Model for RTL-Related Tasks

    cs.AR 2025-05 reject novelty 6.0 of 10

    DeepRTL2 claims state-of-the-art results across RTL generation, understanding, code search, equivalence checking, and performance prediction, but the evidence is weakened by benchmark construction issues and a contrad...

  23. Bringing CLIP to the Clinic: Dynamic Soft Labels and Negation-Aware Learning for Medical Analysis

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Medical CLIP training with text, clinical, and graph soft labels plus negation hard negatives improves chest X-ray zero-shot and fine-tuned performance.

  24. Optimizing fMRI Data Acquisition for Decoding Natural Speech with Limited Participants

    q-bio.NC 2025-05 conditional novelty 6.0 of 10

    In a small cohort, fMRI decoders improve with more data per participant, and multi-subject training or shared stimuli add no benefit, so deep phenotyping is the recommended acquisition strategy.

  25. Aligning Web Query Generation with Ranking Objectives via Direct Preference Optimization

    cs.IR 2025-05 conditional novelty 6.0 of 10

    Using direct preference optimization with reranker or GPT-3.5 preferences to align synthetic query generation improves downstream dense retrieval effectiveness on MS MARCO and TREC-DL.

  26. F$^2$Agent: Financial Fusion of Agentic Intelligence for Multimodal Trading

    cs.MA 2026-08 reject novelty 5.0 of 10

    F2Agent, a hierarchy of specialized LLM and Transformer agents with adaptive cross-modal attention and consistency regularization, is reported to beat 16 trading baselines on six assets, though appendix results from a...

  27. Probe, Don't Prompt: A Hidden-State Probe for Metadata Filtering in Multi-Meta-RAG

    cs.CL 2026-07 conditional novelty 5.0 of 10

    A shallow mean-pooled multi-label probe on a 135M–1.5B open model replaces GPT-3.5 source extraction in Multi-Meta-RAG at 90.9% set-exact accuracy, with the margin coming from null-query abstention.

  28. Reconstructing Content with Collaborative Attention for Universal Multimodal Representation Learning

    cs.IR 2026-03 conditional novelty 5.0 of 10

    CoCoA forces an MLLM to reconstruct masked text through a single EOS token, improving multimodal embedding quality on MMEB-V1 and matching MoCa at 3B with far less pretraining data.

  29. ExploreGS: Explorable 3D Scene Reconstruction with Virtual Camera Samplings and Diffusion Priors

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    Adding information-gain-selected virtual views refined by video diffusion priors to 3D Gaussian Splatting improves arbitrary-view rendering quality.

  30. A Comparative Study of Specialized LLMs as Dense Retrievers

    cs.IR 2025-07 conditional novelty 5.0 of 10

    Specialized Qwen2.5 7B models differ in dense retrieval quality: math and long-reasoning variants degrade performance, while coder and vision-language variants improve zero-shot text and code retrieval.

  31. Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaning

    cs.CL 2025-06 conditional novelty 5.0 of 10

    State-of-the-art text embeddings lag far behind on tasks requiring pragmatic inference, stance detection, and social meaning, relative to their strong performance on surface semantic benchmarks.

  32. VeriLoC: Line-of-Code Level Prediction of Hardware Design Quality from Verilog Code

    cs.AR 2025-06 conditional novelty 5.0 of 10

    A method that predicts line-level timing and congestion issues directly from Verilog code using CL-Verilog embeddings and gradient-boosted classifiers.

  33. Optimizing Recall or Relevance? A Multi-Task Multi-Head Approach for Item-to-Item Retrieval in Recommendation

    cs.IR 2025-06 conditional novelty 5.0 of 10

    A multi-task, multi-head item-to-item retrieval system that merges co-engagement candidates with semantically relevant candidates achieves both higher recall and higher semantic relevance than prior models.

  34. REARANK: Reasoning Re-ranking Agent via Reinforcement Learning

    cs.IR 2025-05 conditional novelty 5.0 of 10

    Training a listwise reranker with reinforcement learning and explicit reasoning on only 179 annotated queries yields reranking quality comparable to GPT-4.

  35. RetrieveAll: A Multilingual Named Entity Recognition Framework with Large Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    RetrieveAll combines per-language LoRA adapters with retrieval of entity and context examples to improve multilingual NER, claiming an average 12.1% F1 gain on PAN-X.

  36. $I^2G$: Generating Instructional Illustrations via Text-Conditioned Diffusion

    cs.CL 2025-05 reject novelty 5.0 of 10

    A pairwise-conditioned diffusion model generates instructional illustrations from procedural text and is finetuned with a text-image alignment reward.

  37. Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modes

    cs.CL 2026-08 conditional novelty 4.0 of 10

    Statistical classifiers built on LLM activation norms and coordinates match or beat trained MLP heads on coarse intent routing and resist camouflage better, while MLPs win on fine-grained subfield distinctions.

  38. HyFedRAG: A Federated Retrieval-Augmented Generation Framework for Heterogeneous and Privacy-Sensitive Data

    cs.AI 2025-09 reject novelty 4.0 of 10

    HyFedRAG is a federated RAG framework over heterogeneous data with local anonymization and three-tier caching, but the experiments do not support its headline performance and privacy claims.

  39. Exploring Reasoning-Infused Text Embedding with Large Language Models for Zero-Shot Dense Retrieval

    cs.CL 2025-08 conditional novelty 4.0 of 10

    Reasoning-infused text embedding, which prepends LLM-generated reasoning to queries before embedding, improves zero-shot dense retrieval on BRIGHT.

  40. TableVault: Managing Dynamic Data Collections for LLM-Augmented Workflows

    cs.DB 2025-06 reject novelty 4.0 of 10

    TableVault describes a system design for managing versioned, reproducible dataframe collections in LLM-augmented workflows, but it ships no implementation or evaluation.

  41. LGAI-EMBEDDING-Preview Technical Report

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A Mistral-7B embedding model trained with in-context instructions, soft labels from an in-house retrieval pipeline, and margin-based hard-negative mining reports top-tier MTEB English v2 scores.

  42. Hidden Persuasion: Detecting Manipulative Narratives on Social Media During the 2022 Russian Invasion of Ukraine

    cs.CL 2025-05 conditional novelty 4.0 of 10

    A shared-task system that fine-tunes Gemma 2 with LoRA and XLM-RoBERTa to classify and locate manipulative narratives in Ukrainian/Russian Telegram posts, placing 2nd and 3rd in the UNLP 2025 competition.

  43. Rethinking the Understanding Ability across LLMs through Mutual Information

    cs.CL 2025-05 conditional novelty 4.0 of 10

    The paper uses token-level recoverability as a computable lower bound on mutual information to compare LLMs and to fine-tune them, finding encoder-only models preserve information better than decoder-only models.

Pith tools