REVIEW 19 cited by
SGPT: GPT Sentence Embeddings for Semantic Search
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Decoder transformers have continued increasing in scale reaching hundreds of billions of parameters. Due to their scale the same decoder sets state-of-the-art results on various language tasks via prompting or fine-tuning. Yet, these large foundation models remain unusable for the related fields of semantic search and sentence embeddings. This prevents possibly new state-of-the-art results and forces organizations to train and maintain separate models. To this end, we propose SGPT to use decoders for sentence embeddings and semantic search via prompting or fine-tuning. At 5.8 billion parameters SGPT improves on the previously best sentence embeddings by a margin of 7% and outperforms a concurrent method with 175 billion parameters as measured on the BEIR search benchmark. Code, models and result files are freely available at https://github.com/Muennighoff/sgpt.
Forward citations
Cited by 19 Pith papers
-
Predicting Multilingual Classification and Translation Performance of LLMs with Cross-Lingual Alignment $\unicode{x2013}$ Is English Enough?
English-based cross-lingual alignment predicts LLM translation quality as well as or better than direct source-target alignment, supporting the English-pivot hypothesis.
-
IRIS: Reusable Identity Representations from Frozen LLMs for Entity Alignment
IRIS extracts identity embeddings from frozen LLMs so each entity is encoded once from its own knowledge graph and matched to other graphs by cosine similarity, hitting 97.99-100.00 Hits@1 on four benchmarks.
-
BitNet Text Embeddings
BITEMBED trains 1.58-bit ternary-weight LLM embedders with contrastive pre-training, supervised distillation, and multi-precision output training, matching FP16 teachers within ~0.6 MMTEB points at ~2x CPU speed.
-
BioHiCL: Hierarchical Multi-Label Contrastive Learning for Biomedical Retrieval with MeSH Labels
BioHiCL applies hierarchical multi-label contrastive learning with MeSH annotations to improve biomedical retrieval, sentence similarity, and question answering using small efficient models.
-
A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)
Only gradient-based (LESS) representations make subset-to-query distance a reliable predictor of instruction-tuning performance; greedy round-robin helps most at small budgets, and random selection is surprisingly com...
-
LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden States
Pooling attention value vectors gives better training-free LLM sentence embeddings than pooling hidden states.
-
LAMDAS: LLM as an Implicit Classifier for Domain-specific Data Selection
LAMDAS selects domain-relevant training data via an LLM likelihood ratio with a learned domain prefix, beating full-data training and nine baselines on code and math.
-
Negative Matters: Multi-Granularity Hard-Negative Synthesis and Anchor-Token-Aware Pooling for Enhanced Text Embeddings
A new MTEB state-of-the-art for text embeddings is reported by combining multi-granularity LLM-generated hard negatives with curriculum training and an anchor-token-aware pooling method.
-
Maximally-Informative Retrieval for State Space Model Generation
RICO ranks documents by how much they reduce an SSM's question perplexity, using gradient-document inner products, and matches BM25 while often beating E5 on answer quality without finetuning.
-
Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddings
Prompt-based text embeddings can be truncated to a small fraction of their dimensions with little performance loss on classification and clustering, but retrieval and STS degrade faster; the difference tracks lower in...
-
How Programming Concepts and Neurons Are Shared in Code Language Models
In Llama-based code models, programming languages are represented through an English-like intermediate token space, with language-specific neurons concentrated in bottom layers and exclusive PL neurons in top layers; ...
-
DeepRTL2: A Versatile Model for RTL-Related Tasks
DeepRTL2 claims state-of-the-art results across RTL generation, understanding, code search, equivalence checking, and performance prediction, but the evidence is weakened by benchmark construction issues and a contrad...
-
Foundation Models for Geospatial Reasoning: Assessing Capabilities of Large Language Models in Understanding Geometries and Topological Spatial Relations
Large language models, especially GPT-4 with few-shot prompts, can classify topological spatial relations between WKT-encoded geometries with roughly 0.6 to 0.66 accuracy, though errors cluster near conceptually simil...
-
Revealing the Numeracy Gap: An Empirical Investigation of Text Embedding Models
Across 13 embedding models and 18 numeric formats, retrieval accuracy on the new EmbedNum-1K benchmark averages 54%, just above chance, showing that embedding models largely fail to encode numeric detail.
-
Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaning
State-of-the-art text embeddings lag far behind on tasks requiring pragmatic inference, stance detection, and social meaning, relative to their strong performance on surface semantic benchmarks.
-
Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modes
Statistical classifiers built on LLM activation norms and coordinates match or beat trained MLP heads on coarse intent routing and resist camouflage better, while MLPs win on fine-grained subfield distinctions.
-
A Multi-Task Evaluation of LLMs' Processing of Academic Text Input
The abstract reports Gemini underperforms on four academic text tasks, but the attached full text is an unrelated biomedical retrieval paper, leaving the claims unverifiable.
-
From Neurons to Semantics: Evaluating Cross-Linguistic Alignment Capabilities of Large Language Models via Neurons Alignment
A neuron-activation-based alignment score for LLMs correlates highly with downstream multilingual performance and transferability across nine open models.
-
Choosing a Text Embedding Model: A Practical Benchmarking and Decision Framework
On four BEIR subsets T3EM leads nDCG@10 (0.638) but mE5-L is the recommended open default; training objective and chunk size dominate size, and no model wins every MTEB task.
Discussion (0). Continue with ORCID to comment.