REVIEW 4 cited by
Understanding LLM Embeddings for Regression
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
With the rise of large language models (LLMs) for flexibly processing information as strings, a natural application is regression, specifically by preprocessing string representations into LLM embeddings as downstream features for metric prediction. In this paper, we provide one of the first comprehensive investigations into embedding-based regression and demonstrate that LLM embeddings as features can be better for high-dimensional regression tasks than using traditional feature engineering. This regression performance can be explained in part due to LLM embeddings over numeric data inherently preserving Lipschitz continuity over the feature space. Furthermore, we quantify the contribution of different model effects, most notably model size and language understanding, which we find surprisingly do not always improve regression performance.
Forward citations
Cited by 4 Pith papers
-
Quantile Regression with Large Language Models for Price Prediction
Fine-tuning Mistral-7B with a multi-quantile head produces calibrated predictive price distributions and better median price estimates than pointwise, embedding-based, and few-shot LLM baselines on three datasets.
-
Emergent Response Planning in LLMs
Hidden representations of LLM prompts encode global attributes of the upcoming response, and simple probes can predict length, content choices, and answer confidence before generation begins.
-
When Dimensionality Hurts: The Role of LLM Embedding Compression for Noisy Regression Tasks
Compressing LLM text embeddings with an autoencoder to about 8 dimensions improves stock return prediction, but this benefit disappears on high-signal tasks, and sentiment features seem to work mainly because of compression.
-
Decoding-based Regression
Autoregressive decoder heads trained with cross-entropy match pointwise heads on tabular regression and can estimate smooth conditional densities.
Discussion (0). Continue with ORCID to comment.