REVIEW 5 cited by
LLM-Select: Feature Selection with Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
In this paper, we demonstrate a surprising capability of large language models (LLMs): given only input feature names and a description of a prediction task, they are capable of selecting the most predictive features, with performance rivaling the standard tools of data science. Remarkably, these models exhibit this capacity across various query mechanisms. For example, we zero-shot prompt an LLM to output a numerical importance score for a feature (e.g., "blood pressure") in predicting an outcome of interest (e.g., "heart failure"), with no additional context. In particular, we find that the latest models, such as GPT-4, can consistently identify the most predictive features regardless of the query mechanism and across various prompting strategies. We illustrate these findings through extensive experiments on real-world data, where we show that LLM-based feature selection consistently achieves strong performance competitive with data-driven methods such as the LASSO, despite never having looked at the downstream training data. Our findings suggest that LLMs may be useful not only for selecting the best features for training but also for deciding which features to collect in the first place. This could benefit practitioners in domains like healthcare and the social sciences, where collecting high-quality data comes at a high cost.
Forward citations
Cited by 5 Pith papers
-
DeepFeature: LLM-Empowered Context-aware Feature Generation for Wearable Biosignals
An LLM-based pipeline that generates context-aware features from raw wearable biosignals and iteratively refines them via performance feedback reports AUROC gains of 4-10% over baselines.
-
FailureSensorIQ: A Multi-Choice QA Dataset for Understanding Sensor Relationships and Failure Modes
A new expert-curated multiple-choice benchmark shows that even top LLMs score near 60%, drop sharply under paraphrasing, and remain below human experts on industrial sensor-failure reasoning.
-
Feature Generation Using LLMs: An Evolutionary Algorithm Approach
A funsearch-style evolutionary loop using LLaMA-3.1 7B-generated Python expressions creates new table features and improves F1 in 13 of 16 evaluated classification settings.
-
LLM-FS: Zero-Shot Feature Selection for Effective and Interpretable Malware Detection
Zero-shot LLM-based feature selection matches traditional statistical feature selection in accuracy on the EMBOD malware dataset while offering interpretability and stability advantages.
-
LLM-ML Teaming: Integrated Symbolic Decoding and Gradient Search for Valid and Stable Generative Feature Transformation
A product-of-experts decoder that blends a fine-tuned LLM's token probabilities with a gradient-searched sequence decoder produces more valid and stable feature transformations than either alone.
Discussion (0). Sign in to comment.