REVIEW 19 cited by
Is ChatGPT a Good Recommender? A Preliminary Study
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recommendation systems have witnessed significant advancements and have been widely used over the past decades. However, most traditional recommendation methods are task-specific and therefore lack efficient generalization ability. Recently, the emergence of ChatGPT has significantly advanced NLP tasks by enhancing the capabilities of conversational models. Nonetheless, the application of ChatGPT in the recommendation domain has not been thoroughly investigated. In this paper, we employ ChatGPT as a general-purpose recommendation model to explore its potential for transferring extensive linguistic and world knowledge acquired from large-scale corpora to recommendation scenarios. Specifically, we design a set of prompts and evaluate ChatGPT's performance on five recommendation scenarios. Unlike traditional recommendation methods, we do not fine-tune ChatGPT during the entire evaluation process, relying only on the prompts themselves to convert recommendation tasks into natural language tasks. Further, we explore the use of few-shot prompting to inject interaction information that contains user potential interest to help ChatGPT better understand user needs and interests. Comprehensive experimental results on Amazon Beauty dataset show that ChatGPT has achieved promising results in certain tasks and is capable of reaching the baseline level in others. We conduct human evaluations on two explainability-oriented tasks to more accurately evaluate the quality of contents generated by different models. And the human evaluations show ChatGPT can truly understand the provided information and generate clearer and more reasonable results. We hope that our study can inspire researchers to further explore the potential of language models like ChatGPT to improve recommendation performance and contribute to the advancement of the recommendation systems field.
Forward citations
Cited by 19 Pith papers
-
From Raw IDs to Semantic Planning: How Recommender Systems Utilize Information at Scale
Recommender systems are moving from raw IDs to semantic IDs, and the next step should be semantic planning that first predicts an exposure's purpose before choosing or generating content.
-
BEAR: Towards Beam-Search-Aware Optimization for Recommendation with Large Language Models
BEAR is a cheap token-level top-B regularizer for LLM-based recommendation, but its central claim that this condition is necessary for beam-search survival is incorrect.
-
GARDRec: Decision-Level Graph Grounding for Large Language Model Recommendation
GARDRec improves LLM-based next-item ranking by grounding decisions in knowledge-graph embeddings, personalized graph contexts, and late-stage scoring rather than prompt text.
-
GrocLM: Grocery Category Recommendation in E-Commerce with Large Language Models
A LoRA-tuned LLM with trie-constrained decoding improves grocery category recommendation and yields a 7.5% cart-add lift in production.
-
RecBase: Generative Foundation Model Pretraining for Zero-Shot Recommendation
A from-scratch model that tokenizes items into hierarchical codes and predicts next-item codes reaches higher average zero-shot AUC on 8 datasets than LLM recommenders up to 7B parameters.
-
Revisiting Prompt Engineering: A Comprehensive Evaluation for LLM-based Personalized Recommendation
For cost-efficient LLMs, rephrasing, step-back, and structured reasoning prompts raise ranking accuracy; for high-performance LLMs, a simple baseline prompt matches complex prompts at a fraction of the cost.
-
GORACS: Group-level Optimal Transport-guided Coreset Selection for LLM-based Recommender Systems
GORACS selects small groups of fine-tuning examples via an optimal-transport and gradient-norm proxy objective, outperforming prior coreset methods for LLM-based recommendation.
-
LLMs as Better Recommenders with Natural Language Collaborative Signals: A Self-Assessing Retrieval Approach
SCORE retrieves similar users' behaviors, uses an LLM self-assessment to rerank them, and injects the top ones in natural language into an LLM prompt, improving recommendation accuracy on two datasets.
-
RouteRec: Strict Evaluation of Recommender-Agent Selection and Aggregation
On MovieLens-1M, request-level hard selection of recommender agents underperforms BM25, while item-level learned shortlist aggregation matches or beats it and improves further with gated LLM use.
-
Why Thinking Hurts: Diagnosing and Rectifying Linguistic Inertia in Large Language Models for Recommendation
Chain-of-thought reasoning degrades semantic-ID recommendation accuracy through 'linguistic inertia,' and a training-free compression-plus-contrastive decoding fix restores and often improves accuracy.
-
Learning to Shop Like Humans: A Review-driven Retrieval-Augmented Recommendation Framework with LLMs
RevBrowse retrieves preference-relevant pros and cons from reviews via a contrastively trained module, then uses an LLM to rerank candidates; experiments on four Amazon datasets show consistent improvements over baselines.
-
Membership Inference Attacks on In-Context Examples in LLM-based Recommender Systems
Simply asking a large language model 'have you seen this user?' or comparing its recommendations after prompt poisoning can reveal whether a user's interactions are in the hidden prompt of an ICL-based recommender.
-
Personas within Parameters: Fine-Tuning Small Language Models with Low-Rank Adapters to Mimic User Behaviors
Persona-level LoRA fine-tuning lets a 3.8B small language model simulate MovieLens users about as accurately as a much larger frozen LLM, at lower cost.
-
R4ec: A Reasoning, Reflection, and Refinement Framework for Recommendation Systems
R4ec trains a small reflection model to critique and refine LLM-generated user and item knowledge, which then improves downstream recommendation accuracy.
-
Rethinking Group Recommender Systems in the Era of Generative AI: From One-Shot Recommendations to Agentic Group Decision Support
The paper proposes reorienting group recommender systems from one-shot preference aggregation to chat-based, agentic decision support powered by large language models.
-
User Behavior Prediction as a Generic, Robust, Scalable, and Low-Cost Evaluation Strategy for Estimating Generalization in LLMs
The authors introduce an entropy-based framework that uses user behavior prediction as a measure of LLM generalization, and find GPT-4o outperforms GPT-4o-mini and Llama-3.1 on movie and music recommendation tasks.
-
TRWH: A Text-Driven Random Walk Heterogeneous GNN for Semantic-Aware Sparse Recommendation
TRWH combines LLM profiles, a heterogeneous GNN, and one-hop random walk edges, but its headline performance claims are contradicted by its own results table.
-
T-POP: Test-Time Personalization with Online Preference Feedback
T-POP uses dueling-bandit token selection to learn a reward function online from pairwise user feedback, enabling test-time personalization of a frozen LLM without fine-tuning.
-
A Language-Driven Framework for Improving Personalized Recommendations: Merging LLMs with Traditional Algorithms
An LLM-based re-ranking layer over SVD/SVD++ improves offline rating and ranking metrics on MovieLens-Latest-Small, but evaluation and baseline gaps limit the strength of the claim.
Discussion (0). Sign in to comment.