REVIEW 3 major objections 5 minor 36 references
Shape Your Feed: An LLM-based Agentic System for Conversational Recommendation
T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Shape Your Feed claims a three-flow LLM agent can turn passive feed ranking into real-time, user-steerable co-curation, backed by offline 98.85% alignment accuracy and online A/B gains.
desk verdict A genuine industrial agentic recommender with real A/B evidence and a human-checked offline pipeline, but the headline offline accuracy is partly circular and the online negative-feedback gains may be channel-substitution artifacts; it deserves serious review. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the Semantic Profile ($S_t$), a persistent, semi-structured, text-verbalized record of interests, disinterests, and intent strengths that is updated by the Perception Flow and consumed by the Serving Flow. Around it sits a list-wise alignment scorer: a single LLM inference scores all candidates $v_i$ against the profile as $\hat{y}_{v_i|S_t}$, enabling hard pruning of negative matches and re-ranking via $s_{\mathrm{final}} = \alpha \hat{y}_{v|S_t} + (1-\alpha)\hat{y}_{u,v}$. The Self-Evolution Flow supplies supervision by combining sparse real user actions with an LLM-as-a-Judge ensemble that labels contexts where explicit feedback is absent, first warm-starting the scorer with SFT and then refining preferences with DPO. This combination is what carries the argument from users being able to say what they want to the feed actually changing.
What would settle it
Take a random sample of profile-candidate pairs from the production serving log that were not used to build the SFT or DPO data, have human raters label match/mismatch against the Semantic Profile, and compare those labels to the ensemble's; if agreement on this out-of-pipeline sample falls well below the reported 96% human-judge agreement, the offline accuracy claim loses its foundation. A second check is to rerun the offline evaluation on the 1,029-sample golden set alone and compare the 98.85% figure to the 95.8% reported there.
Extended reading notes
Core claim
The central claim is that a closed-loop agentic architecture can reconcile expressive user intent with production feed ranking. In SYF, user feedback is resolved into a semi-structured Semantic Profile that persists across turns; candidate items are then scored list-wise by a fine-tuned LLM that outputs one alignment score per candidate, and items falling below a threshold are pruned while the rest are blended with the production engagement score via a fusion weight. The paper reports that this alignment module, after SFT followed by DPO, reaches 98.85% accuracy and 79.51% F1 on its offline benchmark, far above an 83.84%-accuracy few-shot baseline, and that conventional click-based show-more/show-less probabilities barely separate matched from mismatched content (0.289 vs 0.212 at the 90th percentile), which the paper takes as evidence that passive signals cannot capture explicit intent. The online A/B test on production traffic, restricted to the context-aware feedback pills interface, attributes the quality gains to the serving flow: pills alone increase UI clicks but not feed quality, while the full system reduces post dismiss and dislike rates and raises new-interest consumption.
Load-bearing premise
The load-bearing premise is that the LLM-as-a-Judge ensemble labels are a trustworthy stand-in for real human preference judgments; the SFT data, the offline benchmark, and the DPO teacher for sparse contexts all come from the same judge pipeline, so a systematic judge bias would inflate the headline accuracy and distort the learned ranking.
Editorial extensions
If this is right
- Users who actively give feedback should see their explicit preferences reflected in the feed within the serving-path latency budget, because alignment scoring runs in one list-wise inference and adds only +0.043% critical-path latency.
- Negative feedback such as dismiss and dislike should drop whenever the Semantic Profile correctly captures a disinterest, since pruning removes below-threshold candidates before ranking.
- New-interest consumption can increase even for users with weak historical signals, because candidate sourcing retrieves items for freshly extracted positive interests and the fusion weight lets semantic alignment override weak engagement priors.
- The two-stage SFT-then-DPO recipe improves the alignment scorer over SFT alone, with F1 rising from 76.97% to 79.51%, suggesting that preference optimization adds signal beyond supervised imitation.
Reading between the lines
- If the LLM-judge proxy is even mildly biased toward polished or popular content, the 98.85% offline accuracy is an upper bound; a clean test is to re-score a random sample of profile-candidate pairs with human raters on the actual production distribution, not just the 1,029-sample golden set.
- The architecture separates intent capture from rank adjustment, so the same Semantic Profile could plausibly be projected onto other surfaces such as notifications, search, or email digests; this is an extension the paper does not claim.
- Because the online experiment restricted interaction to context-aware pills, the voice and free-text channels remain untested in production; their added value is an open empirical question the paper leaves implicit.
- The reported effect sizes are under three percent on negative-feedback rates, so the long-term practical case rests on whether these gains compound over longer retention windows rather than saturating quickly.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents Shape Your Feed (SYF), an LLM-based agentic framework that lets users steer their recommender feed through text, voice, and UI interactions. A Perception Flow maintains a persistent Semantic Profile; a Serving Flow uses this profile to augment, prune, and re-rank candidates; and a Self-Evolution Flow tunes the alignment scorer via SFT and DPO using LLM-judge labels and behavioral feedback. Offline experiments report 98.85% accuracy for the alignment scorer, and online A/B tests on production traffic report a 1.70% relative reduction in post dismisses, a 2.74% reduction in post dislikes, and a 0.16% increase in interest consumption.
Significance. If the online results are causal, the paper demonstrates a rare industrial deployment of an LLM-based conversational recommender with real-time user steering. The system architecture is thoughtful, and the inclusion of a production A/B test, an ablation study, and an independent human-annotated golden set of 1,029 samples partially de-risks the evaluation. The paper also honestly acknowledges its reliance on explicit user inputs as a limitation. These strengths make the work potentially valuable for the RecSys community.
major comments (3)
- [Section 6.2 (Tables 5 and 6)] The claim that SYF reduces explicit negative feedback is confounded by channel substitution. In the full system, users can express negative preferences via Context-Aware Pills (e.g., 'fewer political posts'), which are not counted as dismisses or dislikes; the control group has no such channel and must use dismiss/dislike. The reported -1.70% dismiss and -2.74% dislike could therefore reflect re-classification of negative feedback into the pill channel rather than an improvement in feed relevance. The sentence in Section 6.2.3 that 'Dismiss and dislike capture all negative feedback including unintentional behavior' is inaccurate when pills are present, and the Pills Only ablation does not resolve the issue because inert pills give users no incentive to substitute. The paper should report a combined negative-feedback rate (dismiss + dislike + negative pill selections + static 'Show Less' equivalents) for both control and treatment groups.
- [Section 5.1.1, Tables 1 and 3] The offline evaluation set is built 'using the same pipeline' that generated the SFT labels, so the headline 98.85% accuracy partly measures agreement with the LLM-judge teacher rather than with human ground truth. The independent human golden set in Section 6.1.4 is a good step, but the paper does not report few-shot or SFT-only accuracy on that set, so the reader cannot assess whether the claimed improvements over the baseline hold for human labels. Please report the golden-set results for all three conditions (few-shot, SFT, SFT+DPO) and the judge-human agreement on that set.
- [Tables 1 and 3] Accuracy is a misleading headline metric for this binary alignment task if the classes are imbalanced. The few-shot baseline already achieves 83.84% accuracy with only 47.06% precision, and the SFT+DPO model reaches 98.85% accuracy with 79.16% precision, suggesting that accuracy is dominated by the majority class. The paper should also report balanced accuracy or PR-AUC for all offline conditions and for the human golden set.
minor comments (5)
- [Section 5, first paragraph] The statement that the LLM judges 'achieved a 96% agreement rate with human judgments on a multi-reviewed benchmark' lacks details; please specify the benchmark, the number of samples, and the annotation protocol.
- [Section 6.2.2] The online results report relative lifts without absolute rates, confidence intervals, or the number of users and traffic volume; please provide these to allow the reader to judge practical significance.
- [Section 5.2.1] The DPO stage also relies on judge-inferred chosen-rejected pairs when explicit feedback is sparse; please quantify the fraction of DPO pairs that come from judge labels versus explicit user actions and discuss the risk of reinforcing judge bias.
- [Section 4.2.2, Eq. (4)] The fusion weight alpha and the pruning threshold tau are free parameters; please report a sensitivity analysis or the criteria used to set them.
- [Section 6.2.3, Table 6] The ablation would be more informative with a condition in which pills are active but the backend re-ranking is disabled, to separate the interface effect from the algorithmic effect.
Circularity Check
Headline offline accuracy (98.85%) is measured on an evaluation set built with the same LLM-judge pipeline that generated the SFT training labels; the independent human golden set and production A/B keep the central claims externally grounded.
-
fitted input called prediction
[Section 5.1.1 (SFT Data Preparation), Section 6.1.2 (Table 1), echoed in the Abstract]
"Because explicit user feedback is limited, labels are generated via the LLM-as-a-Judge ensemble. ... The evaluation set is built using the same pipeline to guarantee a consistent offline benchmark distribution. ... we benchmark against a few-shot Llama3-8B baseline using an evaluation dataset annotated by our LLM judge ensemble."
The SFT stage is trained on labels produced by the LLM-as-a-Judge ensemble, and the offline evaluation set is explicitly constructed with the same pipeline ('to guarantee a consistent offline benchmark distribution'). The reported headline accuracy (98.85% for SFT+DPO) therefore measures agreement with the same teacher that generated the training targets, rather than accuracy against an independent ground truth. The abstract presents this number as 'SYF's alignment scoring module achieves 98.85% accuracy' without this caveat. This is a by-construction non-independence between training signal and evaluation benchmark.
full rationale
The only load-bearing circularity I can exhibit by quotation is the same-pipeline offline evaluation: SFT labels come from the LLM-as-a-Judge ensemble, and the evaluation set is built with the same pipeline, so the 98.85% headline accuracy is a student-teacher agreement metric relative to the very generator that produced the training labels. The paper's own independent human golden set (95.8%, strictly isolated from the judge pipeline) and the online A/B experiment on production traffic (real dismiss/dislike and consumption metrics) provide external grounding for the central claims, so the contribution does not reduce entirely to its inputs. No self-citation chains, imported uniqueness theorems, or ansatz-smuggling citations appear; the references are contextual. The online A/B channel-substitution concern (users may shift negative expressions from dismiss/dislike to Context-Aware Pills) is a measurement-interpretation threat rather than a by-construction circularity, so it is not scored as a circular step. Overall partial circularity of the headline offline metric, with independent support for the core system claims, warrants a score of 4.
Assumptions & free parameters
free parameters (3)
- alpha (fusion weight) =
not reported
- tau (pruning threshold) =
not reported
- K (retrieval top-k per interest) =
not reported
assumptions (5)
- domain assumption LLM-as-a-Judge ensemble annotations are a faithful proxy for human preference judgments.
- domain assumption Explicit user actions such as dismiss and dislike are reliable, high-confidence indicators of negative preference.
- domain assumption The production ranker's engagement probability y_hat_u,v is a valid foundation that blending with semantic scores preserves.
- domain assumption Item features can be faithfully decoded from hashed IDs or dense vectors into natural-language descriptions for the LLM.
- ad hoc to paper Multi-turn state tracking via a persistent Semantic Profile is sufficient for long-horizon preference consistency.
invented entities (4)
-
Semantic Profile (S_t)
-
Intent Trajectory (T_t)
-
Candidate Cache (C_s,t)
-
Context-Aware Feedback Pills
independent evidence
Cite this review
Pith. "Pith review of Shape Your Feed: An LLM-based Agentic System for Conversational Recommendation." pith.science (2026). https://pith.science/paper/V2NQDTGK
@misc{pith2026260806632,
author = {Pith},
title = {Pith review of: Shape Your Feed: An LLM-based Agentic System for Conversational Recommendation},
year = {2026},
howpublished = {\url{https://pith.science/paper/V2NQDTGK}},
note = {Machine review of arXiv:2608.06632}
}
read the original abstract
Industrial recommendation systems predominantly adopt a passive ranking paradigm that infers user preferences from implicit behavioral signals (e.g., clicks, dwell time) rather than explicit, natural language inputs. As a result, users experience a persistent discrepancy between their explicit interests and what passive behavioral algorithms deliver, limiting their ability to express nuanced preferences or steer their feed in real time. To address this growing gap between how recommendations are optimized and how users wish to articulate their interests, we present Shape Your Feed (SYF), an LLM-based agentic recommendation framework that enables real-time, multimodal co-curation of content. SYF employs a three-tier architecture: (i) a Perception Flow that captures fine-grained user intent from text prompts, voice commands, and UI interactions; (ii) a Serving Flow that performs real-time agentic re-ranking and pruning of candidate items, grounded in a persistent Semantic Profile encoding evolving user preferences; and (iii) a Self-Evolution Flow that aligns system behavior with human judgments via Direct Preference Optimization (DPO) and an LLM-as-a-Judge ensemble. Offline evaluations show that SYF's alignment scoring module achieves 98.85% accuracy, substantially improving over strong few-shot baselines. Large-scale online A/B experiments on production traffic further demonstrate that SYF improves feed relevance and user sentiment, indicating a practical and scalable path toward interactive, user-steerable recommendation in industrial settings.
Figures
Reference graph
Works this paper leans on
-
[1]
Li Chen and Pearl Pu. 2012. Critiquing-based Recommenders: Survey and Emerg- ing Trends.User Modeling and User-Adapted Interaction22, 1-2 (2012), 125–150
work page 2012
-
[2]
Qibin Chen, Junyang Lin, Yichang Zhang, Ming Ding, Yukuo Cen, Hongxia Yang, and Jie Tang. 2019. Towards Knowledge-Based Recommender Dialog System. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP). 1803–1813
work page 2019
-
[3]
Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al
-
[4]
Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. InProceedings of the 10th ACM Conference on Recommender Systems (RecSys). 191–198
2016
-
[5]
Nouha Dziri, Andrea Madotto, Osmar Zaïane, and Avishek Joey Bose. 2021. Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). 2197–2214
work page 2021
-
[6]
Wenqi Fan, Zihuai Zhao, Jiatong Li, Yunqing Liu, Xiaowei Mei, Yiqi Wang, Jiliang Tang, and Qing Li. 2023. Recommender Systems in the Era of Large Language Models (LLMs).arXiv preprint arXiv:2307.02046(2023)
arXiv 2023
-
[7]
Yue Feng, Shuchang Liu, Zhenghai Xue, Qingpeng Cai, Lantao Hu, Peng Jiang, Kun Gai, and Fei Sun. 2023. A large language model enhanced conversational recommender system.arXiv preprint arXiv:2308.06212(2023)
arXiv 2023
-
[8]
Luke Friedman, Sameer Ahuja, David Allen, Zhenning Tan, Hakim Sidahmed, Changbo Long, Jun Xie, Gabriel Schubiner, Ajay Patel, Harsh Lara, Brian Chu, Zexi Chen, and Manoj Tiwari. 2023. Leveraging Large Language Models in Conversational Recommender Systems.arXiv preprint arXiv:2305.07961(2023)
arXiv 2023
Show all 36 references
-
[9]
Chongming Gao, Wenqiang Lei, Xiangnan He, Maarten de Rijke, and Tat-Seng Chua. 2021. Advances and Challenges in Conversational Recommender Systems: A Survey.AI Open2 (2021), 100–126
2021
-
[10]
Jaime Govea, Rommel Gutierrez, and William Villegas-Ch. 2024. Transparency and Precision in the Age of AI: Evaluation of Explainability-Enhanced Recom- mendation Systems.Frontiers in Artificial Intelligence7 (2024), 1410790
2024
-
[11]
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural Collaborative Filtering. InProceedings of the 26th International Conference on World Wide Web (WWW). 173–182
2017
-
[12]
Yifan Hu, Yehuda Koren, and Chris Volinsky. 2008. Collaborative Filtering for Implicit Feedback Datasets. InProceedings of the 2008 Eighth IEEE International Conference on Data Mining (ICDM). IEEE, 263–272
2008
-
[13]
Dietmar Jannach, Ahtsham Manzoor, Wanling Cai, and Li Chen. 2021. A Survey on Conversational Recommender Systems.Comput. Surveys54, 5 (2021), 1–36
2021
-
[14]
Wang-Cheng Kang and Julian McAuley. 2018. Self-Attentive Sequential Recom- mendation. InProceedings of the IEEE International Conference on Data Mining (ICDM). 197–206
2018
-
[15]
Sara Kemper, Justin Cui, Kai Dicarlantonio, Kathy Lin, Danjie Tang, Anton Ko- rikov, and Scott Sanner. 2024. Retrieval-augmented conversational recommen- dation with prompt-based semi-structured natural language state tracking. In Proceedings of the 47th International ACM SIGI...
2024
-
[16]
Ivica Kostric, Krisztian Balog, and Filip Radlinski. 2024. Generating Usage-related Questions for Preference Elicitation in Conversational Recommender Systems. ACM Transactions on Recommender Systems2, 2 (2024), 1–24
2024
-
[17]
Raymond Li, Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. 2018. Towards Deep Conversational Recommendations. InAdvances in Neural Information Processing Systems (NeurIPS), Vol. 31
2018
-
[18]
Samuel Louvan and Bernardo Magnini. 2020. Recent Neural Methods on Slot Filling and Intent Classification for Task-Oriented Dialogue Systems: A Survey. InProceedings of the 28th International Conference on Computational Linguistics (COLING). 480–496
2020
-
[19]
Maxim Naumov, Dheevatsa Mudigere, Hao Jiin Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alis- son G Azzolini, et al . 2019. Deep Learning Recommendation Model for Per- sonalization and Recommendation Systems.arXiv preprint a...
2019 arXiv
-
[20]
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback.Advances in neural information processing systems35 (...
2022
-
[21]
Linlu Qiu, Fei Sha, Kelsey Allen, Yoon Kim, Tal Linzen, and Sjoerd van Steenkiste
-
[22]
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model.Advances in neural information processing systems36 (2023), 53728–53741
2023
-
[23]
Xuhui Ren, Tong Chen, Quoc Viet Hung Nguyen, Lizhen Cui, Zi Huang, and Hongzhi Yin. 2024. Explicit knowledge graph reasoning for conversational recommendation.ACM Transactions on Intelligent Systems and Technology15, 4 (2024), 1–21
2024
-
[24]
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme
-
[25]
Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang
-
[26]
S M Towhidul Islam Tonmoy, S M Mehedi Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. 2024. A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models.arXiv preprint arXiv:2401.01313(2024)
2024 arXiv
-
[27]
Wei Wei, Xubin Ren, Jiabin Tang, Qinyong Wang, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, and Chao Huang. 2024. LLMRec: Large Language Models with Graph Augmentation for Recommendation. InProceedings of the 17th ACM International Conference on Web Search and Data Mining (W...
2024
-
[28]
Chi-Man Wong, Fan Feng, Wen Zhang, Chi-Man Vong, Hui Chen, Yichi Zhang, Peng He, Huan Chen, Kun Zhao, and Huajun Chen. 2021. Improving Conversa- tional Recommendation System by Pretraining on Billions Scale of Knowledge Graph. InProceedings of the IEEE International Conference...
2021
-
[29]
Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, et al . 2023. A Survey on Large Language Models for Recommendation.arXiv preprint arXiv:2305.19860(2023)
2023 arXiv
-
[30]
Kun Zhou, Wayne Xin Zhao, Shuqing Bian, Yuanhang Zhou, Ji-Rong Wen, and Jingsong Yu. 2020. Improving conversational recommender systems via knowl- edge graph based semantic fusion. InProceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mini...
2020
-
[31]
Yaochen Zhu, Harald Steck, Dawen Liang, Yinhan He, Vito Ostuni, Jundong Li, and Nathan Kallus. 2025. Rank-GRPO: Training LLM-based Conversational Rec- ommender Systems with Reinforcement Learning.arXiv preprint arXiv:2510.20150 (2025)
2025
-
[32]
Yaochen Zhu, Chao Wan, Harald Steck, Dawen Liang, Yesu Feng, Nathan Kallus, and Jundong Li. 2025. Collaborative Retrieval for Large Language Model-based Conversational Recommender Systems. InProceedings of the ACM Web Conference 2025 (WWW). 3323–3334
2025
-
[2009]
InProceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (UAI)
BPR: Bayesian Personalized Ranking from Implicit Feedback. InProceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (UAI). 452– 461
-
[2016]
InProceedings of the 1st Workshop on Deep Learning for Recommender Systems
Wide & Deep Learning for Recommender Systems. InProceedings of the 1st Workshop on Deep Learning for Recommender Systems. 7–10
-
[2019]
InProceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM)
BERT4Rec: Sequential Recommendation with Bidirectional Encoder Rep- resentations from Transformer. InProceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM). 1441–1450
-
[2026]
Nature Communications(2026)
Bayesian teaching enables probabilistic reasoning in large language models. Nature Communications(2026)
2026
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.