Architecture is All You Need: Improving LLM Recommenders by Dropping the Text

Kavya Priyanka Kakinada; Kevin Foley; Shaghayegh Agah

REVIEW 3 major objections 5 minor 15 references

Architecture is All You Need: Improving LLM Recommenders by Dropping the Text

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 2.2M-parameter recommender that uses Llama's architecture with item IDs beats both SASRec and 7B-parameter BigRec.

desk verdict The empirical result is solid and worth publishing; the causal 'architecture' claim is underdetermined by the experiments as run. read the letter →

arxiv 2506.15833 v1 pith:KRXIV7H6 submitted 2025-06-18 cs.IR

Kevin Foley , Shaghayegh Agah , Kavya Priyanka Kakinada This is my paper

classification cs.IR

keywords sequentialrecommendationLLMarchitectureitem-leveltokenizationmulti-tasklearningMovieLens-10Msupervisedfine-tuningefficientrecommendersnext-itemprediction

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to separate two things that usually come bundled in large language models: the transformer architecture's ability to learn patterns from sequences, and the world knowledge stored in pretrained text weights. It proposes LSRec, a recommender that keeps the Llama-style causal transformer architecture but replaces all text with discrete item-ID tokens and trains from scratch on interaction sequences only. On MovieLens-10M, a 2.2M-parameter version of this model achieves better hit rates and NDCG than a 7B-parameter PLM-based recommender (BigRec) and than the classic SASRec transformer, at a fraction of the compute. The authors take this as evidence that the main benefit of LLMs for recommendation is the architecture itself, not pretrained world knowledge. If true, it would make high-quality LLM-style recommenders much cheaper and easier to deploy.

What carries the argument

The central object is LSRec, a lightweight sequential recommender built on the Llama 3 decoder-only transformer architecture but with the subword vocabulary replaced by discrete tokens that each stand for one content item, plus a few special tokens for prompting. It uses rotary position embeddings, grouped query attention with a single key-value head, RMS normalization, and SiLU activation instead of SwiGLU, which shrinks the model to between 768 thousand and 7.1 million parameters. Training is supervised fine-tuning on a multi-task prompt template where the user's history is a sequence of item IDs and the target is the next item ID, and at inference the ranked list is produced by taking the top K logits of a single generated token.

What would settle it

Train a SASRec-style architecture using LSRec's supervised-fine-tuning objective over all items with the same 200-item context length; if it matches LSRec's HR@1 of about 0.095, then the architecture claim is false. Alternatively, replace LSRec's Llama-style transformer blocks with SASRec-style blocks while holding everything else constant and check whether the performance gap disappears.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a causal decoder-only transformer trained on raw item IDs, with no language modeling, no text prompts, and no pretrained weights, outperforms both a strong classic sequential recommender and a fine-tuned 7B-parameter PLM recommender on a standard benchmark. The best configuration, LSRec-medium-mt, gets HR@1 of 0.0952 and HR@10 of 0.3129, compared with 0.0323 and 0.0456 for the fully fine-tuned BigRec and 0.0142 and 0.1288 for SASRec. The paper interprets this as suggesting that the architecture of LLMs, rather than the world knowledge acquired during pretraining, is the principal source of their value in recommendation.

Load-bearing premise

The experiments attribute LSRec's gains to the Llama architecture, but compared with SASRec they also change the training objective, position embeddings, attention mechanism, normalization, and feed-forward design, and compared with BigRec they change context length and ranking, so if one of those other differences drives the gains, the architecture-only conclusion collapses.

Editorial extensions

If this is right

PLM-based recommenders can be replaced by models hundreds of times smaller in data-rich settings, with large gains in accuracy and a huge reduction in compute.
Because LSRec uses a Llama architecture, it can share the open-source ecosystem of LLM training and serving tools even though it processes no text.
Multi-task training on genre, rating, movie-by-genre, and movie-by-rating tasks improves the main recommendation task across every model size tested.
Token-based recommenders can use a 200-item context window for the same compute that limits PLM-based recommenders to about ten items, giving them more history per user.
Performance plateaus around 2 million parameters on MovieLens-10M, suggesting that larger models of this class only pay off on much larger datasets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: the 'architecture' label covers several differences at once; compared with SASRec the training objective changes from negative sampling to full-vocabulary supervised fine-tuning, and compared with BigRec the context length, tokenization, and ranking mechanism all change, so the architecture-only conclusion still needs ablations that isolate each factor.
Beyond the paper: the multi-task prompting design suggests a single small token-based model could serve rating prediction, genre prediction, and conditioned recommendation as one shared model, though the paper does not evaluate the auxiliary tasks.
Beyond the paper: if the drop-the-text result generalizes, the same approach could be tried for other domains where pretrained text knowledge is weak or stale, replacing text tokens with task-specific discrete tokens and training from scratch.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the paper delivers a genuinely useful empirical data point — a 2.2M-parameter, text-free, Llama-style transformer with multi-task prompts beats a replicated SASRec and a 7B BigRec on MovieLens-10M by wide margins. If you build recommenders, this is the kind of result that makes you reconsider whether you need a PLM at all. The paper is clear, the baseline replications are a real effort, and the prompt/task design is a nice contribution.

The soft spot is the title-level claim. The authors attribute the gains to 'architecture' and dismiss pretrained knowledge, but the comparisons change many variables at once. LSRec sees up to 200 items; BigRec, following its published procedure, sees only 10. That alone can produce large gains in a sequential task. Training objectives also differ: LSRec uses full-vocabulary SFT plus auxiliary tasks, SASRec uses negative sampling, BigRec uses next-title generation. LSRec also differs from SASRec in position embeddings, attention heads, FFN, normalization. The paper's own Section 4 lists these and says 'additional investigation is needed' — which is honest but concedes the causal claim is underdetermined. A matched-context ablation, plus controlling the training objective, is required before 'architecture is all you need' can be taken as stated.

Other weaknesses are proportionate: one dataset (MovieLens-10M), one domain, no error bars or repeated seeds, no code release apparent. The comparison to SPRec/DPO baselines leans on numbers reported elsewhere rather than independent replication, which is okay but should be labeled as such. The diversity metric is also a bit unusual (unique items over vocabulary) and less interpretable.

So: the descriptive result is credible and important for practitioners; the interpretive claim is premature. This is a solid submission for a venue that will push for the right control experiments. I'd send it to review, but the reviewers should require the matched-context ablation and error bars before acceptance.

Referee Report

3 major / 5 minor

Summary. The paper proposes LSRec, a family of small decoder-only transformers that follow the Llama 3 architecture but operate on discrete item tokens rather than text subwords. The models are trained from scratch on MovieLens-10M with supervised fine-tuning, optionally with four auxiliary tasks, and evaluated on next-item recommendation against replicated SASRec and BigRec baselines. The authors report large performance gains (e.g., HR@10 0.3129 for LSRec-medium-mt versus 0.0456 for BigRec-full and 0.1288 for SASRec) and conclude that the principal benefit of LLMs for recommendation is their architecture rather than pretrained world knowledge.

Significance. If the central claim were established, the paper would make a useful practical contribution: it would show that small, item-token transformer models can outperform much larger PLM-based recommenders at a fraction of the computational cost, and it would sharpen the debate about what PLMs actually contribute to recommendation. The manuscript has notable strengths: the architecture is specified clearly in Table 1 and Figure 1; the authors replicate both SASRec and BigRec rather than relying only on published numbers; they report several model sizes and a clean multi-task ablation; and the evaluation protocol is standard for the field. However, the headline causal claim is currently underdetermined because the LSRec and BigRec systems differ along several dimensions simultaneously, and the empirical evidence rests on a single dataset with single-run point estimates.

major comments (3)

[§4, Table 3 and Table 1] The central claim that the gains come from the 'architecture' rather than world knowledge is confounded. LSRec and BigRec differ not only in architecture but also in context length (200 items for LSRec versus 10 items for BigRec following the BigRec procedure), in training objective (SFT over the full item vocabulary plus four auxiliary tasks versus next-title generation), and in ranking mechanism (direct top-K logits versus KNN retrieval over generated titles). Because longer context alone is a known source of large gains in sequential recommendation, the observed HR@1 gap (0.0952 versus 0.0323) cannot be uniquely attributed to the decoder-only architecture. The paper itself says in §4 that 'additional investigation is needed' for the SASRec comparison, but the same concern applies with even more force to the BigRec comparison. A matched-context ablation (for example, training LSRec with a 10-item context, or extending the PLM baseline to a longer context where feasible) and an ablation that removes the auxiliary tasks is required before the title-level conclusion can be accepted.
[§3, Metrics and evaluation protocol] All reported results are single-run point estimates on a single dataset with a single temporal split. No repeated seeds, error bars, or significance tests are provided. This is not fatal given the size of the reported gaps, but it weakens the precision of the comparison, especially for the smaller differences between adjacent LSRec variants (e.g., LSRec-medium-mt versus LSRec-large-mt in Table 3). Reporting at least three seeds for the main comparisons, or providing confidence intervals, would make the descriptive claims appreciably more robust.
[§3.2, Table 4] The comparison against DPO-tuned PLM recommenders (SDPO and SPRec) is based on numbers taken directly from the SPRec paper rather than independent replication. Because Table 4 also reports a new 1,000-user random sample for LSRec and the replicated baselines, it is important to verify that the sampling procedure, the candidate set, and the metric computation exactly match the SPRec protocol. A mismatch in any of these details could make the cross-paper comparisons in Table 4 misleading. At minimum, the authors should state the source of each column and the exact sampling seed or procedure.

minor comments (5)

[§2 and Table 1] The text says the model uses 'a single key-value attention head' while Table 1 lists one KV head and one or two attention heads depending on the variant, and the caption mentions 'Grouped query attention.' The terminology should be harmonized so the reader can see exactly which attention configuration each LSRec size uses.
[§3, Experimental setup] The paper states in §4 that SASRec had a context window of up to 200 items, but the experimental setup section does not give a comparable configuration table for SASRec. Since the comparison with SASRec is important, the context length, number of layers, hidden size, and negative-sampling details should be reported explicitly.
[Table 3, Diversity Ratio] The definition of Diversity Ratio is given as the number of unique recommended items over the vocabulary size, but it is not clear whether this is averaged over users or computed globally. A sentence clarifying the exact formula would prevent misinterpretation, especially because Table 4's diversity values change substantially with sample size.
[Abstract and conclusion] The phrase 'Architecture is All You Need' is stronger than the evidence presented, since the proposed model also changes tokenization, training objective, context length, and multitask training relative to the baselines. A more measured phrasing in the abstract, such as 'a small decoder-only architecture with item tokens is sufficient to outperform these baselines,' would better match the results.
[References] Reference [10] has an arXiv identifier (2308.08747) that is inconsistent with its stated publication year 2025; the citation should be checked and, if the preprint has been updated, the correct version and URL should be provided.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline claim is an empirical comparison against replicated baselines, not a derivation from its own outputs.

full rationale

The paper's central claim is an empirical comparison: LSRec, a Llama-3-style decoder-only transformer trained from scratch on item-ID tokens, outperforms SASRec and BigRec on MovieLens-10M. The load-bearing numbers in Table 3 come from training and evaluating the proposed models and baselines on the same external dataset, with no parameter fitted to the target metric and then renamed a prediction, and no result that is definitionally identical to a training input. The architecture is adopted from an external source (the Llama 3 model family) and the item-level tokenization is a standard sequential-recommendation convention; neither premise rests on a self-citation chain or an imported uniqueness theorem. Section 4 explicitly lists possible alternative explanations for the gains, including the much longer context window of LSRec versus BigRec and the difference between SFT and negative-sampling training, and notes that 'additional investigation is needed.' Those confounds are a threat to the causal attribution of the performance gap to the architecture, but they are experimental-validity concerns rather than circular derivation: the comparison does not reduce to its own inputs by construction. No specific equation, fitted parameter, or self-citation can be exhibited that makes a predicted quantity equivalent to an input, so under the stated rules the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No analytic free parameters or new physical entities are introduced; the model is a standard supervised recommender. Entries above are the hand-chosen design constants that the empirical claim depends on, plus the evaluation assumptions needed for the comparison to be meaningful.

free parameters (4)

context window length = 200 tokens
Hard truncation of user-item sequences; set in Section 2.1 without tuning, but it changes which interactions the model can use.
dropout = 0.2
Chosen by hand in Table 1; no ablation or tuning is reported.
architecture dimensions = three sizes: 768K, 2.2M, 7.1M params
Hidden dims, layers, heads, and intermediate dims are hand-specified; the paper's best-performing size is selected from these, not from a search.
early stopping patience = 20 for non-PLM models, 5 for PLM models
Section 3.3; differing patience may affect relative performance and is not justified.

assumptions (4)

domain assumption Next-item prediction on MovieLens-10M is a valid proxy for recommendation quality.
Section 3 defines the evaluation as predicting the most recent held-out interaction; the paper does not validate this proxy against user studies.
domain assumption Replicated SASRec and BigRec implementations faithfully match the original papers.
Section 3.2 states replication but does not show an apples-to-apples match with original published numbers under the same protocol.
domain assumption SPRec and SDPO reported results in Table 4 are comparable to independently measured LSRec numbers.
Section 3.2 notes replication of those DPO baselines is still in progress; the comparison therefore assumes identical evaluation settings.
domain assumption Ranking by top-K logits from a single token is equivalent to full generative recommendation evaluation.
Section 3.1 uses one-token logits, not sequential generation; sequential generation may behave differently and is deferred to future work.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Architecture is All You Need: Improving LLM Recommenders by Dropping the Text." pith.science (2026). https://pith.science/paper/KRXIV7H6

@misc{pith2026250615833,
  author       = {Pith},
  title        = {Pith review of: Architecture is All You Need: Improving LLM Recommenders by Dropping the Text},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KRXIV7H6}},
  note         = {Machine review of arXiv:2506.15833}
}

read the original abstract

In recent years, there has been an explosion of interest in the applications of large pre-trained language models (PLMs) to recommender systems, with many studies showing strong performance of PLMs on common benchmark datasets. PLM-based recommender models benefit from flexible and customizable prompting, an unlimited vocabulary of recommendable items, and general ``world knowledge'' acquired through pre-training on massive text corpora. While PLM-based recommenders show promise in settings where data is limited, they are hard to implement in practice due to their large size and computational cost. Additionally, fine-tuning PLMs to improve performance on collaborative signals may degrade the model's capacity for world knowledge and generalizability. We propose a recommender model that uses the architecture of large language models (LLMs) while reducing layer count and dimensions and replacing the text-based subword tokenization of a typical LLM with discrete tokens that uniquely represent individual content items. We find that this simplified approach substantially outperforms both traditional sequential recommender models and PLM-based recommender models at a tiny fraction of the size and computational complexity of PLM-based models. Our results suggest that the principal benefit of LLMs in recommender systems is their architecture, rather than the world knowledge acquired during extensive pre-training.

Figures

Figures reproduced from arXiv: 2506.15833 by the authors.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 9 canonical work pages

[1]

A bi-step grounding paradigm for large language models in recommendation systems

Keqin Bao, Jizhi Zhang, Wenjie Wang, Yang Zhang, Zhengyi Yang, Yanchen Luo, Chong Chen, Fuli Feng, and Qi Tian. A bi-step grounding paradigm for large language models in recommendation systems. ACM Transactions on Recommender Systems, 3(4):1–27, 2025

work page 2025
[2]

Direct preference optimization: Your language model is secretly a reward model

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023

2023
[3]

On softmax direct preference optimization for recommendation

Yuxin Chen, Junfei Tan, An Zhang, Zhengyi Yang, Leheng Sheng, Enzhi Zhang, Xiang Wang, and Tat-Seng Chua. On softmax direct preference optimization for recommendation. arXiv preprint arXiv:2406.09215, 2024

arXiv 2024
[4]

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1891, 2020

work page 2020
[5]

Tallrec: An effective and efficient tuning framework to align large language model with recommendation

Keqin Bao, Jizhi Zhang, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. Tallrec: An effective and efficient tuning framework to align large language model with recommendation. In Proceedings of the 17th ACM Conference on Recommender Systems, pages 1007–1014, 2023

work page 2023
[6]

Sprec: Self-play to debias llm-based recommendation

Chongming Gao, Ruijun Chen, Shuai Yuan, Kexin Huang, Yuanqing Yu, and Xiangnan He. Sprec: Self-play to debias llm-based recommendation. In Proceedings of the ACM on Web Conference 2025 , pages 5075–5084, 2025

work page 2025
[7]

Rosepo: Aligning llm-based recommenders with human values

Jiayi Liao, Xiangnan He, Ruobing Xie, Jiancan Wu, Yancheng Yuan, Xingwu Sun, Zhanhui Kang, and Xiang Wang. Rosepo: Aligning llm-based recommenders with human values. arXiv preprint arXiv:2410.12519, 2024. 6 Architecture is All You Need

arXiv 2024
[8]

Llara: Large language-recommendation assistant

Jiayi Liao, Sihang Li, Zhengyi Yang, Jiancan Wu, Yancheng Yuan, Xiang Wang, and Xiangnan He. Llara: Large language-recommendation assistant. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 1785–1795, 2024

work page 2024

Show all 15 references

[9]

Gpt4rec: A generative framework for personalized recommendation and user interests interpretation

Jinming Li, Wentao Zhang, Tian Wang, Guanglei Xiong, Alan Lu, and Gerard Medioni. Gpt4rec: A generative framework for personalized recommendation and user interests interpretation. arXiv preprint arXiv:2304.03879, 2023

2023 arXiv
[10]

An empirical study of catastrophic forgetting in large language models during continual fine-tuning, 2025

Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning, 2025. URL https://arxiv.org/abs/2308. 08747

2025
[11]

Do llms memorize recommendation datasets? a preliminary study on movielens-1m

Dario Di Palma, Felice Antonio Merra, Maurizio Sfilio, Vito Walter Anelli, Fedelucio Narducci, and Tommaso Di Noia. Do llms memorize recommendation datasets? a preliminary study on movielens-1m. In Proceedings of the 48th International ACM SIGIR Conference on Research and Deve...

2025 arXiv
[12]

The llama 3 herd of models

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

2024 arXiv
[13]

Bandit based optimization of multiple objectives on a music streaming platform

Rishabh Mehrotra, Niannan Xue, and Mounia Lalmas. Bandit based optimization of multiple objectives on a music streaming platform. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages 3224–3233, 2020

2020
[14]

Self-attentive sequential recommendation

Wang-Cheng Kang and Julian McAuley. Self-attentive sequential recommendation. In 2018 IEEE international conference on data mining (ICDM) , pages 197–206. IEEE, 2018

2018
[15]

Multi-objective recommendation system utilizing a multi-population knowledge migration framework

Liang Chu and Ye Tian. Multi-objective recommendation system utilizing a multi-population knowledge migration framework. Complex & Intelligent Systems, 11(6):255, 2025. 7

2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

A bi-step grounding paradigm for large language models in recommendation systems

Keqin Bao, Jizhi Zhang, Wenjie Wang, Yang Zhang, Zhengyi Yang, Yanchen Luo, Chong Chen, Fuli Feng, and Qi Tian. A bi-step grounding paradigm for large language models in recommendation systems. ACM Transactions on Recommender Systems, 3(4):1–27, 2025

work page 2025

[2] [2]

Direct preference optimization: Your language model is secretly a reward model

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023

2023

[3] [3]

On softmax direct preference optimization for recommendation

Yuxin Chen, Junfei Tan, An Zhang, Zhengyi Yang, Leheng Sheng, Enzhi Zhang, Xiang Wang, and Tat-Seng Chua. On softmax direct preference optimization for recommendation. arXiv preprint arXiv:2406.09215, 2024

arXiv 2024

[4] [4]

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1891, 2020

work page 2020

[5] [5]

Tallrec: An effective and efficient tuning framework to align large language model with recommendation

Keqin Bao, Jizhi Zhang, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. Tallrec: An effective and efficient tuning framework to align large language model with recommendation. In Proceedings of the 17th ACM Conference on Recommender Systems, pages 1007–1014, 2023

work page 2023

[6] [6]

Sprec: Self-play to debias llm-based recommendation

Chongming Gao, Ruijun Chen, Shuai Yuan, Kexin Huang, Yuanqing Yu, and Xiangnan He. Sprec: Self-play to debias llm-based recommendation. In Proceedings of the ACM on Web Conference 2025 , pages 5075–5084, 2025

work page 2025

[7] [7]

Rosepo: Aligning llm-based recommenders with human values

Jiayi Liao, Xiangnan He, Ruobing Xie, Jiancan Wu, Yancheng Yuan, Xingwu Sun, Zhanhui Kang, and Xiang Wang. Rosepo: Aligning llm-based recommenders with human values. arXiv preprint arXiv:2410.12519, 2024. 6 Architecture is All You Need

arXiv 2024

[8] [8]

Llara: Large language-recommendation assistant

Jiayi Liao, Sihang Li, Zhengyi Yang, Jiancan Wu, Yancheng Yuan, Xiang Wang, and Xiangnan He. Llara: Large language-recommendation assistant. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 1785–1795, 2024

work page 2024