Metamorphic Evaluation of ChatGPT as a Recommender System

Madhurima Khirbat; Mark Sanderson; Pablo Castells; Yongli Ren

REVIEW 4 major objections 5 minor 23 references

Metamorphic Evaluation of ChatGPT as a Recommender System

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GPT-3.5 as a recommender is unstable under rating-scale and prompt perturbations, so LLM recommenders need an evaluation framework beyond traditional metrics.

desk verdict First metamorphic-testing study of an LLM recommender, but an undefended oracle assumption and circular parameter selection undercut the headline conclusion. read the letter →

arxiv 2411.12121 v1 pith:7PEZLHAD submitted 2024-11-18 cs.IR

Madhurima Khirbat , Yongli Ren , Pablo Castells , Mark Sanderson This is my paper · ORCID

classification cs.IR

keywords metamorphictestingrecommendersystemslargelanguagemodelsChatGPTpromptperturbationrankingsimilaritytestoracleMovieLens

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Metamorphic testing checks a system's outputs against relations between inputs, rather than against a known correct answer. The paper uses this to test GPT-3.5-turbo as a recommender: multiplying or shifting the numeric ratings in the prompt, or inserting spaces and random words, should in principle leave the user's preferences unchanged. In practice, these perturbations significantly changed the top-5 recommendation lists, with Kendall $\tau$ falling from 0.91 at baseline to between 0.06 and 0.50 depending on the perturbation (all $p < 0.0001$). The authors conclude that traditional offline evaluation metrics for recommender systems are not enough to assess LLM-based recommenders, and that a dedicated evaluation framework is needed.

What carries the argument

The central object is the metamorphic relation (MR), a specified transformation of the input prompt that should leave the recommendation output valid if the system is robust. The paper defines four: MR1 multiplies every rating and the rating scale by an integer; MR2 shifts every rating and the scale; MR3 inserts spaces between characters; MR4 inserts random words such as fruit names. Two similarity measures, Kendall $\tau$ (full-list order agreement) and Rank-Biased Overlap (top-weighted rank overlap), operationalize whether the relation is satisfied, and unpaired $t$-tests at the 95% confidence level identify significant deviations from the unperturbed baseline. The framework also calibrates prompt length ($l=20$) and list size ($k=5$) so that repeated unperturbed runs already agree closely, isolating the effect of the metamorphic perturbations from GPT's intrinsic output randomness.

What would settle it

Take the same users and hold out part of their MovieLens history, then compute whether the MR1-MR4 perturbed top-5 lists match held-out preferences as well as the unperturbed baseline list; if the perturbed lists are equally accurate, the low similarity reflects output variability rather than a failure that traditional metrics miss.

Watch

Extended reading notes

Core claim

The paper's central claim is that the same user preferences, expressed in different but semantically equivalent forms, produce markedly different top-5 recommendations from GPT-3.5, and this instability is invisible to standard accuracy and ranking metrics. The evidence is the drop in list similarity under all four metamorphic relations: rating multiplication (Kendall $\tau$ 0.4829, RBO 0.8496), rating shifting (0.4966, 0.8460), added spaces (0.0640, 0.4710), and added random words (0.2295, 0.6802), all statistically distinguishable from the baseline ($p < 0.0001$). The authors interpret this as a test-oracle problem: since the correct top-$k$ list is unknown, metamorphic relations are a way to evaluate consistency, and the observed violations signal that LLM-based recommenders cannot be assessed with the same metrics as traditional recommender systems.

Load-bearing premise

The argument assumes that a robust LLM recommender should return essentially the same top-5 list when ratings are multiplied or shifted and when spaces or random words are inserted into the prompt, and treats the unperturbed GPT-3.5 list as the correct baseline; if a legitimate recommender could reasonably change its output under these input changes, the similarity drops would not by themselves prove that a new evaluation framework is needed.

Editorial extensions

If this is right

A user who expresses the same preference on a 10-point scale instead of a 5-point scale can expect a substantially different top-5 recommendation list, since MR1 and MR2 change roughly half the ranking order.
Trivial surface changes to the prompt—extra spaces or random filler words—can almost completely destroy agreement with the unperturbed list, implying GPT-3.5's recommendations are tightly coupled to tokenization and prompt form.
Accuracy and ranking metrics calculated against known user preferences do not by themselves reveal this input sensitivity; evaluating an LLM recommender requires checking output consistency across semantic-preserving input variants.
Any metamorphic evaluation of a probabilistic LLM recommender must first control the model's output randomness, as the paper does by tuning $l$ and $k$, otherwise random variation swamps the signal of interest.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: The paper counts all divergence from the unperturbed list as a defect, but a perturbed list could still be a good recommendation; measuring each perturbed list's accuracy against held-out preferences would separate benign output variability from genuine failure.
Inference: The MR3 result (spaces) is consistent with tokenization sensitivity rather than preference corruption, since inserting spaces changes the model's input tokens; a testable extension is to check whether MR3 violations shrink with a model that operates on more robust tokenization.
Inference: A natural next experiment is to apply the same metamorphic relations to a deterministic non-LLM recommender on the same ratings as a control; if the control is invariant under rating rescaling, the instability is specific to the LLM, strengthening the paper's conclusion, whereas if the control also changes, the relations themselves may be too strict.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a legitimate but thin empirical paper. It is the first to run metamorphic testing on an LLM-based recommender, combining rating-scale MRs from traditional RS with prompt MRs from LLM testing. The measurements are probably real: under MR3 (inserted spaces) the top-5 list barely overlaps with the unperturbed output, and even rating-scale changes cut Kendall tau roughly in half. If you work on LLM recommenders, that is worth knowing.

The main soft spot is the oracle assumption. The paper treats the unperturbed GPT-3.5 output as the correct list and counts any divergence as instability. It never defends the premise that a good recommender should be invariant to '4/10' vs '2/5' or to spaces inserted into words. For a language model, those operations are not semantic-preserving, and a different ranking may be a faithful response to a different input representation. The paper even claims in Section 3.3.2 that the prompt perturbations are semantic-preserving, which is plainly wrong for random-word insertion and for spaces that break tokenization. So the reported Kendall and RBO drops demonstrate prompt sensitivity, which is already known, not necessarily a failure of the recommender or of traditional evaluation metrics.

The second issue is the parameter selection. l=20 and k=5 are chosen because they give the highest iteration-to-iteration similarity on the same metrics later used for the MR evidence. That lookahead inflates the baseline and makes the perturbation drops look larger. An independent validation set or a justification based on user coverage would be needed.

Minor things: the dataset is described as MovieLens 100k but the reference cites MovieLens 1m; no code or API configuration is provided; the t-test is described as unpaired but the design is clearly paired. Also '10 iterations' is small, though the SDs are small enough that this may not matter.

What the paper does not do is compare against any traditional recommender metric. Without showing that, e.g., NDCG or Recall would not flag these same perturbations, the conclusion that LLM-based RS need a new evaluation framework is not established. The evidence supports a weaker and still useful claim: these perturbations should be reported when evaluating LLM recommenders.

Who it's for: people evaluating LLM-based recommenders and wanting a checklist of robustness checks. It deserves a serious referee because the combination is novel and the underlying question is real, but acceptance should hinge on fixing the oracle assumption and the circular parameter selection.

Recommendation: send it to peer review, with a strong request for major revision.

Referee Report

4 major / 5 minor

Summary. This paper proposes a metamorphic testing framework for evaluating large language model based recommender systems. The authors define four metamorphic relations: rating multiplication (MR1), rating shifting (MR2), addition of spaces (MR3), and insertion of random words (MR4). Using GPT-3.5-turbo on MovieLens 100k, they compare top-5 recommendation lists under each MR against an unperturbed baseline, measuring similarity with Kendall tau, Rank-Biased Overlap, and overlap ratio. They report significant drops in similarity for all MRs (p < 0.0001) and conclude that traditional offline recommender system metrics are insufficient for LLM-based recommenders. The paper also includes a randomness-control step to choose the number of history items l and list size k.

Significance. If the central claim is supported, the paper makes a useful contribution as an early application of metamorphic testing to LLM-based recommenders, and the randomness-control framework is a reasonable methodological component. The empirical measurements are clearly reported with standard deviations and significance tests. However, the interpretation hinges on an unstated oracle assumption—that a robust recommender should be invariant to the chosen perturbations—which is not validated. The paper's significance is therefore conditional on a stronger justification of the metamorphic relations as semantic-preserving, or a careful restatement of the conclusion as demonstrating sensitivity rather than defect.

major comments (4)

[Section 3.3.2 and Table 4] The paper calls MR3 (adding spaces) and MR4 (adding random words) "semantic-preserving prompt perturbation," but inserting spaces disrupts tokenization and inserting random words such as "banana" and "grape" changes the instruction (e.g., "as a banana recommender system"). The paper never justifies why a correct recommender should be invariant to these changes. Since the oracle is the model's own unperturbed output, low similarity in Table 4 may reflect legitimate sensitivity to prompt phrasing rather than an evaluative shortcoming. This assumption is load-bearing for the conclusion; without it, the experiments only confirm known prompt sensitivity. I suggest validating the oracle (e.g., by human annotation that the perturbed prompts retain the same recommendation task) or reinterpreting the results as robustness evidence rather than evidence that new metrics are needed.
[Section 4.2 and Tables 2-3] The hyperparameters k=5 and l=20 are selected using the same similarity metrics (Kendall tau, RBO, overlap) that later constitute the evidence in Table 4. Choosing the configuration with the highest unperturbed consistency (k=5; see Table 2) and a l value with high consistency (Table 3) ensures a highly stable baseline, which may inflate the contrast with perturbed outputs. The paper should report the MR results for a range of k and l values, or provide a principled selection criterion independent of the evaluation metrics, to rule out that the observed drops are an artifact of the chosen baseline.
[Sections 4.3 and Table 4] The baseline is described as "one iteration of top-5 recommendations" in Section 4.3, yet Table 4 lists a "No change (baseline)" row with a mean and standard deviation over multiple runs, and the t-test is described as comparing "each MR output and the baseline list." Please clarify the exact composition of the baseline (a single list vs. a distribution) and how the unpaired t-test is computed with that baseline, as this affects the statistical claims.
[Section 4.4] The stated limitation, "there are certain limitation to the experiments which includes internal validity," is too vague and does not actually identify the oracle assumption or the selection bias described above. The discussion should explicitly acknowledge that the MRs' invariance assumptions are unverified and that the conclusions depend on them.

minor comments (5)

[Section 3.3.1] The definitions of MR1 and MR2 ("R/5 becomes λ[R/5]" and "(λ+R)/(λ+5)") are unclear because the total-rating phrase and the scale change are not specified; please rewrite with the actual prompt scales (e.g., 5-point to 10-point and 5-point to 6-point).
[Table 3 and Section 4.2] The justification for choosing l=20 is that it "represents a broader range of user interests while requiring less computational time," but Table 3 shows l=30 with higher similarity; please explain why l=20 is preferred.
[Section 4.1] The reference [23] points to the MovieLens 1M dataset while the text says MovieLens 100k; please correct the mismatch.
[Abstract and Section 4.4] The phrases "lower similarity are obtained" and "there are certain limitation" should be grammatically corrected for clarity.
[Section 2.3] The attribution of metamorphic testing to "Mao et. al." while citing Chen et al. for the general definition could be clarified to avoid confusion about historical priority.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the MR evaluations are empirical comparisons and do not reduce to their inputs.

full rationale

The paper's derivation chain is not circular. The four metamorphic relations (rating multiplication/shifting and adding spaces/random words) are input-side transformations defined in Section 3.3 and are drawn from external prior work ([10], [11]), not from the authors' own conclusions. The outputs are measured with Kendall tau and RBO against an unperturbed prompt; the reported low similarities are empirical results, not consequences of the MR definitions. The randomness-control step (Section 3.4) selects l=20 and k=5 to obtain a stable unperturbed baseline, but this is a methodological calibration for consistency, not a fitted 'prediction'—the MR outcomes in Table 4 are then independently observed. The central interpretation does depend on an unverified oracle assumption, namely that a robust recommender should be invariant under MR1–MR4, and MR3/MR4 may not be semantic-preserving; however, an unsupported assumption is a correctness/validity risk, not circularity, because the measured similarities are not forced by construction nor by a self-citation chain. There are no load-bearing self-citations by the present authors. Score 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper contributes a prompt template and measurements; the remaining structure rests on borrowed MR definitions, standard metrics, and one model's behavior. Five free parameters are chosen on the basis of the same data that produces the headline results, and the central oracle assumption is unverified.

free parameters (5)

k (top-k list size) = 5
Selected in Section 4.2 as the best-performing k on consistency metrics; determines the length of all evaluated lists.
l (number of user-history items in prompt) = 20
Selected from Table 3 based on high Kendall/RBO/overlap and computational cost; affects all MR measurements.
Rating filter threshold = 3 (only ratings > 3 included)
Positive-preference cutoff used to build prompts in Section 4.2; not justified or varied.
MR1 multiplier lambda = 2 (ratings doubled, 5-point to 10-point scale)
One value used in Table 1; no sweep over lambda.
MR2 shift lambda = 1 (ratings shifted up by 1, 5-point to 6-point scale)
One value used in Table 1; no sweep over lambda.

assumptions (4)

domain assumption A robust recommender should be invariant to rating multiplication and shifting (MR1, MR2) and to word-level prompt perturbations (MR3, MR4)
Defines the oracle for metamorphic testing in Section 3.3; without it, low similarity is not evidence of a defect.
domain assumption The model's unperturbed output is a valid reference oracle for testing
Section 4.3 compares all MR outputs against a no-change baseline; the baseline's correctness is not established.
domain assumption User histories with ratings > 3 capture positive preference sufficient for top-k recommendation
Prompt construction in Section 4.2 filters on this threshold without validation.
standard math Kendall tau, RBO, and overlap ratio are valid measures of metamorphic relation satisfaction
Adopted from prior evaluation literature; standard measures of ranking agreement.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Metamorphic Evaluation of ChatGPT as a Recommender System." pith.science (2026). https://pith.science/paper/7PEZLHAD

@misc{pith2026241112121,
  author       = {Pith},
  title        = {Pith review of: Metamorphic Evaluation of ChatGPT as a Recommender System},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7PEZLHAD}},
  note         = {Machine review of arXiv:2411.12121}
}

abstract

With the rise of Large Language Models (LLMs) such as ChatGPT, researchers have been working on how to utilize the LLMs for better recommendations. However, although LLMs exhibit black-box and probabilistic characteristics (meaning their internal working is not visible), the evaluation framework used for assessing these LLM-based recommender systems (RS) are the same as those used for traditional recommender systems. To address this gap, we introduce the metamorphic testing for the evaluation of GPT-based RS. This testing technique involves defining of metamorphic relations (MRs) between the inputs and checking if the relationship has been satisfied in the outputs. Specifically, we examined the MRs from both RS and LLMs perspectives, including rating multiplication/shifting in RS and adding spaces/randomness in the LLMs prompt via prompt perturbation. Similarity metrics (e.g. Kendall tau and Ranking Biased Overlap(RBO)) are deployed to measure whether the relationship has been satisfied in the outputs of MRs. The experiment results on MovieLens dataset with GPT3.5 show that lower similarity are obtained in terms of Kendall $\tau$ and RBO, which concludes that there is a need of a comprehensive evaluation of the LLM-based RS in addition to the existing evaluation metrics used for traditional recommender systems.

Figures

Figures reproduced from arXiv: 2411.12121 by the authors.

**Figure 1.** Overview with MR1 as an example The metamorphic testing operates by generating source test inputs by using any test case generation strategies, and MRs are defined for these inputs on the basis of the properties of the system. The follow up test inputs are generated on the basis of the defined MRs and then the outputs for both the inputs are calculated and evaluated. If the defined MR is maintained between the outpu… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 14 canonical work pages

[1]

Is chatgpt a good recom- mender? a preliminary study,

J. Liu, C. Liu, P. Zhou, R. Lv, K. Zhou, and Y. Zhang, “Is chatgpt a good recom- mender? a preliminary study, ” 2023

work page 2023
[2]

Evaluating ChatGPT as a recommender system: A rigorous approach

D. Di Palma, G. M. Biancofiore, V. W. Anelli, F. Narducci, T. Di Noia, and E. Di Sciascio, “Evaluating ChatGPT as a recommender system: A rigorous approach. ” [Online]. Available: http://arxiv.org/abs/2309.03613

arXiv
[4]

Recommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5),

S. Geng, S. Liu, Z. Fu, Y. Ge, and Y. Zhang, “Recommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5), ” inProceedings of the 16th ACM Conference on Recommender Systems , ser. RecSys ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 299–315. [Online]. Available: https://doi.org/10.11...

arXiv 2022
[5]

Large language mod- els meet collaborative filtering: An efficient all-round llm-based recommender system,

S. Kim, H. Kang, S. Choi, D. Kim, M. Yang, and C. Park, “Large language mod- els meet collaborative filtering: An efficient all-round llm-based recommender system, ” 2024

work page 2024
[6]

Chat-rec: Towards interactive and explainable llms-augmented recommender system,

Y. Gao, T. Sheng, Y. Xiang, Y. Xiong, H. Wang, and J. Zhang, “Chat-rec: Towards interactive and explainable llms-augmented recommender system, ” 2023

work page 2023
[7]

The simpson’s paradox in the offline evaluation of recommendation systems,

A. H. Jadidinejad, C. Macdonald, and I. Ounis, “The simpson’s paradox in the offline evaluation of recommendation systems, ” vol. 40, no. 1, pp. 4:1–4:22. [Online]. Available: https://dl.acm.org/doi/10.1145/3458509

doi:10.1145/3458509
[8]

Data-efficient fine-tuning for llm-based recommendation,

X. Lin, W. Wang, Y. Li, S. Yang, F. Feng, Y. Wei, and T.-S. Chua, “Data-efficient fine-tuning for llm-based recommendation, ” 2024

work page 2024
[9]

Metamorphic testing: A review of challenges and opportunities,

T. Y. Chen, F.-C. Kuo, H. Liu, P.-L. Poon, D. Towey, T. H. Tse, and Z. Q. Zhou, “Metamorphic testing: A review of challenges and opportunities, ” vol. 51, no. 1, pp. 1–27. [Online]. Available: https://dl.acm.org/doi/10.1145/3143561

doi:10.1145/3143561

Show all 23 references

[10]

An empirical study on metamorphic testing for recommender systems,

C. Mao, J. Chen, X. Yi, and L. Wen, “An empirical study on metamorphic testing for recommender systems, ” vol. 169, p. 107410. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0950584924000156
[11]

METAL: Metamorphic testing framework for analyzing large-language model qualities

S. Hyun, M. Guo, and M. A. Babar, “METAL: Metamorphic testing framework for analyzing large-language model qualities. ” [Online]. Available: http: //arxiv.org/abs/2312.06056

arXiv
[12]

Testing chatbots using metamorphic relations,

J. Bozic and F. Wotawa, “Testing chatbots using metamorphic relations, ” inTesting Software and Systems, C. Gaston, N. Kosmatov, and P. Le Gall, Eds. Springer International Publishing, pp. 41–55
[13]

Language models as recommender systems: Evaluations and limitations,

Y. Zhang, H. Ding, Z. Shui, Y. Ma, J. Zou, A. Deoras, and H. Wang, “Language models as recommender systems: Evaluations and limitations, ” in NeurIPS 2021 Workshop on I (Still) Can’t Believe It’s Not Better , 2021. [Online]. Available: https://www.amazon.science/publications/l...

2021
[14]

Ppt: Pre-trained prompt tuning for few-shot learning,

Y. Gu, X. Han, Z. Liu, and M. Huang, “Ppt: Pre-trained prompt tuning for few-shot learning, ” 2022

2022
[15]

Making pre-trained language models better few-shot learners,

T. Gao, A. Fisch, and D. Chen, “Making pre-trained language models better few-shot learners, ” inProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long P...

2021
[16]

The turking test: Can language models understand instruc- tions?

A. Efrat and O. Levy, “The turking test: Can language models understand instruc- tions?” 2020

2020
[17]

Llm based generation of item- description for recommendation system,

A. Acharya, B. Singh, and N. Onoe, “Llm based generation of item- description for recommendation system, ” in Proceedings of the 17th ACM Conference on Recommender Systems , ser. RecSys ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 1204–1207. [Online]. ...

2023
[18]

A survey on conversational recommender systems,

D. Jannach, A. Manzoor, W. Cai, and L. Chen, “A survey on conversational recommender systems, ” vol. 54, no. 5, pp. 105:1–105:36. [Online]. Available: https://dl.acm.org/doi/10.1145/3453154

doi
[19]

Evaluating collaborative filtering recommender systems,

J. L. Herlocker, J. A. Konstan, L. G. Terveen, and J. T. Riedl, “Evaluating collaborative filtering recommender systems, ”ACM Trans. Inf. Syst., vol. 22, no. 1, p. 5–53, jan 2004. [Online]. Available: https://doi.org/10.1145/963770.963772

2004
[20]

Cumulated gain-based evaluation of ir techniques,

K. Järvelin and J. Kekäläinen, “Cumulated gain-based evaluation of ir techniques, ” ACM Trans. Inf. Syst. , vol. 20, no. 4, p. 422–446, oct 2002. [Online]. Available: https://doi.org/10.1145/582415.582418

2002
[21]

Recommender systems: Past, present, future,

D. Jannach, P. Pu, F. Ricci, and M. Zanker, “Recommender systems: Past, present, future, ” vol. 42, no. 3, pp. 3–6. [Online]. Available: https: //onlinelibrary.wiley.com/doi/10.1609/aimag.v42i3.18139

doi
[22]

A prompt pattern catalog to enhance prompt engi- neering with chatgpt,

J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. Elnashar, J. Spencer- Smith, and D. C. Schmidt, “A prompt pattern catalog to enhance prompt engi- neering with chatgpt, ” 2023

2023
[23]

[Online]

Mar 2021. [Online]. Available: https://grouplens.org/datasets/movielens/1m/

2021
[24]

A similarity measure for indefinite rankings,

W. Webber, A. Moffat, and J. Zobel, “A similarity measure for indefinite rankings, ”ACM Trans. Inf. Syst. , vol. 28, no. 4, nov 2010. [Online]. Available: https://doi.org/10.1145/1852102.1852106

2010

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Is chatgpt a good recom- mender? a preliminary study,

J. Liu, C. Liu, P. Zhou, R. Lv, K. Zhou, and Y. Zhang, “Is chatgpt a good recom- mender? a preliminary study, ” 2023

work page 2023

[2] [2]

Evaluating ChatGPT as a recommender system: A rigorous approach

D. Di Palma, G. M. Biancofiore, V. W. Anelli, F. Narducci, T. Di Noia, and E. Di Sciascio, “Evaluating ChatGPT as a recommender system: A rigorous approach. ” [Online]. Available: http://arxiv.org/abs/2309.03613

arXiv

[3] [4]

Recommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5),

S. Geng, S. Liu, Z. Fu, Y. Ge, and Y. Zhang, “Recommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5), ” inProceedings of the 16th ACM Conference on Recommender Systems , ser. RecSys ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 299–315. [Online]. Available: https://doi.org/10.11...

arXiv 2022

[4] [5]

Large language mod- els meet collaborative filtering: An efficient all-round llm-based recommender system,

S. Kim, H. Kang, S. Choi, D. Kim, M. Yang, and C. Park, “Large language mod- els meet collaborative filtering: An efficient all-round llm-based recommender system, ” 2024

work page 2024

[5] [6]

Chat-rec: Towards interactive and explainable llms-augmented recommender system,

Y. Gao, T. Sheng, Y. Xiang, Y. Xiong, H. Wang, and J. Zhang, “Chat-rec: Towards interactive and explainable llms-augmented recommender system, ” 2023

work page 2023

[6] [7]

The simpson’s paradox in the offline evaluation of recommendation systems,

A. H. Jadidinejad, C. Macdonald, and I. Ounis, “The simpson’s paradox in the offline evaluation of recommendation systems, ” vol. 40, no. 1, pp. 4:1–4:22. [Online]. Available: https://dl.acm.org/doi/10.1145/3458509

doi:10.1145/3458509

[7] [8]

Data-efficient fine-tuning for llm-based recommendation,

X. Lin, W. Wang, Y. Li, S. Yang, F. Feng, Y. Wei, and T.-S. Chua, “Data-efficient fine-tuning for llm-based recommendation, ” 2024

work page 2024

[8] [9]

Metamorphic testing: A review of challenges and opportunities,

T. Y. Chen, F.-C. Kuo, H. Liu, P.-L. Poon, D. Towey, T. H. Tse, and Z. Q. Zhou, “Metamorphic testing: A review of challenges and opportunities, ” vol. 51, no. 1, pp. 1–27. [Online]. Available: https://dl.acm.org/doi/10.1145/3143561

doi:10.1145/3143561