{"id":"85e79489-678e-412d-9d12-b5a1dff98fe3","arxiv_id":"2411.12121","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Metamorphic testing of GPT-3.5 on MovieLens shows rating-scale and prompt perturbations sharply reduce ranking similarity, suggesting LLM recommender evaluation needs new methods.","lead":"This paper tests whether OpenAI's ChatGPT gives consistent movie recommendations when the input ratings or prompt wording are changed in ways that should not matter. It finds that small changes, like switching rating scales or adding spaces to words, produce very different top-5 lists, and argues that LLM-based recommender systems need a separate evaluation framework.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Oracle assumption in MR1–MR4 is undefended; observed divergence may reflect legitimate sensitivity to prompt and scale changes rather than instability.","rationale":"The reader identified the oracle assumption as the weakest premise, and I agree. This is the single most load-bearing concern because the paper's headline contribution—the need for a new evaluation framework for LLM recommenders—is only warranted if the four metamorphic relations are valid oracles. The paper's empirical measurements are internally consistent and the randomness control is a reasonable attempt, though the post-hoc selection of l and k and the inconsistent dataset description are additional weaknesses. However, none of those are as central as the oracle. The proposed test would directly settle whether the MRs measure genuine instability or merely the model's sensitivity to format, which is a known phenomenon. The reader's CONDITIONAL verdict is appropriate: the paper should be accepted only if the oracle assumption is empirically validated or if the claims are softened to 'prompt sensitivity.' My concern does not move the verdict because it is already reflected in the reader's assessment; it reinforces the condition.","tokens_in":8901,"tokens_out":2815,"duration_ms":29056,"concrete_test":"For a random subset of 50 users, run the four MRs with a 'calibration' variant that makes semantic equivalence explicit: for MR1, write '2/5 (same as 4/10 on a 10-point scale)'; for MR2, write '3/6 (same as 2/5 on a 5-point scale)'; for MR3, use a paraphrase that preserves tokenization (e.g., insert extra spaces after punctuation only, not within words); for MR4, insert random words only after commas or in a separate 'ignore this' field. Compare these calibrated outputs to the same unperturbed baseline using Kendall tau and RBO. If similarity rises substantially (e.g., Kendall tau > 0.8), the original drops are caused by miscomprehension of the perturbed prompt, falsifying the oracle assumption and undermining the conclusion. If similarity remains low, the MRs test genuine robustness and the conclusion is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that low Kendall tau and RBO under MR1–MR4 show a need for new evaluation of LLM-based RS—depends on an unstated oracle: a robust recommender should produce the same top-k list under rating multiplication/shifting and under spaces/random words. This premise is never justified. MR3 inserts spaces that disrupt tokenization, and MR4 inserts random words that alter the instruction; for a language model these are not semantic-preserving operations, and different outputs are not necessarily defects. MR1 and MR2 change the numeric scale; a model may have learned that '4/10' and '2/5' are not interchangeable, so different recommendations could be a faithful response to a different preference representation. The paper treats any divergence from the unperturbed output as a failure, but it provides no evidence that a correct recommender would be invariant under these relations. Without validating the oracle, the empirical drops only demonstrate prompt sensitivity, which is already known for LLMs; they do not demonstrate that traditional RS metrics are insufficient. This is the load-bearing weak point because the entire conclusion rests on it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a metamorphic testing framework for evaluating large language model based recommender systems. The authors define four metamorphic relations: rating multiplication (MR1), rating shifting (MR2), addition of spaces (MR3), and insertion of random words (MR4). Using GPT-3.5-turbo on MovieLens 100k, they compare top-5 recommendation lists under each MR against an unperturbed baseline, measuring similarity with Kendall tau, Rank-Biased Overlap, and overlap ratio. They report significant drops in similarity for all MRs (p < 0.0001) and conclude that traditional offline recommender system metrics are insufficient for LLM-based recommenders. The paper also includes a randomness-control step to choose the number of history items l and list size k.","tokens_in":9106,"tokens_out":5842,"duration_ms":54032,"significance":"If the central claim is supported, the paper makes a useful contribution as an early application of metamorphic testing to LLM-based recommenders, and the randomness-control framework is a reasonable methodological component. The empirical measurements are clearly reported with standard deviations and significance tests. However, the interpretation hinges on an unstated oracle assumption—that a robust recommender should be invariant to the chosen perturbations—which is not validated. The paper's significance is therefore conditional on a stronger justification of the metamorphic relations as semantic-preserving, or a careful restatement of the conclusion as demonstrating sensitivity rather than defect.","major_comments":[{"comment":"The paper calls MR3 (adding spaces) and MR4 (adding random words) \"semantic-preserving prompt perturbation,\" but inserting spaces disrupts tokenization and inserting random words such as \"banana\" and \"grape\" changes the instruction (e.g., \"as a banana recommender system\"). The paper never justifies why a correct recommender should be invariant to these changes. Since the oracle is the model's own unperturbed output, low similarity in Table 4 may reflect legitimate sensitivity to prompt phrasing rather than an evaluative shortcoming. This assumption is load-bearing for the conclusion; without it, the experiments only confirm known prompt sensitivity. I suggest validating the oracle (e.g., by human annotation that the perturbed prompts retain the same recommendation task) or reinterpreting the results as robustness evidence rather than evidence that new metrics are needed.","section":"Section 3.3.2 and Table 4"},{"comment":"The hyperparameters k=5 and l=20 are selected using the same similarity metrics (Kendall tau, RBO, overlap) that later constitute the evidence in Table 4. Choosing the configuration with the highest unperturbed consistency (k=5; see Table 2) and a l value with high consistency (Table 3) ensures a highly stable baseline, which may inflate the contrast with perturbed outputs. The paper should report the MR results for a range of k and l values, or provide a principled selection criterion independent of the evaluation metrics, to rule out that the observed drops are an artifact of the chosen baseline.","section":"Section 4.2 and Tables 2-3"},{"comment":"The baseline is described as \"one iteration of top-5 recommendations\" in Section 4.3, yet Table 4 lists a \"No change (baseline)\" row with a mean and standard deviation over multiple runs, and the t-test is described as comparing \"each MR output and the baseline list.\" Please clarify the exact composition of the baseline (a single list vs. a distribution) and how the unpaired t-test is computed with that baseline, as this affects the statistical claims.","section":"Sections 4.3 and Table 4"},{"comment":"The stated limitation, \"there are certain limitation to the experiments which includes internal validity,\" is too vague and does not actually identify the oracle assumption or the selection bias described above. The discussion should explicitly acknowledge that the MRs' invariance assumptions are unverified and that the conclusions depend on them.","section":"Section 4.4"}],"minor_comments":[{"comment":"The definitions of MR1 and MR2 (\"R/5 becomes λ[R/5]\" and \"(λ+R)/(λ+5)\") are unclear because the total-rating phrase and the scale change are not specified; please rewrite with the actual prompt scales (e.g., 5-point to 10-point and 5-point to 6-point).","section":"Section 3.3.1"},{"comment":"The justification for choosing l=20 is that it \"represents a broader range of user interests while requiring less computational time,\" but Table 3 shows l=30 with higher similarity; please explain why l=20 is preferred.","section":"Table 3 and Section 4.2"},{"comment":"The reference [23] points to the MovieLens 1M dataset while the text says MovieLens 100k; please correct the mismatch.","section":"Section 4.1"},{"comment":"The phrases \"lower similarity are obtained\" and \"there are certain limitation\" should be grammatically corrected for clarity.","section":"Abstract and Section 4.4"},{"comment":"The attribution of metamorphic testing to \"Mao et. al.\" while citing Chen et al. for the general definition could be clarified to avoid confusion about historical priority.","section":"Section 2.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short conference-style paper (5 pages) and the main result would be more convincing with a larger-scale validation. The oracle assumption is a known difficulty in metamorphic testing; the authors should be encouraged to address it head-on. There is no indication of missing references beyond the dataset link mismatch."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a legitimate but thin empirical paper. It is the first to run metamorphic testing on an LLM-based recommender, combining rating-scale MRs from traditional RS with prompt MRs from LLM testing. The measurements are probably real: under MR3 (inserted spaces) the top-5 list barely overlaps with the unperturbed output, and even rating-scale changes cut Kendall tau roughly in half. If you work on LLM recommenders, that is worth knowing.\n\nThe main soft spot is the oracle assumption. The paper treats the unperturbed GPT-3.5 output as the correct list and counts any divergence as instability. It never defends the premise that a good recommender should be invariant to '4/10' vs '2/5' or to spaces inserted into words. For a language model, those operations are not semantic-preserving, and a different ranking may be a faithful response to a different input representation. The paper even claims in Section 3.3.2 that the prompt perturbations are semantic-preserving, which is plainly wrong for random-word insertion and for spaces that break tokenization. So the reported Kendall and RBO drops demonstrate prompt sensitivity, which is already known, not necessarily a failure of the recommender or of traditional evaluation metrics.\n\nThe second issue is the parameter selection. l=20 and k=5 are chosen because they give the highest iteration-to-iteration similarity on the same metrics later used for the MR evidence. That lookahead inflates the baseline and makes the perturbation drops look larger. An independent validation set or a justification based on user coverage would be needed.\n\nMinor things: the dataset is described as MovieLens 100k but the reference cites MovieLens 1m; no code or API configuration is provided; the t-test is described as unpaired but the design is clearly paired. Also '10 iterations' is small, though the SDs are small enough that this may not matter.\n\nWhat the paper does not do is compare against any traditional recommender metric. Without showing that, e.g., NDCG or Recall would not flag these same perturbations, the conclusion that LLM-based RS need a new evaluation framework is not established. The evidence supports a weaker and still useful claim: these perturbations should be reported when evaluating LLM recommenders.\n\nWho it's for: people evaluating LLM-based recommenders and wanting a checklist of robustness checks. It deserves a serious referee because the combination is novel and the underlying question is real, but acceptance should hinge on fixing the oracle assumption and the circular parameter selection.\n\nRecommendation: send it to peer review, with a strong request for major revision.","headline":"First metamorphic-testing study of an LLM recommender, but an undefended oracle assumption and circular parameter selection undercut the headline conclusion.","tokens_in":9658,"tokens_out":2103,"would_cite":false,"duration_ms":20020,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GPT-3.5 as a recommender is unstable under rating-scale and prompt perturbations, so LLM recommenders need an evaluation framework beyond traditional metrics.","keywords":["metamorphic testing","recommender systems","large language models","ChatGPT","prompt perturbation","ranking similarity","test oracle","MovieLens"],"falsifier":"Take the same users and hold out part of their MovieLens history, then compute whether the MR1-MR4 perturbed top-5 lists match held-out preferences as well as the unperturbed baseline list; if the perturbed lists are equally accurate, the low similarity reflects output variability rather than a failure that traditional metrics miss.","tokens_in":8703,"feed_emoji":"🧪","tokens_out":7919,"duration_ms":76444,"temperature":0.7,"pith_summary":"Metamorphic testing checks a system's outputs against relations between inputs, rather than against a known correct answer. The paper uses this to test GPT-3.5-turbo as a recommender: multiplying or shifting the numeric ratings in the prompt, or inserting spaces and random words, should in principle leave the user's preferences unchanged. In practice, these perturbations significantly changed the top-5 recommendation lists, with Kendall $\\tau$ falling from 0.91 at baseline to between 0.06 and 0.50 depending on the perturbation (all $p < 0.0001$). The authors conclude that traditional offline evaluation metrics for recommender systems are not enough to assess LLM-based recommenders, and that a dedicated evaluation framework is needed.","feed_headline":"Tweaked ratings and prompts scramble ChatGPT's top-5 lists","feed_subtitle":"Metamorphic testing shows GPT-3.5 top-5 recommendations shift sharply, so LLM recommenders need stability checks.","key_machinery":"The central object is the metamorphic relation (MR), a specified transformation of the input prompt that should leave the recommendation output valid if the system is robust. The paper defines four: MR1 multiplies every rating and the rating scale by an integer; MR2 shifts every rating and the scale; MR3 inserts spaces between characters; MR4 inserts random words such as fruit names. Two similarity measures, Kendall $\\tau$ (full-list order agreement) and Rank-Biased Overlap (top-weighted rank overlap), operationalize whether the relation is satisfied, and unpaired $t$-tests at the 95% confidence level identify significant deviations from the unperturbed baseline. The framework also calibrates prompt length ($l=20$) and list size ($k=5$) so that repeated unperturbed runs already agree closely, isolating the effect of the metamorphic perturbations from GPT's intrinsic output randomness.","core_discovery":"The paper's central claim is that the same user preferences, expressed in different but semantically equivalent forms, produce markedly different top-5 recommendations from GPT-3.5, and this instability is invisible to standard accuracy and ranking metrics. The evidence is the drop in list similarity under all four metamorphic relations: rating multiplication (Kendall $\\tau$ 0.4829, RBO 0.8496), rating shifting (0.4966, 0.8460), added spaces (0.0640, 0.4710), and added random words (0.2295, 0.6802), all statistically distinguishable from the baseline ($p < 0.0001$). The authors interpret this as a test-oracle problem: since the correct top-$k$ list is unknown, metamorphic relations are a way to evaluate consistency, and the observed violations signal that LLM-based recommenders cannot be assessed with the same metrics as traditional recommender systems.","pith_inferences":["Inference: The paper counts all divergence from the unperturbed list as a defect, but a perturbed list could still be a good recommendation; measuring each perturbed list's accuracy against held-out preferences would separate benign output variability from genuine failure.","Inference: The MR3 result (spaces) is consistent with tokenization sensitivity rather than preference corruption, since inserting spaces changes the model's input tokens; a testable extension is to check whether MR3 violations shrink with a model that operates on more robust tokenization.","Inference: A natural next experiment is to apply the same metamorphic relations to a deterministic non-LLM recommender on the same ratings as a control; if the control is invariant under rating rescaling, the instability is specific to the LLM, strengthening the paper's conclusion, whereas if the control also changes, the relations themselves may be too strict."],"forward_implications":["A user who expresses the same preference on a 10-point scale instead of a 5-point scale can expect a substantially different top-5 recommendation list, since MR1 and MR2 change roughly half the ranking order.","Trivial surface changes to the prompt—extra spaces or random filler words—can almost completely destroy agreement with the unperturbed list, implying GPT-3.5's recommendations are tightly coupled to tokenization and prompt form.","Accuracy and ranking metrics calculated against known user preferences do not by themselves reveal this input sensitivity; evaluating an LLM recommender requires checking output consistency across semantic-preserving input variants.","Any metamorphic evaluation of a probabilistic LLM recommender must first control the model's output randomness, as the paper does by tuning $l$ and $k$, otherwise random variation swamps the signal of interest."],"supporting_citations":[{"why":"Defines metamorphic testing and the test-oracle problem, the methodology the paper is adapting.","marker":"[9]"},{"why":"Supplies the rating multiplication and shifting metamorphic relations for recommender systems, which become MR1 and MR2.","marker":"[10]"},{"why":"Supplies the linguistic metamorphic relations for LLM testing, which become MR3 and MR4.","marker":"[11]"},{"why":"Provides the MovieLens dataset on which all experiments are run.","marker":"[23]"},{"why":"Defines Rank-Biased Overlap, one of the two similarity metrics used to compare recommendation lists.","marker":"[24]"},{"why":"Establishes the GPT-based recommender setting and the output randomness problem that motivates the randomness-control step.","marker":"[1]"}],"fun_headline_variants":["ChatGPT's top-5 picks wobble under tiny rating tweaks","Metamorphic test: ChatGPT recommenders flunk consistency checks","Same tastes, different rankings: GPT-3.5 recommender instability","Prompt and rating shifts expose shaky GPT-3.5 recommendations","New testing method reveals ChatGPT's recommendation instability"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that a robust LLM recommender should return essentially the same top-5 list when ratings are multiplied or shifted and when spaces or random words are inserted into the prompt, and treats the unperturbed GPT-3.5 list as the correct baseline; if a legitimate recommender could reasonably change its output under these input changes, the similarity drops would not by themselves prove that a new evaluation framework is needed.","fun_headline_variants_meta":{"raw":{"variants":["ChatGPT's top-5 picks wobble under tiny rating tweaks","Metamorphic test: ChatGPT recommenders flunk consistency checks","Same tastes, different rankings: GPT-3.5 recommender instability","Prompt and rating shifts expose shaky GPT-3.5 recommendations","New testing method reveals ChatGPT's recommendation instability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000183,"raw_usage":{"total_tokens":1328,"prompt_tokens":972,"completion_tokens":356,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":270}},"tokens_in":588,"tokens_out":356,"duration_ms":4142,"temperature":1.0,"reasoning_tokens":270,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:53:17.557035+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same users and hold out part of their MovieLens history, then compute whether the MR1-MR4 perturbed top-5 lists match held-out preferences as well as the unperturbed baseline list; if the perturbed lists are equally accurate, the low similarity reflects output variability rather than a failure that traditional metrics miss.","supporting_citations":[{"cited_title":"An empirical study on metamorphic testing for recommender systems,","cited_arxiv_id":null,"evidence_quote":"Supplies the rating multiplication and shifting metamorphic relations for recommender systems, which become MR1 and MR2."},{"cited_title":"METAL: Metamorphic Testing Framework for Analyzing Large-Language Model Qualities","cited_arxiv_id":"2312.06056","evidence_quote":"Supplies the linguistic metamorphic relations for LLM testing, which become MR3 and MR4."},{"cited_title":"[Online]","cited_arxiv_id":null,"evidence_quote":"Provides the MovieLens dataset on which all experiments are run."},{"cited_title":"Is chatgpt a good recom- mender? a preliminary study,","cited_arxiv_id":null,"evidence_quote":"Establishes the GPT-based recommender setting and the output randomness problem that motivates the randomness-control step."}],"review_version":1}