{"id":"82ca91f8-0813-495b-b003-a942ab76c4af","arxiv_id":"2509.10245","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Deleting each user or item from training data and retraining the model shows which observations help or hurt a recommender's overall performance, a straightforward application of leave-one-out influence analysis.","lead":"The paper measures how much each user and each product changes a recommender's overall accuracy when that user or product is removed from training, a technique called deletion diagnostics. It applies this idea to two common recommender models on movie and product review data, reporting which users and items most affect performance.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1/2 delete the target user/item before the train/test split, so eval(-i) is measured on a different test population than eval; the influence scores conflate training removal with evaluation-population shift.","rationale":"The paper's central claim is that deletion diagnostics quantify how a data point influences the recommender. For that to be true, Eq. (9) must isolate the effect of removing the point from training. The pseudocode in Algorithms 1 and 2 removes the point before the train/test split, which means eval(-i) is computed over a different set of test users/items. This is a standard confounding in deletion studies, and it is visible in the manuscript itself (Section 3.2 and Section 4.1). The reader's conditional verdict focuses on training stochasticity (10 epochs, no seeds). That concern is real, but it presupposes the quantity being measured is the right one. If the evaluation population shifts, the influence scores are not what the paper claims, even in expectation. My proposed test uses the reported global model to estimate the compositional component directly; a high correlation would demonstrate that the rankings in Figures 2, 4, 6, and 7 can be produced without any retraining effect. I do not move the verdict to reject because the confound is testable and fixable: the authors could adopt a fixed-test protocol (with a cold-start handling for deleted users/items) and re-run the experiments. Until that is done, the empirical results should be treated as conditional.","tokens_in":14656,"tokens_out":10781,"duration_ms":102643,"concrete_test":"From the global NCF model's test predictions on MovieLens 100K, compute the per-user AP and the population-shift component delta_pop(u) = MAP_all - MAP_{all\\without u}, where MAP_{all\\without u} is the global model's MAP after removing user u from the test set (no retraining). Compute the Spearman correlation between delta_pop(u) and the influence scores reported in Figure 2/Algorithm 1. If rho > 0.7, the reported influence ranking is largely an evaluation-population artifact, confirming the concern. If rho is low, the ranking may reflect true retraining effects, but the authors should still re-run Algorithm 1 with a fixed test set (remove u only from the training split, evaluate on the original test set) to obtain unconfounded influence values and verify that the top/bottom user and item lists survive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantity Eq. (9), Influence(-i) = eval - eval(-i), is only interpretable as the effect of removing i from training if eval and eval(-i) are computed on the same test set. Algorithm 1 ('Generate the data without participant u and split train/test; X(-u)=X\\{u}') and Algorithm 2 remove the user/item from the data before the split, so the retrained model is evaluated on a test set that excludes that user (or item). The difference therefore contains a purely compositional component: even with no retraining effect, eval - eval(-u) equals (1/(|U|-1))(AP_u - MAP_all) when eval is averaged over users. Users with above-average AP will look positively influential and below-average users negatively influential simply because they are absent from the evaluation population. The same issue affects item deletion, where the relevance sets of all users change when item i is removed from test. The group-deletion 'validation' in Tables 5, 6, 8, and 9 inherits the confound: removing the 10 'least influential' users also removes their test instances, so the observed improvement may reflect the removal of low-AP users from the denominator, not a better-trained model. This is more load-bearing than training stochasticity: it undermines the definition of the measured quantity, not only its precision.","agreement_with_reader":"partial"},"referee_report":null,"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know two things about arXiv:2509.10245. First, it is a clean, readable application of leave-one-out deletion diagnostics to recommender systems: retrain without a user or item and compare MAP. Second, the implementation as written has a confound that undercuts the headline quantity. Algorithms 1 and 2 remove the target user or item before the train/test split, so eval(-i) is measured on a test population that excludes i. The difference in Eq. (9) therefore mixes the true training effect with a compositional shift: a user with above-average AP will look positively influential, and one with below-average AP negatively influential, even if the model never changes. The same applies to item deletion, where relevance sets in the test split change. The group-deletion tables (5, 6, 8, 9) inherit this problem, so the \"validation\" is partly tautological.\n\nWhat is genuinely good: the paper is honest about its lineage (it cites Koh and Liang), states its computational cost clearly, and applies the idea to two models and two datasets. The writing is straightforward. The citation pattern is unobjectionable.\n\nThe soft spots go beyond the confound. There are no error bars, no seed variations, no significance tests; NCF gets only 10 epochs; and there is no comparison to influence functions or other attribution baselines. The word \"explainability\" overstates what is really a debugging metric.\n\nWho this is for: someone looking for a template for leave-one-out analysis could mine the algorithms, but as it stands the numbers do not support the conclusions. The fix is straightforward: split the original data first, then delete the user/item from training only while keeping the test set fixed. Without that, the paper is not ready for serious consideration.\n\nMy recommendation: I would not send this to peer review in its current form, because the central quantity is not the one claimed. After fixing the split and adding variance estimates, it could become a solid application note. But the novelty is low, so I would want to see the corrected results before investing referee time.","headline":"Deletion diagnostics applied to recommenders, but the implementation deletes the target user/item before the train/test split, so the reported influence scores conflate training removal with evaluation-population shift.","tokens_in":15398,"tokens_out":3833,"would_cite":false,"duration_ms":34879,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Deletion diagnostics reveal which users and items move a recommender's performance.","keywords":["recommender systems","explainability","deletion diagnostics","influence estimation","neural collaborative filtering","singular value decomposition","post-hoc explanation","data curation"],"falsifier":"Train the same recommender several times from different random seeds with and without a fixed user, and compare the spread of influence scores across seeds to the spread across users; if the seed-to-seed variation is comparable to or larger than the user-to-user variation, the deletion diagnostic is not isolating the effect of that user.","tokens_in":14447,"feed_emoji":"🎯","tokens_out":6056,"duration_ms":427579,"temperature":0.7,"pith_summary":"The paper claims that a recommender's reliance on any single user or item can be read directly from the change in evaluation metrics when that observation is removed and the model retrained. The influence score $\\mathrm{Influence}(-i)=\\mathrm{eval}-\\mathrm{eval}^{(-i)}$ is positive when deleting the observation hurts and negative when deleting it helps. Applied to a deep neural recommender and to a classical matrix-factorization recommender on two public benchmarks, the method identifies high-leverage users and items and shows that removing the least influential ones can improve ranking quality, with one reported gain of about 18 percent in MAP@K. A sympathetic reading is that this turns black-box systems into auditable objects without needing any internal access.","feed_headline":"Deleting 10 least-influential users lifts ranking quality 18 percent.","feed_subtitle":"Retraining without each user or item reveals who helps and who hurts, and pruning the hurtful ones improves the model.","key_machinery":"The carrying mechanism is the deletion diagnostic, formalized in Algorithms 1 and 2: iterate over users (or items), generate the dataset without that observation, retrain the model, evaluate with MAP, and store the difference from the original model's MAP. The difference is computed as $\\mathrm{Influence}(-i)=\\mathrm{eval}-\\mathrm{eval}^{(-i)}$. This is a leave-one-out perturbation analysis adapted to recommendation metrics; its explanatory power comes from full retraining, which captures global effects that gradient-based or surrogate approximations can miss, at the cost of one retraining per observation.","core_discovery":"The central claim is that influence in recommender systems is a measurable, model-agnostic quantity obtained by full retraining. For each user or item $i$, train the same architecture on the data with $i$ omitted, recompute a ranking metric, and record $\\mathrm{eval}-\\mathrm{eval}^{(-i)}$. Observations whose deletion lowers performance are positively influential; observations whose deletion raises performance are negatively influential. Because the procedure only requires a performance metric and the ability to retrain, it applies to any recommender, and the experiments demonstrate it on both a neural architecture and SVD. The reported results also show an asymmetry: removing the ten least influential users from the MovieLens NCF model improved MAP@K by 18.49% and Precision@K by 16.75%, suggesting that some training instances contribute noise rather than signal.","pith_inferences":["The reported influence scores are single runs with no variance or seed information, so a natural next test is whether the ranking of influential users is stable across random seeds; without that stability, part of the signal may be training noise.","The asymmetry between removing positive and negative influencers suggests that many recommenders are trained on a long tail of near-redundant interactions, and pruning that tail may offer larger gains than protecting the top.","A testable extension is to compare the deletion-based ranking with influence estimates from cheaper approximations, asking whether exact retraining changes the practical decisions a data curator would make."],"forward_implications":["The recipe for influence is metric-agnostic and architecture-agnostic, so the same procedure can audit any recommender that can be retrained.","Removing or down-weighting consistently negatively influential users and items is a data-curation strategy that can improve MAP, NDCG, and precision.","The method gives developers a debugging signal: the most positively influential observations are the ones the model depends on most, and their characteristics can be inspected.","Because it needs no gradients or internal parameters, the method works on proprietary or otherwise opaque recommenders.","Retraining cost can be reduced by parallelization, subsampling, or focusing on top-K candidates, making the approach feasible at larger scale."],"supporting_citations":[{"why":"Supplies the Neural Collaborative Filtering architecture whose black-box behavior the deletion diagnostics are used to explain.","marker":"[23]"},{"why":"Provides the MovieLens 100K dataset used for the main NCF and SVD experiments.","marker":"[34]"},{"why":"Provides the Amazon Reviews dataset used for the sparse-data NCF experiment.","marker":"[39]"},{"why":"Establishes SVD as the classical matrix-factorization baseline used to demonstrate model-agnosticism.","marker":"[29]"},{"why":"Represents gradient-based influence estimation whose retraining-cost contrast motivates full deletion diagnostics.","marker":"[22]"},{"why":"Supplies the evaluation metrics, including MAP and NDCG, that define the influence measure.","marker":"[32, 33]"}],"fun_headline_variants":["Delete a user, retrain, and watch the recommender change","Who helps your recommender? Retrain without each user to find out","Pruning least-influential users lifts ranking quality 18%","Model-agnostic deletion diagnostics for transparent recommenders","Measure any recommender's user impact by retraining without them"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the difference in evaluation metrics after one retraining is a stable property of the deleted observation, rather than a product of random initialization and training noise; the NCF runs use 10 epochs and no reported seeds, which is exactly the regime where that assumption can fail.","fun_headline_variants_meta":{"raw":{"variants":["Delete a user, retrain, and watch the recommender change","Who helps your recommender? Retrain without each user to find out","Pruning least-influential users lifts ranking quality 18%","Model-agnostic deletion diagnostics for transparent recommenders","Measure any recommender's user impact by retraining without them"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000595,"raw_usage":{"total_tokens":2748,"prompt_tokens":869,"completion_tokens":1879,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":1791}},"tokens_in":485,"tokens_out":1879,"duration_ms":12593,"temperature":1.0,"reasoning_tokens":1791,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:55:31.948601+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same recommender several times from different random seeds with and without a fixed user, and compare the spread of influence scores across seeds to the spread across users; if the seed-to-seed variation is comparable to or larger than the user-to-user variation, the deletion diagnostic is not isolating the effect of that user.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Neural Collaborative Filtering architecture whose black-box behavior the deletion diagnostics are used to explain."},{"cited_title":"Koren, R","cited_arxiv_id":null,"evidence_quote":"Establishes SVD as the classical matrix-factorization baseline used to demonstrate model-agnosticism."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents gradient-based influence estimation whose retraining-cost contrast motivates full deletion diagnostics."}],"review_version":2}