{"id":"40dac7e4-961d-49af-af91-3671b36656c9","arxiv_id":"2505.00981","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A multi-LLM debate framework extracts Schwartz value labels from user interaction histories, and adding these labels through contrastive learning improves recommendation accuracy on PENS and MovieLens-1M.","lead":"This paper reports a system called ZOOM that uses several AI language models working together to guess a person's core values from the news or movies they have interacted with. The guessed values are then added to recommendation algorithms, and the authors report better recommendation accuracy on two public datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The value-mining step is not specified to exclude the held-out interaction; if H_u includes the test item, Table 2's recommendation gains are test-item leakage rather than evidence for value-based recommendation.","rationale":"The reader's weakest-assumption analysis focuses on the 50-user human annotation study: its protocol, number of annotators, inter-annotator agreement, and label definitions are all unreported, so the F1 scores in Table 3 are not fully interpretable. I agree that is a genuine weakness. However, I think the more load-bearing concern is the relationship between the leave-one-out split and value mining. Algorithm 1 takes H_u as raw interaction history without saying whether the held-out test interaction is removed before summarization or debate. If it is not removed, the user value representation is computed from the very item the model is asked to rank, making the 'Original vs DC/CL' comparison in Table 2 a comparison between a model that sees the test item through the value channel and one that does not. That would invalidate the headline empirical claim regardless of how clean the annotation study is. The proposed check is cheap, unambiguous, and decisive: rerun with training-only H_u. If the gains persist, the paper still needs the annotation protocol and significance tests that the reader requested; if the gains do not persist, the core empirical contribution is not supported. I therefore retain a conditional verdict, with the split-leakage check as the primary condition.","tokens_in":20980,"tokens_out":8548,"duration_ms":96388,"concrete_test":"Run the PENS and MovieLens-1M pipelines with two value-extraction inputs: (1) H_u as the full interaction history, and (2) H_u constructed strictly from the pre-split training history, i.e., drop the most recent interaction before summarization, evaluation, and debate. Re-train MoRec and EasyRec with DC and CL under protocol (2), using the same hyperparameters, and recompute HR@10/20 and NDCG@10/20 from Table 2. If the gains over Original collapse or reverse, leakage is confirmed; if they reproduce within variance, the central recommendation claim survives this test.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 6.1.2 defines a leave-one-out split: the most recent interaction is the test item and all earlier interactions form the training set. Yet Section 4.2 and Algorithm 1 define ZOOM's input as H_u, the user's raw interaction history, and never state that H_u excludes the held-out interaction. If value mining is run on the full history, the content of the test item enters the summarization, evaluator, and supervisor steps, so the value vector/embedding used in Section 5 carries information about the item whose rank is being evaluated. The large gains in Table 2 (e.g., HR@10 0.1957 to 0.2749 and NDCG@20 0.0242 to 0.0379 for EasyRec on MovieLens-1M) would then reflect test-label leakage, not the benefit of stable user values. This is a stronger threat than the also-valid underspecified 50-user annotation study: it directly undermines the recommendation experiment rather than only the F1 interpretation. The check below settles it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ZOOM, a zero-shot multi-LLM framework that mines Schwartz personal values from users' raw interaction histories. ZOOM condenses item text by summarization and then uses evaluator and supervisor agents, with a debate mechanism, to generate and refine value predictions. The mined values are integrated into two language-model-based recommender systems, MoRec and EasyRec, either by direct concatenation or by a contrastive-learning auxiliary loss. Experiments on PENS and MovieLens-1M report consistent improvements in HR@K and NDCG@K over the base recommenders, as well as F1 alignment scores between ZOOM's extracted values and human annotations.","tokens_in":21212,"tokens_out":4186,"duration_ms":41891,"significance":"If the results hold, the paper offers a practical and low-cost way to obtain stable user-value signals from textual interaction data, and the contrastive fusion strategy is simple enough to port to other recommender architectures. The framework is described in enough detail to be reproducible in principle, and the choice of two public datasets and two state-of-the-art baselines is appropriate. However, the central claims currently rest on two evaluation gaps: the train/test separation for the value-mining step is not specified, and the human annotation study that grounds the value-extraction accuracy is severely underspecified. These issues must be resolved before the reported gains can be attributed to user values rather than to leakage or to a generic auxiliary-text effect.","major_comments":[{"comment":"The leave-one-out protocol in Section 6.1.2 reserves the most recent interaction as the test item, but the paper never states whether the raw interaction history H_u fed into Algorithm 1 excludes that held-out item. Section 4.2 and Algorithm 1 define ZOOM's input as the user's raw interaction history without any restriction. If H_u includes the test item, the test item's content enters the summarization, evaluator, and supervisor steps, and the value embedding used in Section 5 carries information about the very item whose rank is evaluated. This would make the Table 2 gains, such as EasyRec on MovieLens-1M improving HR@10 from 0.1957 to 0.2749, an artifact of test-label leakage rather than evidence for value-aware recommendation. The authors must explicitly state that value mining uses only the training-prefix interactions, and if that is not the case, the recommendation experiments must be rerun with the held-out item excluded from H_u.","section":"Section 6.1.2 vs. Section 4.2 and Algorithm 1"},{"comment":"The claim that ZOOM's extracted values align with human judgment rests on an F1 evaluation of only 50 randomly selected users per dataset, yet the annotation protocol is not reported. The paper does not say how many annotators labeled each user, what instructions or definitions of the ten Schwartz values were provided, whether annotators saw the same interaction histories that ZOOM saw, how disagreements were resolved, or what the inter-annotator agreement was. Without this information, the F1 scores in Table 3 cannot be interpreted, and the premise that ZOOM mines true user values rather than plausible-sounding labels is unsupported. The authors should provide a complete annotation protocol and agreement statistics, or temper the grounding claim accordingly.","section":"Section 6.4 and Table 3"},{"comment":"The experimental design does not isolate the effect of user-value semantics from the effect of any text-derived auxiliary signal. The contrastive loss aligns the user preference embedding with an embedding computed from the same interaction history used to train the recommender, so the gains in Table 2 could come from a regularization or self-supervision effect rather than from the specific content of Schwartz values. A control condition is needed, for example using randomly assigned value labels, generic text summaries unrelated to values, or value embeddings derived from a random subset of items, to show that the improvements are specifically attributable to the mined user values. As written, the causal claim 'incorporating user values ... consistently improves recommendation performance' is not established.","section":"Section 5, Eq. (7), and Table 2"},{"comment":"All recommendation results in Table 2 are single point estimates with no error bars, multiple seeds, or significance tests. Several improvements are very small, such as MoRec on MovieLens-1M HR@20 (0.3792 to 0.3812, a 0.53% gain) and NDCG@20 (0.0455 to 0.0460, a 1.10% gain). Given the authors' claim that the improvements are consistent, the paper should report at least the variance across multiple random seeds or a standard significance test; otherwise, the claim of consistency across all scenarios is not statistically supported.","section":"Section 6.3 and Table 2"}],"minor_comments":[{"comment":"The description of the direct concatenation method is internally inconsistent: Section 5 and the introduction say the user value vector is 'appended' or 'concatenated' to the user preference vector, while Section 6.2.3 says it is 'simply added' to the user preference embedding. Please clarify whether DC performs vector concatenation or vector addition, since this changes the implementation.","section":"Section 5 and Section 6.2.3"},{"comment":"The text describing the 'ZOOM w/o Sum' ablation says the F1 score drops to 0.36 on MovieLens-1M, but Table 3 reports 0.3200. The reported number in the text should be corrected to match the table.","section":"Section 6.5 and Table 3"},{"comment":"The sentence reporting improvements for the worst DC scenarios mixes HR@10 and NDCG@20 numbers in a way that is hard to parse: it claims 'improvements of 5.22% and 6.25% in HR@10, and 1.10% and 1.15% in NDCG@20' over Original and DC, but the mapping between the two scenarios and the four percentages is unclear. Please rephrase for clarity.","section":"Section 6.3"},{"comment":"The notation for the contrastive loss is imprecise: the set V, the embedding e_v^i, and the negative-set V^- are not formally defined before Eq. (7), and it is unclear whether the 'N embeddings with the lowest cosine similarity' are recomputed at every training step or fixed. Please define these objects and state the sampling frequency.","section":"Equation (7)"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the unresolved train/test separation in the value-mining pipeline. If the held-out interaction was included in H_u during value extraction, the headline recommendation results would be invalid rather than merely noisy. I would ask the authors to clarify and, if necessary, rerun the experiments with H_u restricted to the training prefix before considering the paper for publication. The underspecified human evaluation and the absence of control conditions are secondary but also need to be addressed in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know first: this paper proposes ZOOM, a multi-LLM debate framework that extracts Schwartz values from raw interaction histories, then integrates them into recommenders via contrastive learning. The pipeline is genuinely new as a combination: text summarization to fit long histories, multiple evaluators with different decoding strategies, and paired supervisors who debate to consensus. The ablations show each component matters, and the case study is illustrative. Credit where it is due: the framing of stable user values as a recommendation signal is sensible, and the contrastive integration strategy is cleaner than naive concatenation.\n\nThe soft spots are real, and one is load-bearing. The human F1 study (Table 3) uses 50 users per dataset with no annotation protocol: no number of annotators, no instructions, no inter-annotator agreement. That alone is a serious gap, but it only undermines the value-extraction claim. More damaging is what the stress-test note caught: Section 6.1.2 defines the split as the most recent interaction held out, but Section 4.2 and Algorithm 1 define ZOOM's input H_u as the raw interaction history, with no statement that the held-out test item is excluded. If H_u includes the test item, the value vector encodes the very item whose rank is evaluated, and Table 2's gains (e.g., EasyRec on MovieLens-1M, HR@10 from 0.1957 to 0.2749) are leakage, not evidence for value-based recommendation. The paper never closes this loophole, so the central recommendation experiment is currently ambiguous. This is not a minor quibble; it is a request the authors must answer before the main claim is credible.\n\nOther soft spots are milder. There are no error bars or significance tests anywhere, which is typical for this kind of paper but still matters given the modest absolute gains on PENS. And there is no control condition showing that a generic text-derived auxiliary signal (e.g., latent interests from the same summaries) would not produce similar gains. So the specific contribution of \"values\" over \"more text features\" is not isolated.\n\nWho is this for? Researchers working on LLM-based recommendation, value alignment, and personalization with auxiliary semantic signals. It deserves a serious referee rather than a desk reject, because the core idea is worth testing and the methodology, once clarified and tightened, could be a useful building block. But as it stands, the paper needs a major revision: spell out the exact data split for value mining, add a leakage-free rerun, document the annotation protocol, and add a non-value text-signal control.\n\nRecommendation: send to peer review, but condition acceptance on the leakage question being resolved explicitly, not implicitly.","headline":"A clever LLM-agent pipeline for mining user values, but the main recommendation result is fragile until the authors rule out test-item leakage and shore up the human evaluation.","tokens_in":21704,"tokens_out":1868,"would_cite":false,"duration_ms":19855,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that stable user values mined by a zero-shot multi-LLM debate improve recommender accuracy, with contrastive fusion beating direct concatenation in every tested setting.","keywords":["user values mining","recommender systems","large language models","multi-agent debate","contrastive learning","text summarization","zero-shot extraction","ten universal basic values"],"falsifier":"Re-run the human evaluation with at least three independent annotators per user, a pre-registered coding manual for the ten value categories, and reported inter-annotator agreement: if agreement is low or ZOOM's F1 against the independent labels falls toward the baseline LLM's roughly 0.12, then the claim that ZOOM mines true user values is unsupported; a complementary control would replace the mined values with randomly assigned values in the same contrastive fusion and check whether the ranking gains persist, which would show the gains come from regularization rather than value content.","tokens_in":20805,"feed_emoji":"🎯","tokens_out":7305,"duration_ms":72940,"temperature":0.7,"pith_summary":"The paper sets out to show that a user's stable values, not just their transient clicks, can be recovered automatically from interaction history and put to work in recommender systems. It anchors values in the ten universal categories of a cross-cultural value taxonomy and extracts them with ZOOM, a zero-shot collaboration of LLM agents: evaluators summarize item content and generate candidate values from different orderings and decoding strategies, while supervisors review and debate the candidates until a consensus emerges. The paper claims this extraction aligns with human judgment far better than a naive LLM baseline, with F1 of 0.6813 on PENS and 0.7282 on MovieLens-1M, both more than six times the baseline, and that adding the mined values to two language-model recommenders improves Hit Rate and NDCG across all reported settings, with the contrastive-learning fusion outperforming simple concatenation. A sympathetic reader would care because it offers a path from noisy, short-term behavior signals to a stable representation of what a user actually values, which could make recommendations steadier and more explainable.","feed_headline":"Agent-debated user values lift recommendation accuracy up to 56%","feed_subtitle":"Zero-shot LLM teams mine ten basic human values from click histories and fuse them into recommenders via contrastive learning.","key_machinery":"The machinery is ZOOM's three-stage pipeline plus a contrastive fusion loss. First, an evaluator LLM summarizes each item's content so that more interactions fit into a limited context window, then generates candidate value sets from original-order and shuffled-order histories using four decoding strategies: beam search sampling, temperature sampling, top-k sampling, and nucleus sampling. The output space is a fixed set of ten universal basic values: achievement, power, hedonism, stimulation, self-direction, security, conformity, tradition, universalism, and benevolence. Second, supervisor LLMs built on different backbones, Llama-3-8B and Gemma-2-9B, review each evaluator's candidates and debate for up to three rounds, with a final supervisor resolving any non-consensus; the debate is designed to filter hallucinations and enforce consistency. Third, the extracted value labels are embedded and fused into the recommender either by direct concatenation or through a contrastive auxiliary loss that pulls the user preference embedding toward the user's own value embedding and away from the most similar unrelated users' value embeddings, with the loss weight and negative-sample count tuned per dataset.","core_discovery":"The central claim is that user values, understood as stable long-term motivational goals rather than ephemeral interests, can be mined from historical item interactions by a zero-shot multi-agent LLM framework. The paper's evidence has two parts. First, ZOOM's extracted value sets match human annotations with F1 of 0.6813 on PENS and 0.7282 on MovieLens-1M, more than six times the 0.1160 and 0.1187 achieved by a base LLM, and ablations show that both text summarization and supervisor debate are required for that accuracy. Second, incorporating these values into the MoRec and EasyRec recommenders improves recommendation performance: direct concatenation helps in most settings, while the contrastive-learning fusion consistently beats both the original models and concatenation, with the largest gains on EasyRec with MovieLens-1M, up to 40.47 percent in HR@10 and 56.61 percent in NDCG@20. The paper interprets the larger gains for EasyRec as evidence that value embeddings align more naturally with text-based user representations than with sequence-based ones.","pith_inferences":["A natural extension the paper does not test is whether the mined value vectors are stable over time: if values are genuinely stable, the same user's extracted values from two different time windows should match, which would validate the core premise independently of recommendation metrics.","The same evaluator-supervisor debate pipeline could be repurposed for other stable user attributes, such as personality traits or ethical preferences, whenever a fixed taxonomy exists.","If value embeddings carry the gains, a smaller distilled encoder might be trained to map interaction text directly to value embeddings, removing the cost of invoking several large LLMs per user at deployment.","The contrastive loss effectively regularizes user representations toward a psychologically meaningful semantic anchor, which suggests recommendations could become more robust to accidental or adversarial clicks, though the paper only implies this via its clickbait example.",""],"forward_implications":["If the central claim holds, any text-grounded recommender can be augmented with a value module without retraining the agent pipeline, because ZOOM runs zero-shot before recommendation and only supplies side information.","Contrastive fusion is the safer integration choice: it improved results in every reported configuration, while direct concatenation occasionally dipped below baseline, for example HR@10 for EasyRec on PENS.","The larger gains on MovieLens-1M with EasyRec suggest value integration pays off most when the value distribution is diverse and user representations are already text-based.","The ablation results imply both summarization and debate are load-bearing: removing summarization drops F1 from 0.68 to 0.21 on PENS and from 0.72 to 0.32 on MovieLens-1M, and removing debate cuts F1 to less than half.","Deployment needs tuning of the negative-sample count to value diversity: skewed value distributions need fewer negatives, while diverse distributions need more.",""],"supporting_citations":[{"why":"Supplies the ten universal value categories that ZOOM uses as its output space for extracted user values.","marker":"[41]"},{"why":"Provides the PENS news dataset used to evaluate user value mining and recommendation performance.","marker":"[1]"},{"why":"Provides the MovieLens-1M dataset used as the second evaluation benchmark.","marker":"[13]"},{"why":"Defines the MoRec modality-based recommender whose performance is measured with and without value integration.","marker":"[61]"},{"why":"Defines the EasyRec text-collaborative recommender used to test value integration and shows the largest gains.","marker":"[36]"},{"why":"Supplies the Llama-3-8B backbone used for evaluator and supervisor agents in ZOOM.","marker":"[10]"},{"why":"Supplies the Gemma-2-9B backbone used alongside Llama to diversify the supervisor debate.","marker":"[45]"},{"why":"Provides the self-consistency principle used to justify multiple decoding strategies and randomized interaction order.","marker":"[50]"},{"why":"Supports the claim that multiple LLMs can converge to consensus through debate, the basis of the supervisor mechanism.","marker":"[9]"}],"fun_headline_variants":["Zero-shot LLM debate extracts hidden user values for better recs","AI agents argue out your deep values, boosting recommendations","Beyond clicks: multi-agent LLMs infer lasting values to fix rec engines","User values, not just views: LLM agent teams lift rec quality up to 56%","Debate-club LLMs mine stable values, slashing recommendation noise"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a user's true values can be read off their click or rating history and that the human annotations used to score the extraction are reliable; the paper reports annotation on only 50 randomly selected users per dataset without stating how many annotators labelled each user, what instructions mapped interactions to the ten value categories, or how disagreements were resolved, so noisy ground truth would undermine the F1 claims and the whole value-mining premise.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot LLM debate extracts hidden user values for better recs","AI agents argue out your deep values, boosting recommendations","Beyond clicks: multi-agent LLMs infer lasting values to fix rec engines","User values, not just views: LLM agent teams lift rec quality up to 56%","Debate-club LLMs mine stable values, slashing recommendation noise"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000233,"raw_usage":{"total_tokens":1538,"prompt_tokens":1032,"completion_tokens":506,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":424}},"tokens_in":648,"tokens_out":506,"duration_ms":5171,"temperature":1.0,"reasoning_tokens":424,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:29:39.581309+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the human evaluation with at least three independent annotators per user, a pre-registered coding manual for the ten value categories, and reported inter-annotator agreement: if agreement is low or ZOOM's F1 against the independent labels falls toward the baseline LLM's roughly 0.12, then the claim that ZOOM mines true user values is unsupported; a complementary control would replace the mined values with randomly assigned values in the same contrastive fusion and check whether the ranking gains persist, which would show the gains come from regularization rather than value content.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the PENS news dataset used to evaluate user value mining and recommendation performance."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the MoRec modality-based recommender whose performance is measured with and without value integration."}],"review_version":1}