{"id":"72e074fa-6767-43fd-9f4c-f7b070de4a60","arxiv_id":"2608.11604","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Shopping agents improved by training on purchase outcomes plus LLM-mined conversational feedback via GRPO and on-policy distillation.","lead":"This paper introduces LOFA, a framework that trains shopping agents on real user chat logs by combining purchase-based reinforcement learning with feedback extracted from follow-up user messages. It reports consistent improvements in recommendation ranking and response quality on real JD.com shopping data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated DeepSeek-V3.2 judge used for both feedback mining and Success Rate evaluation; claimed helpfulness/alignment gains may be judge-specific artifacts.","rationale":"The reader's weakest assumption correctly identifies the deepest problem. The paper's strongest evidence for the 'response helpfulness' and 'user-satisfaction alignment' parts of the central claim is the SR column, and SR is computed by the same unvalidated LLM that generates the supervision. This is a validity threat, not just a reproducibility concern: if DeepSeek-V3.2 systematically prefers outputs that resemble its own feedback-conditioned revisions, then OPD-trained models will trivially score higher. The absence of human validation (or even a second judge) means the reported gains could be entirely an artifact of the judge's inductive bias. Recommendation-quality metrics (NDCG/Recall/MAP) are objective and the ablation/backbone experiments support the method's internal consistency, but they do not rescue the subjective half of the claim. The Table 4 inconsistency about pipeline order is a separate weakness that further erodes confidence in the reported SR numbers, as does the lack of error bars. These issues are addressable: a human-evaluation study on a sample of test turns would settle whether the concern lands. If human-judged SR shows LOFA still beats baselines, the conditional accept can be upgraded; if not, the central claim needs to be weakened. For now, CONDITIONAL remains the right verdict, so I agree with the reader and recommend no change.","tokens_in":14669,"tokens_out":5168,"duration_ms":51319,"concrete_test":"Extract the 812 JD-conv test instances used in Table 3. For a random subset of 200 instances, have three human annotators independently judge whether the revised response resolves the issue revealed by the user's feedback (same rubric as the SR prompt). Compute Cohen's kappa between DeepSeek-V3.2 and the majority human label, and compute human-judged SR for each method (Base, +OPD, +RL, RL→OPD, OPD→RL). If the human-judged ordering differs from Table 3/Table 4, or if agreement is below ~0.6 kappa, the LLM-judged SR is not a valid measure of user satisfaction. As a cheaper secondary check, recompute SR on all 812 instances using a different LLM judge (e.g., GPT-4o or Claude) and compare the ranking of methods.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LOFA improves response helpfulness and user-satisfaction alignment rests entirely on the Success Rate (SR) metric computed by DeepSeek-V3.2 in Section 4.1.3. The same model is used in Section 3.3.1 to mine the directive-feedback categories and explanations that construct the teacher's privileged context. This creates a circular evaluation: the student is trained to match a teacher conditioned on DeepSeek-generated labels, then judged by DeepSeek on whether the issue is resolved. Systematic leniency or stylistic preference in the judge could produce large SR gains that do not correspond to genuine user satisfaction. No human agreement study, alternative judge, or error bars are reported; Section 4.1.4 states \"All results are averaged over multiple runs\" but no variance is given, so differences like the 0.6022 vs 0.5924 overall SR could be noise. Additionally, the paper's claim that RL→OPD \"consistently outperforms\" OPD→RL is contradicted by Table 4, where OPD→RL beats RL→OPD on Explicit Criticism (0.5846 vs 0.5692) and Comparative Preference (0.5294 vs 0.4706), further suggesting the reported SR differences are not stable. A human-annotated sample of test instances is needed to establish that the LLM judge's ratings reflect real resolution of user issues.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LOFA, a framework for learning a shopping agent from real-world e-commerce interaction logs without human annotation. It combines two supervision sources: (i) explicit behavioral feedback in the form of purchase outcomes, optimized with GRPO under format and NDCG-based ranking rewards; and (ii) in-dialogue directive feedback, where an LLM (DeepSeek-V3.2) annotates each user follow-up into one of four actionable categories (explicit criticism, implicit deduction, pure negative feedback, comparative preference) plus topic shift/abandonment, together with an explanation. The annotated feedback is used to construct a teacher that sees privileged future feedback and distills corrections into the student via on-policy reverse-KL distillation. Experiments on JD.com 'Jingyan' logs compare Qwen3-8B variants (and Qwen3-4B) on ranking metrics and an LLM-judged Success Rate. The main findings are that RL improves ranking metrics, OPD improves Success Rate, and the combined RL→OPD pipeline gives the best overall numbers, with ablations showing the contribution of each reward and feedback category.","tokens_in":14930,"tokens_out":4831,"duration_ms":44531,"significance":"If the results hold, LOFA offers a practical route to exploit abundant online interaction logs for shopping-agent improvement without human annotation, treating purchase outcomes as verifiable rewards and conversational corrections as dense token-level supervision. The framework is novel in combining GRPO on real purchase data with feedback-aware on-policy distillation, and the paper clearly targets an industrial deployment scenario. Strengths include the use of real purchase outcomes as objective ranking targets, the category-level ablations showing that each directive-feedback type contributes, and the evaluation across two backbone scales. However, the reported helpfulness and alignment gains rest entirely on an LLM judge that is also used to create the training labels, and the paper lacks statistical grounding; the central claims therefore need additional validation before the conclusions can be accepted.","major_comments":[{"comment":"The paper states that 'RL→OPD consistently outperforms OPD→RL' but Table 4 contradicts this on two of the four categories: on Explicit Criticism, OPD→RL achieves 0.5846 vs RL→OPD's 0.5692, and on Comparative Preference, OPD→RL achieves 0.5294 vs RL→OPD's 0.4706. The overall Success Rate margin (0.6022 vs 0.5924) is also small. This contradicts the 'consistently' claim and weakens the conclusion that outcome-level learning should precede directive learning. Please rephrase the claim, report per-category significance tests, or provide an analysis that reconciles the ordering.","section":"Section 4.2, observation (3); Table 4"},{"comment":"The Success Rate metric is computed by prompting DeepSeek-V3.2 (Section 4.1.3), and the same model is used in Section 3.3.1 to mine the feedback categories and explanations that build the teacher. Because the student is trained to imitate a teacher constructed from DeepSeek-labeled directives and is then evaluated by DeepSeek, the reported SR gains may reflect the judge's preferences rather than genuine user satisfaction. The paper reports no human agreement study, no alternative judge, and no error analysis for the judge. Please add a human-annotated sample of test instances scored by independent raters with agreement statistics, and/or report SR using a second independent evaluator; without this, the helpfulness and alignment claims are not verifiable.","section":"Sections 4.1.3, 3.3.1, 4.1.4"},{"comment":"The text says 'All results are averaged over multiple runs' but no variance, confidence intervals, or significance tests are reported anywhere. Most headline differences are small, for example overall SR 0.6022 vs 0.5924, NDCG@10 0.6512 vs 0.6386, and Recall@10 0.8911 vs 0.8595. Without measures of uncertainty, 'consistent improvements' cannot be distinguished from run-to-run noise. Please report means with standard deviations across runs and paired significance tests (e.g., bootstrap or permutation) for the main comparisons.","section":"Section 4.1.4; Tables 3-6"},{"comment":"The Comparative Preference test set contains only 17 instances (Table 2). Consequently, the CP rows in Table 4 (e.g., 0.4706 vs 0.5294) represent differences of one or two responses and cannot support any reliable conclusion, including the claim that gains on Comparative Preference are 'slightly smaller.' Please either enlarge the CP test set or explicitly state that CP results are anecdotal and exclude CP from aggregate conclusions.","section":"Table 2; Sections 4.2-4.3"}],"minor_comments":[{"comment":"Figure 1 contains the typo 'snecific reason' and Table 7 case 2 contains 'understant'; both should be corrected.","section":"Figure 1; Table 7"},{"comment":"The observations are enumerated (1) through (4) and then jump to (6), skipping (5); renumber the list.","section":"Section 4.2"},{"comment":"The column headers read 'MAP@1, 10, 20' but the tables list MAP@1, MAP@5, and MAP@10; align the headers with the actual columns.","section":"Tables 3 and 5"},{"comment":"References [24] and [25] are identical duplicates of the same Christiano et al. paper; remove one of them.","section":"References"},{"comment":"The baseline 'Qwen3-8B-reflect' appearing in Table 3 is not defined in the compared-methods list, which only mentions a 'Self-Reflection' baseline; add a description or rename it for consistency.","section":"Section 4.1.2 and Table 3"},{"comment":"The on-policy distillation objective is presented without an equation number, making it awkward to reference in later sections; consider numbering it.","section":"Section 3.3.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the journal's scope well, addressing agent-based recommendation and LLM alignment with an industrially grounded problem. The main risks are the unvalidated LLM judge used for both training-label mining and evaluation, and the overclaimed 'consistently outperforms' ordering that contradicts the paper's own Table 4. Neither issue seems unfixable in scope: the former requires a human-annotated evaluation sample or an alternative judge, and the latter requires rephrasing plus significance testing. The lack of released code or data is a reproducibility concern that could be mentioned to the authors, though industrial data restrictions are understandable. I would not reject, but the central claims should not be accepted without the requested validation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper: it trains shopping agents directly on real JD interaction logs without human annotation, by pairing purchase-outcome GRPO with on-policy distillation from a teacher that sees the user's next-turn feedback. That combination is new, and the ranking half of the results looks solid. The dialogue-helpfulness half is shakier, because the only metric for it is a Success Rate judged by the same LLM that creates the training signal.\n\nWhat's genuinely good: the data are real online logs, not synthetic. The purchase-outcome reward is verifiable, and the ranking gains on NDCG, Recall, and MAP are large and consistent across Qwen3-4B and Qwen3-8B. The feedback taxonomy (explicit criticism, implicit deduction, pure negative, comparative preference) is reasonable, and the ablations show each category contributes. The privileged-teacher construction is a sensible way to convert sparse user complaints into dense token-level supervision. The 'data flywheel' framing is practically valuable for deployment.\n\nThe soft spots are proportionate. The main evaluation of helpfulness and satisfaction alignment is the Success Rate computed by DeepSeek-V3.2, the same model used for feedback mining and for writing the teacher's explanations. That is a mild circularity, and there is no human agreement study or second judge to check it. The Comparative Preference test set has 17 instances, so a one-sample shift flips the ranking; yet the paper claims RL-to-OPD 'consistently outperforms' OPD-to-RL while Table 4 shows OPD-to-RL winning on Explicit Criticism and Comparative Preference. No confidence intervals or significance tests appear anywhere. Data and code are not released. These issues are fixable and they do not undermine the ranking results, which depend only on purchase outcomes.\n\nI agree with the stress-test note: the declared helpfulness/alignment gains rest entirely on an unvalidated judge, and the 'consistently' claim overreaches. Still, the core framework is plausible and the empirical story on ranking is credible. This paper is for people who build deployed shopping agents and for researchers working on learning from interaction logs. It deserves a serious referee, but I'd send it back with a request for human-annotated SR on a sample, error bars, and a softened ordering claim. If the authors can supply those, it becomes a practical contribution.","headline":"Real-log learning loop with a plausible design, but the LLM-judge circularity and a missing variance estimate leave the headline dialogue gains unproven.","tokens_in":15442,"tokens_out":2538,"would_cite":true,"duration_ms":27898,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A shopping agent can be trained automatically from real user logs, using purchases as outcome rewards and conversational corrections as token-level teaching signals.","keywords":["shopping agents","conversational recommendation","online user feedback","reinforcement learning","GRPO","policy distillation","LLM feedback mining","e-commerce interaction logs"],"falsifier":"Take a random sample of the directive-feedback test instances and have human annotators independently label the feedback category and judge whether each revised response resolves the user's complaint; recompute Success Rate with human labels. If LOFA's advantage over baselines shrinks to near zero or reverses under human judgment, the reported gains are artifacts of the automated judge rather than real user-satisfaction improvements.","tokens_in":14454,"feed_emoji":"🛒","tokens_out":6858,"duration_ms":68832,"temperature":0.7,"pith_summary":"LOFA tries to show that deployed shopping agents can be improved directly from their own real interaction logs, with no human annotation. The central step is to treat two kinds of user feedback as complementary supervision: whether a conversation ended in a purchase, and what the user said in later turns when the agent's answer missed. Purchase outcomes are turned into a ranking reward, while conversational corrections are mined by a large language model into feedback categories and explanations that construct a privileged teacher. The teacher then guides the student's own generated replies through token-level distillation. If the claims hold, shopping agents could continuously retrain from the complaints and purchases recorded in their daily logs instead of relying on static offline datasets.","feed_headline":"User complaints become training data for shopping agents","feed_subtitle":"Mining conversational corrections and purchase outcomes together lifts recommendation ranking and response helpfulness.","key_machinery":"The mechanism is a two-stage training loop. First, behavioral feedback learning turns sessions that ended in a purchase into reinforcement-learning contexts, where the agent samples recommendations and receives a format reward for valid output plus a ranking reward that favors putting the purchased item near the top; the update uses GRPO, a policy-gradient method that scores a sampled group of responses against the verifiable purchase signal. Second, directive feedback learning augments each turn with privileged future information — the agent's previous response, the user's next message, the mined feedback category, and a generated explanation — to form a teacher; the deployed student generates its own reply, and the teacher's token distribution over that same reply is used as dense supervision via reverse-KL divergence. The load-bearing identity is that sparse, noisy natural-language corrections become dense teacher signals only when the teacher sees the user's subsequent utterance and a category label for it.","core_discovery":"On its own terms, the paper's discovery is that real online user logs contain two complementary supervision signals — outcome-level purchase signals and turn-level directive feedback (explicit criticism, implicit deduction, pure negative feedback, and comparative preference) — and that jointly learning from both, sequentially, outperforms learning from either alone. Concretely, optimizing the agent with GRPO over sessions ending in purchases substantially improves recommendation ranking metrics, while feedback-aware on-policy distillation from the user's next utterance substantially improves the rate at which revised responses resolve the user's stated concern. Combining the two in the order purchase-RL first, distillation second yields the best ranking and success-rate numbers across both a 4-billion- and an 8-billion-parameter backbone.","pith_inferences":["Beyond the paper: because the same large language model both mines feedback categories and judges success, some of the reported gap may come from self-consistency rather than from genuinely better responses; an independent human-judged subset would reveal the true effect size.","Beyond the paper: the four-category feedback taxonomy and the privileged-teacher recipe should transfer to other deployed conversational assistants, where users routinely correct the agent in the next turn.","Beyond the paper: a natural extension is to feed richer outcome signals — returns, refunds, repeated queries, or cart abandonment after acceptance — into the reward, since purchase completion alone is a delayed and partial proxy for satisfaction."],"forward_implications":["Deployed shopping agents can be retrained from their own logs on a schedule, because both purchase outcomes and conversational feedback are extracted automatically.","Purchase-level reinforcement learning and turn-level distillation improve different capabilities: RL mainly lifts recommendation ranking, while distillation mainly lifts response helpfulness; the combined LOFA pipeline gets both.","Training order matters: learning reliable outcome-level signals before fine-grained directive signals beats the reverse order, so the sequential pipeline is itself part of the method.","Reasoning traces are not required for feedback-driven optimization; the feedback-learning methods outperform supervised fine-tuning on response-only data while preserving the model's own reasoning.","All four directive feedback categories contribute, with implicit deduction providing the largest gains, so discarding or underweighting any one type would cost performance."],"supporting_citations":[{"why":"Supplies the GRPO reinforcement-learning update used to optimize the agent from purchase outcomes.","marker":"[27]"},{"why":"Supplies the on-policy distillation objective from self-generated mistakes that LOFA adapts with privileged feedback.","marker":"[28]"},{"why":"Defines the backbone agent model that all compared methods start from.","marker":"[32]"},{"why":"Provides the large language model used both to mine feedback categories from user turns and to judge Success Rate; the central results depend on it.","marker":"[33]"}],"fun_headline_variants":["Shopping agents learn from real user directives","Mining purchase outcomes and feedback for shopping agents","Combining purchase and conversational feedback boosts shopping agents","User feedback in dialogue trains shopping agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the large language model used to mine feedback categories and to judge whether revised responses are successful is accurate enough that its decisions track real user satisfaction; the paper reports no human validation of either role, so if that model is biased or lenient, the reported gains could be artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Shopping agents learn from real user directives","Mining purchase outcomes and feedback for shopping agents","Combining purchase and conversational feedback boosts shopping agents","User feedback in dialogue trains shopping agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000232,"raw_usage":{"total_tokens":1445,"prompt_tokens":860,"completion_tokens":585,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":530}},"tokens_in":476,"tokens_out":585,"duration_ms":6696,"temperature":1.0,"reasoning_tokens":530,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:33:22.717275+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the directive-feedback test instances and have human annotators independently label the feedback category and judge whether each revised response resolves the user's complaint; recompute Success Rate with human labels. If LOFA's advantage over baselines shrinks to near zero or reverses under human judgment, the reported gains are artifacts of the automated judge rather than real user-satisfaction improvements.","supporting_citations":[{"cited_title":"On-policy distillation of language models: Learning from self-generated mistakes","cited_arxiv_id":null,"evidence_quote":"Supplies the on-policy distillation objective from self-generated mistakes that LOFA adapts with privileged feedback."}],"review_version":1}