{"id":"90afaaf4-a056-46fd-bccd-0f256c97b95e","arxiv_id":"2412.07472","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"SmartAgent is a GUI agent that adds user-preference reasoning through three thought steps, but its intermediate 'underlying requirement' step does not improve item recommendation over end-to-end training.","lead":"This paper introduces Chain-of-User-Thought (COUT), a reasoning paradigm for embodied agents that adds user preference reasoning to GUI navigation, and a companion dataset SmartSpot built from Meituan. It presents SmartAgent, a vision-language model fine-tuned in two stages to navigate GUIs, infer user needs, and recommend items.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 5's own ablation contradicts the central claim: the end-to-end model without the explicit COUT intermediate reaches Imp.Acc 0.31 versus 0.24 for the proposed two-stage pipeline, so the reported data do not support that COUT improves personalized reasoning.","rationale":"I read the paper as attempting to establish COUT, an explicit intermediate 'underlying requirement' step, as a valuable paradigm for embodied personalized agents. The strongest empirical support would be an ablation showing that inserting this step improves downstream personalized recommendation. Table 5 is designed to be that ablation, but its numbers go the wrong way: end-to-end Imp.Acc 0.31 > two-stage 0.24, and end-to-end embodied metrics are also slightly higher. The paper's rhetorical explanation does not supply significance testing or error bars. The benchmark is small (144 episodes, about 1,400 steps), so the difference could be noise, but the burden is on the authors to show the claimed benefit. Without released code and data, verification is impossible. I therefore agree with the reader's weakest assumption and see no reason to change the REJECT verdict.","tokens_in":14278,"tokens_out":2994,"duration_ms":29333,"concrete_test":"Release the SmartSpot dataset and training code, then reproduce Table 5 with at least 5 random seeds per condition, keeping LoRA budget and training steps identical between Two-stage and End-to-End. Report paired bootstrap 95% confidence intervals and per-channel Imp.Acc for each condition. If end-to-end Imp.Acc is not significantly below two-stage—or remains higher—the paper's assertion that the explicit COUT intermediate improves personalized recommendation is falsified by its own design.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that COUT, the two-stage chain from GUI actions through an explicit 'underlying requirement' (Thought #2) to item recommendation, delivers full-stage embodied personalized reasoning. The decisive evidence should be an ablation, and Section 6.6 provides it: 'End-to-End' training without the underlying-thought step achieves Imp.Acc 0.31, while Two-stage training achieves 0.24. End-to-end also edges out embodied metrics (Ele.Acc 0.67 vs 0.64, SSR 0.53 vs 0.50). Since Imp.Acc is the paper's own metric for implicit personalized recommendation, the ablation shows the intermediate COUT step does not improve—and if anything hurts—the central personalized outcome. The paper's explanation (overfitting to long episodes) is post hoc, and no confidence intervals, random seeds, or per-channel breakdowns are reported. The 'first full-stage embodied personalized reasoning' claim therefore rests on an assumption that the paper's own data contradict. This is a correctness risk in the central argument, not merely a framing issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Chain-of-User-Thought (COUT), a personalized reasoning paradigm for embodied GUI agents in which the agent first navigates a GUI (Thought #1), then summarizes the user's explicit underlying requirement (Thought #2), and finally recommends items from the resulting pool (Thought #3). The authors instantiate COUT in SmartAgent, a Qwen-VL-based model trained with LoRA in two stages, and introduce SmartSpot, a 144-episode benchmark built from Meituan GUI trajectories with annotations for instructions, underlying requirements, and item recommendations. The evaluation reports embodied action metrics (Ele.Acc, SSR) and personalized metrics (Exp.Acc, Imp.Acc) on SmartSpot, plus transfer results on ScreenSpot and Mind2Web.","tokens_in":14463,"tokens_out":5151,"duration_ms":48483,"significance":"The paper addresses a real gap: embodied agents that handle ambiguous user queries and personalize item recommendations are rarely studied, and the SmartSpot benchmark with explicit annotations for embodied actions and user preferences is a potentially useful community resource. The authors also evaluate on standard GUI benchmarks (ScreenSpot, Mind2Web), which is good practice. However, the central claim that COUT's explicit intermediate reasoning improves personalized recommendation is not supported by the reported experiments: the end-to-end baseline in Table 5 obtains a higher Imp.Acc than the proposed two-stage pipeline, and the absolute values of the key personalized metric are low. As submitted, the evidence does not substantiate the 'first full-stage embodied personalized reasoning' claim.","major_comments":[{"comment":"The ablation reported in Table 5 undercuts the central claim of the paper. End-to-End training without the explicit underlying-requirement step achieves Imp.Acc 0.31, while the proposed Two-stage COUT pipeline achieves 0.24; End-to-End is also higher on Ele.Acc (0.67 vs. 0.64) and SSR (0.53 vs. 0.50). Since Imp.Acc is the paper's own metric for implicit personalized recommendation, the reported data do not support the claim that the COUT intermediate improves personalized reasoning and in fact suggest the opposite. The explanation in Section 6.6 (overfitting to long episodes) is post hoc, and no confidence intervals, number of random seeds, or per-channel breakdowns are reported. This is a load-bearing issue for the paper's main contribution.","section":"Section 6.6, Table 5"},{"comment":"The claim of 'first full-stage embodied personalized reasoning' is evaluated only on the self-constructed SmartSpot benchmark. There is no external personalized benchmark, and the baselines in Table 2 report no Exp.Acc or Imp.Acc values, so no comparison supports the personalized claims. The absolute performance is low: SmartAgent's overall Imp.Acc is 0.24, and zero-shot Imp.Acc in Table 4 is 0.14. Additionally, the benchmark construction in Section 4.2 relies on annotator-generated intention seeds and the authors' own definitions of explicit and implicit preferences, but no inter-annotator agreement or validation of these definitions is provided. The generalizability of the task formulation therefore remains unverified.","section":"Section 6.1, Table 2; Section 4.2"},{"comment":"The zero-shot results are described as showing that SmartAgent 'surprisingly exceeds' full-stage fine-tuning on Exp.Acc (0.77 vs. 0.71) and as evidence of 'preliminary zero-shot reasoning capability.' However, the same table reports zero-shot Imp.Acc of 0.14, which is far below the 50% chance level of the binary Yes/No recommendation action under a balanced class assumption. No statistical tests or confidence intervals are provided, and the overfitting explanation is speculative. The evidence does not support the claim of robust zero-shot personalized reasoning.","section":"Section 6.5, Table 4"}],"minor_comments":[{"comment":"The dataset contains only 144 episodes and roughly 1,400 steps total, with 10-20 episodes per channel. Results on such a small benchmark are likely noisy; please report the train/validation/test split and per-channel standard errors or bootstrap confidence intervals.","section":"Section 4.1, Table 1"},{"comment":"The distinction between the Perceiver and the Reasoner is underspecified: it is unclear whether both are initialized from the same Qwen-VL weights, whether the Reasoner is fine-tuned or frozen, and how Thought #2 is fed into the personalized stage.","section":"Section 5.2"},{"comment":"Exp.Acc is defined only as 'semantic similarity' between the predicted underlying requirement and ground truth, but no similarity measure, threshold, or aggregation procedure is specified, making the reported 0.71 value difficult to interpret or reproduce.","section":"Section 6, Metrics"},{"comment":"On Mind2Web, SmartAgent is below SeeClick on most metrics (for example, Cross-Task SSR 20.6 vs. 25.5), so the statement that it achieves 'second place generally' should be qualified to the specific category and metric.","section":"Section 6.3, Appendix Table 6"},{"comment":"The case study in Figure 5 contains apparent typos (e.g., a flight duration '23.5h' and a price ￥710 for a 7-hour route), which should be corrected before publication.","section":"Section 6.4, Figure 5"}],"recommendation":"reject","confidential_remarks":"The central contribution is an empirical claim that the two-stage COUT pipeline improves personalized embodied reasoning. The paper's own ablation (Table 5) shows the end-to-end variant performs better on the key personalized metric, and the zero-shot results are weak. Unless the authors can provide substantially stronger evidence, such as a large-scale external benchmark, multiple seeds with variance estimates, and an ablation that actually favors COUT, I do not see how the central claim can be sustained. The SmartSpot benchmark may still be a useful resource, but the current manuscript overstates what is demonstrated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful new benchmark and task formulation, but the paper's central claim about the benefit of Chain-of-User-Thought is undercut by its own ablation in Table 5. End-to-end training (without the intermediate \"underlying requirement\" step) reaches 0.31 Imp.Acc vs 0.24 for the proposed two-stage pipeline, on the paper's own implicit-preference metric. That is load-bearing, not a framing quibble.\n\nWhat the paper does well: it defines an embodied personalized task—GUI navigation plus item recommendation from ambiguous user instructions—and builds SmartSpot, a 144-episode benchmark from Meituan screenshots with five single-channel and two multi-channel scenarios. The authors also check whether personalized training destroys basic GUI competence by evaluating on ScreenSpot and Mind2Web. COUT, a three-step chain from GUI action to explicit underlying need to implicit preference, is a reasonable extension of Chain-of-Action-Thought [51], and the paper honestly reports the ablation that contradicts its own thesis. Credit for that transparency.\n\nThe soft spots: the central argument does not survive the reported numbers. Table 5 shows the end-to-end variant beating the two-stage model not only on Imp.Acc (0.31 vs 0.24) but also on Ele.Acc and SSR. The authors' explanation—overfitting on long episodes—is post hoc and unsupported by additional experiments. There are no confidence intervals, no multiple seeds, and code and data are not yet released despite the abstract promising them. Absolute accuracies are low (0.24 Imp.Acc on the main benchmark, 0.14 zero-shot), which weakens the \"first full-stage embodied personalized reasoning\" claim. The zero-shot result on MEDICINE is interesting but mixed: Exp.Acc is high (0.77) while Imp.Acc drops to 0.14.\n\nWho this is for: people working on GUI agents or multimodal recommendation will find the benchmark and task definition worth following up on. As a method paper, the evidence for COUT is not there yet. A serious referee could ask for seeds, error bars, a stronger baseline, and a revised claim. I would send it to review rather than desk-reject, but with major revision expectations.\n\nRecommendation: engage with this as a benchmark contribution, not as a demonstrated method. It deserves a serious referee, and the authors should be pushed to either fix the evaluation or scale back the claim.","headline":"New benchmark and task formulation, but the central COUT claim is contradicted by the paper's own ablation: end-to-end training beats the two-stage pipeline on the paper's main personalization metric.","tokens_in":15016,"tokens_out":2349,"would_cite":false,"duration_ms":23316,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SmartAgent's three-step thought chain aims to make embodied agents personalize recommendations.","keywords":["Chain-of-User-Thought","embodied agent","personalized recommendation","GUI navigation","large vision-language model","SmartSpot benchmark","implicit preference","explicit requirement"],"falsifier":"On a larger held-out version of SmartSpot, train the same backbone with and without the explicit Thought #2 intermediate text, keeping training compute and data identical; if end-to-end training matches or exceeds the two-stage Imp.Acc across channels, the claim that COUT's intermediate underlying-requirement reasoning improves item recommendation would be refuted. Similarly, an ablation that removes Thought #2 at inference and still gets the same Imp.Acc would show the explicit requirement is not load-bearing.","tokens_in":14058,"feed_emoji":"🤖","tokens_out":3871,"duration_ms":34560,"temperature":0.7,"pith_summary":"This paper claims that embodied agents, systems that perceive and act on a user interface, can be trained to personalize by reasoning in three explicit steps: deciding the next GUI action, stating the user's underlying requirement in words, and then answering yes or no to each candidate item. It introduces Chain-of-User-Thought (COUT) as a general paradigm for this progressive reasoning, and SmartAgent as an implementation built on a vision-language model. To test the idea, the authors built SmartSpot, a benchmark of 144 GUI episodes across seven life-service channels, with annotations for both actions and user preferences. The central claim is that this is the first system to achieve full-stage embodied personalized reasoning, from basic screen operation to implicit item preference.","feed_headline":"SmartAgent chains user thoughts for personalized GUI agents","feed_subtitle":"A new benchmark tests agents that reason from GUI actions to item picks, with zero-shot results on unseen channels.","key_machinery":"The central mechanism is the Chain-of-User-Thought pipeline, instantiated as a two-stage training regime. In the embodied stage, a Perceiver model, initialized from the SeeClick GUI agent, maps screenshots, instructions, and eight past actions to the next GUI action; a Reasoner then condenses that action history into a short textual 'underlying requirement.' In the personalization stage, the same Perceiver reads item-pool screenshots and outputs 'Yes' or 'No' per item. The explicit intermediate Thought #2 is the load-bearing connector: it turns ambiguous instructions into concrete constraints that narrow the item pool before recommendation.","core_discovery":"SmartAgent is the first agent to couple GUI navigation, explicit user requirement inference, and implicit item recommendation in a single embodied reasoning chain. The paper's core claim is that a two-stage training pipeline, first teaching a Perceiver model to emit GUI actions (Thought #1) and a Reasoner to summarize the user's underlying requirement (Thought #2), then teaching the same Perceiver to output 'Yes' or 'No' recommendations (Thought #3), enables personalized reasoning that task-only embodied agents lack. The authors report that SmartAgent reaches 0.64 Element Accuracy and 0.50 Step Success Rate on SmartSpot, predicts 71% of explicit underlying requirements correctly, and recommends the correct items 24% of the time. They also report competitive GUI-grounding results on ScreenSpot and second-best pure-vision results on Mind2Web, which they interpret as evidence that adding personalization does not destroy basic embodied skills.","pith_inferences":["The reported Table 5 comparison suggests that the intermediate underlying-requirement text may not actually improve item recommendation accuracy over end-to-end training; a reader might infer that COUT's benefit lies in interpretability or step-wise supervision rather than raw Imp.Acc.","The benchmark's small scale of 144 episodes and over 1,400 steps means the headline numbers are early evidence; an extension would be to build a larger, multi-platform version to test whether the two-stage advantage, or lack of it, persists.","If the explicit requirement text is what enables zero-shot transfer on MEDICINE, then a testable extension is to measure how recommendation quality degrades when Thought #2 is corrupted or removed at inference.","The paper's framing suggests that user-oriented diversity, not task completion, is the ultimate objective; a further implication is that evaluation metrics for embodied agents should include preference alignment, not just action accuracy."],"forward_implications":["If COUT works, personal assistants on phones and web platforms can be trained on ambiguous queries without golden action trajectories, allowing diverse user paths as supervision.","The explicit underlying-requirement text gives a human-readable explanation for why an item was recommended, which could support debugging and user trust.","The same two-stage recipe could transfer to other cyber environments such as e-commerce, travel booking, and hotel selection wherever GUI access to an item pool is followed by selection.","The zero-shot result on the held-out MEDICINE channel suggests that COUT agents can be dropped into new channels with little or no fine-tuning, which matters for cold-start personalization.","Combining GUI behavior signals like clicks and scrolls with item content could improve recommendation in scenarios where back-end item sequences are unavailable."],"supporting_citations":[{"why":"Supplies the SeeClick base model and ScreenSpot benchmark that SmartAgent continues pre-training from for GUI grounding.","marker":"[3]"},{"why":"Provides the Mind2Web benchmark used to test SmartAgent's autonomous GUI operation and general embodied skill retention.","marker":"[6]"},{"why":"Defines the action-space schema and evaluation metrics for click, type, scroll, complete, and recommendation actions.","marker":"[31]"},{"why":"Qwen-VL is the backbone large vision-language model for both the Perceiver and Reasoner components.","marker":"[1]"},{"why":"LoRA is the parameter-efficient fine-tuning method used to adapt both the visual encoder and the LLM.","marker":"[19]"},{"why":"AndroidWorld is cited as a dynamic environment where task-only agents fail to capture user preference, motivating the COUT paradigm.","marker":"[30]"}],"fun_headline_variants":["SmartAgent: first to chain GUI actions to user preferences","SmartAgent models user thoughts to recommend items","Chain-of-User-Thought: reasoning from clicks to picks","SmartAgent: turning GUI actions into personalized recommendations","First embodied agent to reason about user preferences from GUI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that inserting an explicit 'underlying requirement' text between GUI actions and item selection improves personalized recommendation compared with training the same model end-to-end on the same data; the paper's own Table 5 reports end-to-end Imp.Acc of 0.31 versus 0.24 for two-stage, so that premise is not supported by the reported numbers.","fun_headline_variants_meta":{"raw":{"variants":["SmartAgent: first to chain GUI actions to user preferences","SmartAgent models user thoughts to recommend items","Chain-of-User-Thought: reasoning from clicks to picks","SmartAgent: turning GUI actions into personalized recommendations","First embodied agent to reason about user preferences from GUI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000817,"raw_usage":{"total_tokens":3607,"prompt_tokens":999,"completion_tokens":2608,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":2532}},"tokens_in":615,"tokens_out":2608,"duration_ms":17477,"temperature":1.0,"reasoning_tokens":2532,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:47:37.295449+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a larger held-out version of SmartSpot, train the same backbone with and without the explicit Thought #2 intermediate text, keeping training compute and data identical; if end-to-end training matches or exceeds the two-stage Imp.Acc across channels, the claim that COUT's intermediate underlying-requirement reasoning improves item recommendation would be refuted. Similarly, an ablation that removes Thought #2 at inference and still gets the same Imp.Acc would show the explicit requirement is not load-bearing.","supporting_citations":[{"cited_title":"Mind2web: Towards a generalist agent for the web","cited_arxiv_id":null,"evidence_quote":"Provides the Mind2Web benchmark used to test SmartAgent's autonomous GUI operation and general embodied skill retention."},{"cited_title":"Androidinthewild: A large- scale dataset for android device control","cited_arxiv_id":null,"evidence_quote":"Defines the action-space schema and evaluation metrics for click, type, scroll, complete, and recommendation actions."},{"cited_title":"Lora: Low- rank adaptation of large language models","cited_arxiv_id":null,"evidence_quote":"LoRA is the parameter-efficient fine-tuning method used to adapt both the visual encoder and the LLM."}],"review_version":1}