{"id":"363987ec-53ee-4a95-9913-d336a68e532c","arxiv_id":"2509.08500","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"TCPO uses stepwise preference optimization and an action consistency constraint to raise ALFWorld average success from 20.0% to 26.7%.","lead":"This paper introduces TCPO, a training method that compares good and bad reasoning steps to fine-tune vision-language models for embodied household tasks. It reports higher success rates than a reinforcement-learning baseline in the ALFWorld simulator.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Eq. 6 APW approximation is supported only by post-training action-token probabilities, so p(a|T)≈1 is not established at initialization or during early optimization; the claimed equivalence to the Eq. 4 DPO loss remains unverified.","rationale":"The reader's weakest_assumption identifies exactly the point on which the central claim rests. The APW weighting is what differentiates TCPO from a generic stepwise DPO, and the derivation from Eq. 4 to Eq. 6 depends on p(a|T)≈1. The only offered evidence is self-generated by the method and cannot justify the approximation at the point of first use. I do not see an internal inconsistency in Eq. 6 itself; rather, the support is incomplete. The empirical headline (26.7% vs 20.0%) could survive even if this derivation fails, so the appropriate status is conditional, matching the reader. Missing error bars and κ selection on the evaluation metric are additional validity concerns but secondary to the method's theoretical identity. No verdict change is needed.","tokens_in":16573,"tokens_out":8347,"duration_ms":375987,"concrete_test":"One decisive check: after SFT and before any TCPO update, sample the exact ALFWorld preference pairs that will form the first online update, compute p(a|T) under the current πθ and π_ref for winning and losing action tokens, and evaluate Δ in Eq. 7. If the mean |Δ| is not below a small fraction (e.g., 10%) of the mean log-ratio magnitude, Eq. 6 is not a faithful surrogate for Eq. 4 in the regime where it is first applied. This pre-update measurement tests the approximation directly and avoids the circularity of Figure 5c.","verdict_should_be":"UNCHANGED","load_bearing_attack":"TCPO's central derivation replaces the joint stepwise DPO loss (Eq. 4) with the weighted loss (Eq. 6) on the strength of p(a|T)≈1. Equation 7 gives the per-pair error as Δ = log p(a_i|T_i) + (1 − p(a_i|T_i)) log[πθ(T_i|τ)/π_ref(a_i,T_i|τ)], so Eq. 6 is an accurate surrogate only when p(a|T) is close to 1 for every preference pair used. The paper's only empirical support is Figure 5c, which plots action-token probabilities after training with TCPO itself. That evidence is circular: Eq. 6 up-weights actions that already have high p(a|T), so a post-hoc concentration near 1 is a property the loss induces, not an independent measurement of the regime in which the approximation was first applied. This matters because the replay buffer deliberately contains low-reward trajectories (Section 3.1), the same loss is applied to losing samples, and p(a|T) is θ-dependent; during early updates p(a|T) can be far from 1, in which case the gradient of Eq. 6 is not the Eq. 4 gradient (and the Eq. 5 derivation also drops ∂log p(a|T)/∂θ terms). Moreover, the second term in Δ multiplies (1−p) by a log-ratio that can be large during optimization, so even p≈0.99 need not make the approximation error negligible. Until p(a|T) is measured on the initial replay data, the main theoretical justification for APW, and hence for TCPO as a 'thought-centric' DPO variant, is unestablished.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Thought-Centric Preference Optimization (TCPO), an online preference-based fine-tuning method for vision-language model (VLM) embodied agents. TCPO constructs stepwise preference pairs from interaction trajectories, derives a DPO-style loss (Eq. 4) over thought-action pairs, introduces an Action Probability Weighting (APW) loss (Eq. 6) intended to approximate it, and adds an Action Policy Consistency (APC) term that applies L2 regularization between the fine-tuning model's action distribution and a reference model's action distribution (Eq. 8). Experiments on GymCards and ALFWorld report average success rates of 42.9% and 26.7%, respectively, compared with 42.7% and 20.0% for the RL4VLM baseline. The paper also includes ablations over the APC weight κ, a comparison of APW with classic DPO, a joint-optimization study, and a sample-efficiency table.","tokens_in":17123,"tokens_out":6434,"duration_ms":51590,"significance":"If established, the paper's central claim—that stepwise preference optimization of chain-of-thought reasoning, combined with an action-consistency regularizer, improves embodied decision-making and mitigates model degradation—would be a useful contribution to VLM-based embodied agents. The manuscript provides algorithmic pseudocode, a parameter study, and multiple ablations, which are strengths. However, the theoretical justification of the APW loss as a faithful approximation of the stepwise DPO objective is not sound as presented: the gradient derivation omits model-dependent terms, and the only empirical support for the key approximation is measured after training with the very loss that induces the approximation. The experimental reporting also needs improvement, as the main results table lacks variance information and contains inconsistencies with the baseline comparison table. The significance of the empirical result is moderate, but the manuscript's theoretical framing is substantially overclaimed relative to the evidence provided.","major_comments":[{"comment":"The gradient in Eq. 5 is derived as if the action probabilities p(a_t^i|T_t^i) were independent of the policy parameters θ, but these probabilities are outputs of the same model π_θ. The gradient of Eq. 4 with respect to θ should therefore contain terms of the form ∇_θ log p(a_t^i|T_t^i) as well as gradients of the sigmoid argument through the thought probabilities. These terms are omitted without justification, so Eq. 5 does not establish the claimed 'elimination of direct action probability influences.' If p(a|T) is intended to be held constant under a stop-gradient, that must be stated explicitly, and the resulting bias in the update should be analyzed. This issue is load-bearing because it underlies the derivation of the APW loss in Eq. 6.","section":"Section 3.2, Eq. 5"},{"comment":"The validation of the key approximation p(a|T)≈1 is circular. The only empirical evidence, Figure 5c, plots the action-token probability distribution after training with TCPO itself; since the APW loss reweights the objective toward high-probability actions, the loss actively compresses the distribution toward 1, so the post-training measurement cannot independently confirm that p(a|T) was close to 1 during the initial or early optimization phases when the approximation was first applied. The authors should measure p(a|T) on the initial replay buffer and at early training steps, before the loss has substantially shaped the distribution, and compare those values with the assumption.","section":"Section 3.2, Eq. 7 and Figure 5c"},{"comment":"The approximation error Δ in Eq. 7 is not bounded. The second term, (1−p(a|T))·log(π_θ(T|τ)/π_ref(a,T|τ)), contains a log-ratio that can be large during online fine-tuning when the current policy diverges from the reference model. Even for p(a|T)=0.99, a large log-ratio can make the error non-negligible. This is particularly concerning because the replay buffer deliberately includes low-reward trajectories (Section 3.1), for which p(a|T) is most likely to be far from 1. Without a quantitative bound that accounts for the log-ratio, Eq. 6 cannot be considered a reliable surrogate for Eq. 4.","section":"Section 3.2, Eq. 7 and Section 3.1"},{"comment":"The empirical comparison is not fully reported. Table 1 lists only point estimates, although the text and Appendix F state that results across at least 5 seeds with mean and variance were computed; without error bars or confidence intervals, the claimed 6% improvement over RL4VLM cannot be statistically assessed. Additionally, the GymCards result for the PPO/RL4VLM baseline is inconsistent: Table 1 reports 42.7% for RL4VLM, while Table 2 reports 32.8% for PPO. Since the text identifies PPO as the RL4VLM baseline, this apparent contradiction must be resolved by clarifying the experimental conditions or correcting the table.","section":"Section 4.1, Tables 1 and 2"}],"minor_comments":[{"comment":"Several cells in Table 1 are concatenated without separators (e.g., '28.60', '40.370.0', '28.65.9'), making the table difficult to read and interpret correctly.","section":"Table 1"},{"comment":"The phrase 'achieving a 6% improvement' is ambiguous: the reported difference between 26.7% and 20.0% is 6.7 percentage points, or about 33% relative improvement. The paper should state which quantity is meant.","section":"Abstract and Section 1"},{"comment":"The sentence 'where we have Λ instead of σ(Q̂_θ(...) − Q̂_θ(...))' does not define Λ. Please give an explicit definition (e.g., Λ = 1 − σ(...) or similar) so that Eq. 5 can be checked.","section":"Section 3.2, Eq. 4"},{"comment":"Figure 5c shows a density over action-token probabilities, yet the text says it 'quantitatively characterizes the action probability evolution during the initial 2000 training iterations.' A static density does not show evolution; either clarify how aggregation is done or provide a time-resolved plot.","section":"Section 4.3 and Figure 5c"},{"comment":"The phrase 'past trajectories wises buffer' appears to contain a typo; it should likely be 'replay buffer' or 'wise buffer.'","section":"Appendix D, Algorithm 1"},{"comment":"The notation ⊮{invalid} is nonstandard; it should be written as an indicator function, e.g., 1[action not admissible], for clarity.","section":"Section 4.1, Eq. 9"}],"recommendation":"major_revision","confidential_remarks":"The paper's core theoretical claim—that the APW loss faithfully approximates the stepwise DPO objective—is not currently supported; the derivation drops model-dependent gradient terms and the empirical validation is circular. The experimental tables also contain an inconsistency between the baseline numbers in Table 1 and Table 2 that should be checked carefully. The method itself may still be effective, but the manuscript needs either a corrected derivation or a reframing of APW as a heuristic reweighting with separate empirical justification, along with variance reporting and table corrections, before it is publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the empirical headline is real and worth taking seriously: TCPO gets 26.7% average success on ALFWorld against 20.0% for RL4VLM, and the ablations in Table 3 suggest both the action-probability weighting (APW) and the action-policy consistency (APC) term contribute. Second, the theoretical story in Section 3.2 is not as solid as the prose suggests, and the gap is the kind of thing that matters for whether the method generalizes beyond these two simulators.\n\nWhat is actually new: applying stepwise DPO to thought-action pairs in an embodied setting is a natural but non-obvious move, and the L2 consistency constraint that ties the current policy's action distribution to the reference model is a sensible way to combat the common degradation during online RL fine-tuning. The paper does a good job framing the problem — sparse rewards, action-only optimization, model degradation — and the sample-efficiency claim in Appendix E, if it holds up, is a genuine plus. I also appreciate that they test on two environments with different structure rather than one toy task.\n\nThe soft spots are real but proportionate. The derivation from Eq. 4 to Eq. 6 drops gradient terms of the action probability and then re-introduces them heuristically as weights. Equation 7 explicitly quantifies the error as log p(a|T) + (1-p(a|T)) log-ratio, and the paper's only evidence that this is small is Figure 5c, which plots action-token probabilities after training with TCPO itself. That is circular: the loss up-weights high-probability actions, so a post-training concentration near 1 is exactly what the algorithm induces, not an independent measurement of the regime in which the approximation was first applied. The stress-test note is correct that the second term can be large even when p is near 0.99 because the log-ratio can be big early in training. The paper should either measure p(a|T) on the initial replay buffer or reframe APW as a heuristic with empirical justification rather than a derivation.\n\nAlso worth flagging: Table 1 has no variance or error bars even though the appendix says five seeds were used, the κ=0.1 result is tuned on the same ALFWorld evaluation set, and no code is released. These are addressable. The core idea is not invalidated — the empirical result is a few points on two benchmarks, and the method is simple enough that independent replication is feasible.\n\nMy bottom line: this deserves a serious referee. It is a plausible incremental contribution to a real problem, with an honest limitations section and a derivation that is weak but not fraudulent. Send it to review, but the reviewers should demand error bars, code, and either a non-circular validation of the approximation or a rewrite of the theory as motivation rather than proof.","headline":"A plausible incremental DPO variant for embodied VLMs with a real but modest ALFWorld gain; the theoretical justification for the weighted loss is circular, yet the empirical claim deserves replication and the paper merits review with revisions.","tokens_in":17546,"tokens_out":1799,"would_cite":false,"duration_ms":18716,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training the chain of thought lifts embodied AI to 26.7% success","keywords":["Thought-Centric Preference Optimization","embodied decision-making","chain-of-thought reasoning","preference learning","vision-language models","action policy consistency","ALFWorld","model degradation"],"falsifier":"Train the same model with the unweighted loss of Eq. 4 and the weighted loss of Eq. 6 in a setting where initial action probabilities are spread out, such as a randomly initialized policy or an unfamiliar task, and record $p(a_i|T_i)$ during the first 500 training steps; if a substantial fraction of chosen actions have $p < 0.9$, the approximation underpinning APW is violated and the weighted loss should diverge from the intended DPO objective.","tokens_in":16389,"feed_emoji":"🤖","tokens_out":3311,"duration_ms":29339,"temperature":0.7,"pith_summary":"This paper tries to establish that aligning an embodied vision-language agent’s chain-of-thought reasoning, rather than only its final actions, yields better and more stable decision-making in dynamic environments. It proposes TCPO, which converts sparse rewards into stepwise preference pairs and adds a consistency constraint that keeps the action distribution close to the reference model. On ALFWorld, TCPO achieves a 26.7% average success rate versus 20.0% for the RL4VLM baseline, a 6-point improvement, while also mitigating the model degradation that online fine-tuning typically causes. If correct, this means preference-based stepwise optimization over reasoning traces is a more sample-efficient and stable alternative to PPO-style reinforcement learning for embodied agents.","feed_headline":"Training the chain of thought lifts embodied AI to 26.7% success","feed_subtitle":"TCPO steers preference optimization at reasoning steps and holds actions close to the reference model, beating RL4VLM on ALFWorld.","key_machinery":"The central object is the TCPO loss in Eq. 8, $\\tilde{L} + \\kappa \\cdot L_2(\\pi_\\theta(a_1|T_1), \\pi_{\\mathrm{ref}}(a_1|T_1))$, where $\\tilde{L}$ is the action-probability-weighted stepwise preference loss of Eq. 6. APW reweights the log-probability of each reasoning step by $p(a_t|T_t)$, the conditional probability of the action given the thought, suppressing highly random positive samples and promoting deterministic thought-aligned actions. APC adds an L2 penalty pulling the fine-tuned action distribution toward the reference model’s thought-to-action mapping. Preference pairs are built from sparse-reward trajectories by turning success or failure into contrastive stepwise rankings, which supplies dense supervision and higher sample efficiency.","core_discovery":"The central claim is that optimizing a stepwise DPO-style preference loss on thought-action pairs, reweighted by the probability that the action follows the thought (APW), and regularizing the action distribution against a reference model (APC), improves embodied decision-making and prevents degradation during online fine-tuning. The paper reports an average success rate of 26.67% on ALFWorld, compared with 20.0% for RL4VLM and 22.1% for D3PO, and shows that jointly optimizing APW and APC outperforms either component alone. The claim is that thought-centric alignment, not action-only optimization, is what drives both higher task success and more deterministic, legal action generation.","pith_inferences":["The weighting scheme could transfer to any hybrid reasoning-and-acting model, not only VLMs, wherever a conditional probability $p(\\text{action}|\\text{thought})$ is computable, making it a general regularizer for chain-of-thought policies.","The approximation $p(a|T) \\approx 1$ may be partially self-fulfilling: APW itself drives action probabilities toward 1, so the evidence in Figure 5c is generated by the loss it is meant to justify; a clean test would measure the approximation error under an unweighted baseline at the same training steps.","Because the paper notes the Markovian assumption, extending TCPO with explicit memory or temporal modeling could retain its stability gains in long-horizon, non-Markovian tasks.","The APC term could serve as a deployment safety mechanism, keeping an online-adapted agent’s actions close to a vetted reference policy in real-world settings where distribution shift is costly."],"forward_implications":["TCPO achieves higher average task success than PPO and DPO baselines in both ALFWorld and GymCards, suggesting that thought-centric preference optimization generalizes beyond a single benchmark.","The action probability weighting pushes decision action token probabilities toward 1, making the policy more deterministic and reducing the incidence of illegal actions during embodied interaction.","The APC regularization with $\\kappa \\approx 0.1$ stabilizes online fine-tuning and mitigates model degradation; both too little and too much constraint reduce performance.","TCPO reaches the same success-rate thresholds in fewer training steps than PPO or DPO, indicating a practical sample-efficiency gain in sparse-reward interactive environments."],"supporting_citations":[{"why":"Supplies the base DPO objective that TCPO adapts into a stepwise preference loss for thought-action pairs.","marker":"(Rafailov et al., 2024)"},{"why":"Provides the RL4VLM baseline, the SFT data corpus (LEVI-Project/sft-data), and the PPO reproduction setup that TCPO compares against.","marker":"(Zhai et al., 2024)"},{"why":"Offers the stepwise derivation and the Bradley-Terry-style Q-form used to derive the thought-action DPO loss in Eq. 4.","marker":"(Yang et al., 2024)"},{"why":"Defines the ALFWorld benchmark environment used for the main empirical evaluation.","marker":"(Shridhar et al., 2020)"},{"why":"Introduces stepwise preference optimization for long-chain reasoning, which TCPO extends to embodied decision-making.","marker":"(Lai et al., 2024)"}],"fun_headline_variants":["Thought-centric preference optimization: 26.7% on ALFWorld","Stepwise preference over thoughts lifts embodied success to 26.7%","Align reasoning steps, not just actions: TCPO beats RL4VLM","Think-first optimization improves embodied AI by 6%","TCPO: Preference on thought-action pairs outperforms RL4VLM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method relies on the assumption that the probability of the action given the thought is close to 1 during optimization, and the only evidence offered is the action-token distribution produced by the method itself; if that approximation fails in a new environment, the weighted loss changes the objective and the reported gains may not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Thought-centric preference optimization: 26.7% on ALFWorld","Stepwise preference over thoughts lifts embodied success to 26.7%","Align reasoning steps, not just actions: TCPO beats RL4VLM","Think-first optimization improves embodied AI by 6%","TCPO: Preference on thought-action pairs outperforms RL4VLM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000519,"raw_usage":{"total_tokens":2506,"prompt_tokens":931,"completion_tokens":1575,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":1482}},"tokens_in":547,"tokens_out":1575,"duration_ms":11980,"temperature":1.0,"reasoning_tokens":1482,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:01:36.353461+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same model with the unweighted loss of Eq. 4 and the weighted loss of Eq. 6 in a setting where initial action probabilities are spread out, such as a randomly initialized policy or an unfamiliar task, and record $p(a_i|T_i)$ during the first 500 training steps; if a substantial fraction of chosen actions have $p < 0.9$, the approximation underpinning APW is violated and the weighted loss should diverge from the intended DPO objective.","supporting_citations":[],"review_version":2}