{"id":"3514d076-7c26-45ca-b118-c601b4063da1","arxiv_id":"2504.21383","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"FAST-Q estimates counterfactual recommendation outcomes offline by learning policy-invariant state representations, and reports business metric gains on a gaming platform.","lead":"This paper describes an offline reinforcement learning system for recommending game challenges to online players, designed to estimate what would happen under recommendations the player never received. The authors report small measured improvements in engagement and cost on their own gaming platform.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Causal identification in Section 3.4 is asserted, not established: the GRL balancing condition does not imply unconfoundedness, and the paper never validates counterfactual estimates against ground truth.","rationale":"I read the paper in good faith: FAST-Q is an applied offline RL system for recommending challenges on a gaming platform, and the strongest claim is that the GRL-balanced representation removes policy-specific confounding and thereby enables reliable counterfactual action estimation. The most load-bearing condition for that claim is the identification step in Section 3.4. The paper asserts that the balanced representation makes E[R_t | Θ(β(S_t)), A_t] a counterfactual estimate, but it does not establish the causal assumptions needed for that equality. The balancing condition actually trained—matching the representation distribution across policies conditional on the policy-specific predicted action—does not imply unconfoundedness: actions may still depend on unmeasured time-varying confounders (e.g., the player's intent, which the paper itself names as a driver), and the critic remains a purely observational regression. Without a ground-truth counterfactual benchmark, the paper's offline plots (wider Q spread, lower dropout variance) and online A/B lifts cannot distinguish genuine counterfactual estimation from the regularizing effects of the decomposition loss, the exploration schedule, or the policy mixture. This is why the concern is load-bearing rather than a stylistic objection: the paper's central contribution is the counterfactual claim, and it is unverified. I considered other weaknesses: the Q-decomposition loss in Section 3.5.2 is not standard and could bias Q-values (the paper itself notes γ cannot exceed 0.7), the D4RL results are only against TD3+BC and not against the closest prior balancing method [5], and the online results lack confidence intervals. These are real but secondary; even if fixed, they would not resolve the identification gap. The reader's weakest_assumption already identifies the same issue, and the appropriate remedy is a conditional acceptance with a requirement to validate the counterfactual estimates on a semi-synthetic benchmark where ground truth is known. Therefore I recommend no change to the reader's verdict.","tokens_in":13610,"tokens_out":10577,"duration_ms":111744,"concrete_test":"On a semi-synthetic environment with known time-varying confounding and known ground-truth counterfactual outcomes (e.g., the tumor-growth simulator from Bica et al., 2020, adapted to multiple policies and continuous actions), train FAST-Q exactly as specified, including the PE, BR, and critic. Then estimate E[R_t | Θ(β(S_t)), A_t] for counterfactual action sequences under a target policy and compare to the true Monte Carlo counterfactual returns. Report bias and RMSE for the BR-augmented critic versus an unadjusted supervised critic trained on S_t directly, over the same states and actions. If the BR does not substantially reduce bias relative to the unadjusted critic, the central counterfactual claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that FAST-Q 'enables counterfactual estimation' (abstract, Section 1) depends on the identification assumption in Section 3.4: after GRL training, the balanced representation Θ(β(S_t)) removes 'policy-specific bias', so the supervised critic estimates E[R_t | Θ(β(S_t)), A_t] as the counterfactual outcome. The paper never states or verifies the required causal assumptions (sequential ignorability, positivity, consistency). The balancing condition written as P[Θ(β_{p1}(S_t)) | φ_{p1}(β_{p1}(S_t))] = ... = P[Θ(β_{pn}(S_t)) | φ_{pn}(β_{pn}(S_t))] only equalizes the representation distribution across policies conditional on each policy's predicted action. It does not imply treatment assignment independence from potential outcomes given the representation, nor does it control for unmeasured confounders such as player intent, which the paper itself identifies as a driver of both actions and rewards. If unmeasured confounding remains, the counterfactual Q-values are biased; the reported online lifts could come from the added exploration/regularization rather than from valid counterfactual estimation. No experiment in the paper tests the estimated counterfactual values against ground truth: Table 2 uses single-policy D4RL datasets, which cannot assess cross-policy counterfactual validity, and the online A/B test only measures aggregate outcomes. The paper's own limitation that γ could not be raised beyond 0.7 (Section 3.5.3) further weakens the long-term counterfactual interpretation. This gap is load-bearing because the method's distinguishing claim is exactly that the balanced representation makes counterfactual Q-values reliable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FAST-Q, an offline reinforcement learning method for challenge recommendation on a real-money gaming platform. The architecture combines per-policy LSTM-based \"policy experts,\" a gradient reversal layer that learns a policy-balanced state representation, a TD3-style actor-critic with behavior cloning, a Q-value decomposition into learned weights over reward components, and epsilon-greedy selection of counterfactual actions from other policies. Evaluation consists of retrospective offline analyses, a 21-day online A/B test against TD3+BC, Diffusion-QL, and two in-house policies, and D4RL Gym-MuJoCo benchmarks. The paper reports improvements in engagement, dwell time, LTV, and reduced recommendation cost, and claims that the balanced representation enables counterfactual action estimation.","tokens_in":13974,"tokens_out":10012,"duration_ms":105788,"significance":"If the counterfactual estimation claim were established, this would be a valuable practical advance: it directly targets multi-policy logged data with state-space shift, provides a deployed system with a real A/B test, ships code and hyperparameters, and compares against strong baselines. The D4RL experiments and ablations are also useful for the community. However, the central causal claim is not currently validated: the balancing procedure is not shown to remove confounding, and most offline evidence is based on the model's own Q-values rather than ground-truth counterfactual outcomes. The real-world A/B results are a strength, but they do not by themselves identify which component causes the observed gains.","major_comments":[{"comment":"The balancing condition in Section 3.4 asserts that after adversarial training the critic estimates E[R_t | Θ(β(S_t)), A_t] as a counterfactual outcome, but the condition only enforces invariance of the representation distribution across observed policies conditional on each policy's predicted action. It does not establish unconfoundedness (sequential ignorability), positivity, or consistency; these assumptions are never stated or verified. The text itself identifies player intent as a driver of both actions and rewards, so if intent is not fully captured in S_t, or if state support does not overlap across policies, the counterfactual Q-values remain biased. No experiment compares estimated counterfactual values against ground truth: the higher Q-spread in Figure 13 and the lower dropout variance in Figure 16 are internal consistency checks, not correctness checks. The D4RL experiments in Section 4.4 use single-policy datasets and therefore cannot test cross-policy counterfactual validity. I recommend either adding semi-synthetic validation with known counterfactual outcomes and overlap/positivity diagnostics, or explicitly reframing the contribution as robust out-of-distribution generalization rather than causal counterfactual estimation.","section":"Section 3.4"},{"comment":"The Q-value decomposition weights w1..w3 (and overflow w4) are optimized to minimize (R^i_t - w_i Q(s,a))^2, and Figures 10-12 then report these same weights as \"objective prioritization.\" This interpretation is circular: the weights restate the fitted linear relationship between the scalar Q and the reward components; they are not an independent explanation of the agent's trade-offs. Furthermore, because Q is a discounted sum of future rewards rather than the instantaneous reward vector, the loss does not in general identify the contribution of each objective to Q. The paper should validate the decomposition on held-out reward-component prediction or against known objective preferences, or the explainability claim should be softened.","section":"Section 3.5.2"},{"comment":"The claimed offline \"time speedup\" and the ablation effects are measured in terms of FAST-Q's own Q-values normalized to its own full-data or deployed model (Figures 17 and 18). Faster convergence of internal Q-values under counterfactual exploration does not establish that a better policy is learned faster; a biased critic can converge quickly. These analyses should be reported with a policy-performance measure such as normalized return on a held-out evaluation set or D4RL-style scores, or at minimum the paper should state clearly that only internal Q-value convergence is being measured.","section":"Sections 4.2.3 and 4.2.4"},{"comment":"The adaptation to single-policy D4RL datasets is under-specified and internally inconsistent. Removing the PE layer and \"modifying BR layer training\" is not sufficient for reproducibility: gradient reversal requires a domain/policy classifier, but a single-policy medium-v0 dataset has no policy labels. The paper should specify what domains are used, how the GRL is trained, and why this still corresponds to the proposed method. In addition, the text states that FAST-Q \"misses on two out of the remaining three by narrow margins,\" but Table 2 shows three losses, including halfcheetah-medium-expert-v0 with a severe degradation (50.9 vs 97.9). This inconsistency should be corrected and the applicability claim tempered.","section":"Section 4.4 and Table 2"}],"minor_comments":[{"comment":"The \"weighted MSE loss\" for the policy experts does not specify the weights applied to the three action dimensions; please provide the exact weighting.","section":"Section 3.3"},{"comment":"Please clarify how the softmax over the four weights interacts with the three reward components and with the cost objective, which the text says is not controlled; the relationship between C=3 and four weights needs a precise statement.","section":"Section 3.5.2"},{"comment":"The inability to increase the discount factor beyond gamma=0.7 is a substantive limitation for long-term objectives and should be acknowledged in the conclusion as well as in the body.","section":"Section 3.5.3"},{"comment":"The retrospective analysis that led to the \"policy switch\" conclusion is not described: please provide the matching or confounder-adjustment procedure used to define \"statistically similar states.\"","section":"Section 4.2.1"},{"comment":"There are typographical and consistency issues, including \"Grandient\" in Section 1, \"Diffuion-QL\" in the Conclusion, and inconsistent capitalization of \"FAST-Q\" versus \"Fast-Q\" in Table 2 and elsewhere.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The real-world deployment and A/B test are genuine strengths, but the title and abstract overclaim counterfactual validity in a way that will likely attract serious criticism. The causal identification gap in Section 3.4, the circularity of the Q-decomposition explanation, and the reliance on internal Q-values for speedup/ablation claims all need to be addressed before the paper can be considered further. If the authors cannot supply ground-truth counterfactual validation or a clear reframing, I would not support acceptance. The halfcheetah-medium-expert-v0 result (50.9 vs 97.9) should be reported and discussed honestly rather than subsumed by the \"misses on two of three\" wording."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This one is worth a look if you care about offline RL outside Gym benchmarks. The paper builds a domain-adversarial representation layer into an actor-critic recommender, claims it makes counterfactual action evaluation reliable, and backs it with a 21-day A/B test on a real gaming platform. My honest take: the engineering is plausible and the deployment data is rare, but the causal claim is where the paper overreaches.\n\nWhat's actually new: applying gradient reversal to balance state representations across policies for offline counterfactual action evaluation. That's a sensible adaptation of Ganin et al. and Bica et al., and it fits the setting: different recommendation policies see different state distributions, so there is a real domain shift. The fast-track exploration--randomly sampling actions from other policies during actor updates--is a neat way to expand offline data without online experiments. The Q-value decomposition into reward components is a nice touch for explainability, and the authors ship code and describe a 2M-recommendation, 4-month dataset. That is real evidence for an applied paper.\n\nThe soft spots are in proportion. The central identification assumption in Section 3.4 is simply asserted: making the representation Θ(β(S_t)) \"balanced\" across policies does not imply unconfoundedness. The paper never states or checks sequential ignorability, positivity, or consistency. It also never tests the estimated counterfactual values against ground truth. The D4RL experiments use single-policy datasets, so they cannot validate the cross-policy counterfactual machinery. The online A/B test shows lifts, but they are small (0.15% returns, 2% LTV) and the attribution is muddled: the system changes representation, exploration, and objective decomposition all at once, so the gains could come from added regularization or a better policy mixture. The Q-decomposition is partly circular--estimating weights to reconstruct rewards and then reporting those weights as \"objective prioritization\" is a restatement of the fit. And the gamma cap of 0.7 means long-term value is effectively under-discounted, weakening the \"long-term objective\" story.\n\nThat said, the method may well work as a practical heuristic. The authors are honest about some limitations, and the deployed results on a large user base are not nothing. The paper should either soften its counterfactual language or run a validation experiment (e.g., synthetic data with known counterfactuals, or a hold-out where actions are chosen by a known different policy).\n\nWho is this for? Practitioners in recommender systems and anyone interested in offline RL with multiple behavior policies. It deserves a serious referee; the main revision request should be a ground-truth counterfactual check or a more modest claim. I'd send it out.","headline":"A solid engineering contribution that imports domain-adversarial training into offline recommender RL, but the 'counterfactual estimation' claim is not backed by validation; the paper deserves peer review with a request for a ground-truth check or a softened claim.","tokens_in":14497,"tokens_out":2918,"would_cite":false,"duration_ms":29431,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FAST-Q claims that adversarially balancing state representations across policies enables offline reinforcement learning to estimate counterfactual actions reliably, and reports live-platform gains in returns, lifetime value, engagement…","keywords":["offline reinforcement learning","counterfactual action estimation","gradient reversal layer","balanced state representation","recommender systems","multi-objective Q-value decomposition","policy regularization","gaming platform"],"falsifier":"Run a prospective experiment where players are randomly reassigned among the three policies, record the actual rewards for actions that FAST-Q would have judged counterfactual, and compare realized rewards with FAST-Q's predicted Q-values. If the prediction error is systematically larger for actions originating from non-serving policies than for actions from the serving policy, balancing has not removed the policy bias; equivalently, a held-out policy classifier that retains above-chance accuracy on the balanced representation would falsify the invariance.","tokens_in":13435,"feed_emoji":"🎮","tokens_out":6298,"duration_ms":63806,"temperature":0.7,"pith_summary":"FAST-Q tries to make offline reinforcement learning usable in recommender systems where logged data comes from several competing policies with only partially overlapping state spaces. Its central move is to erase policy-specific bias from player-state representations using gradient-reversal adversarial training, so that actions taken by a different policy than the one that served a player can be evaluated as true counterfactuals. The paper argues this fixes the standard offline-RL failure of clipping such counterfactual actions as out-of-distribution, and it couples the balanced representation with a decomposed Q-value loss and an offline exploration schedule. If this works, a recommender can compare alternative challenge policies on static logs and switch policies without months of online experiments; the paper reports lifts in player returns, lifetime value, engagement, dwell time, and cost on its live gaming platform.","feed_headline":"Balanced states unlock counterfactual actions for offline RL","feed_subtitle":"A gradient-reversal layer strips policy bias from player states, so offline models can try counterfactual actions.","key_machinery":"The Balancing Representation $\\Theta(\\beta(S_t))$ is the load-bearing object: each policy expert's LSTM hidden state passes through a dense layer, and the result is trained adversarially so that a softmax policy classifier cannot tell which policy produced the state while a critic still predicts outcomes from it. The Gradient Reversal Layer forces the representation to trade off policy invariance against outcome prediction. This representation feeds an actor-critic built on TD3+BC-style behavior-cloning regularization, while a complementary loss decomposes the critic output into per-reward-dimension weights for dwell time, engagement, return time, and an overflow term, which keeps Q-values bounded and makes objective priority explainable. A stepwise discount schedule and an $\\epsilon$-greedy sampler of counterfactual actions complete the training loop.","core_discovery":"The paper's central claim is that a Balanced Representation, built by training a policy classifier with a Gradient Reversal Layer so that the representation maximizes policy-classification error while minimizing critic loss, makes the state representation invariant to which policy generated it. Under that invariance, the supervised critic estimates $E[R_t \\mid \\Theta(\\beta(S_t)), A_t]$ as the counterfactual outcome of any policy's action, and the actor can be trained on counterfactual actions sampled from other policy experts without the Q-value overestimation that out-of-distribution clipping causes. The paper also claims that decomposing the critic's Q-value into per-objective weighted components yields both explainable objective prioritization and more reliable Q-values, and that removing any of the three components—balanced representation, offline exploration, or Q-value decomposition—measurably lowers estimated returns.","pith_inferences":["Beyond the paper, the same balancing trick should transfer to any logged recommender or decision-support setting where experimental path logic biases state coverage, such as personalized pricing, content feeds, or clinical treatment assignment.","A direct testable extension is to train the policy classifier on a held-out set of states and measure whether its accuracy is at chance after the gradient-reversal step; residual accuracy would quantify how much policy bias survives.","The paper's own report that training stops at a discount factor of 0.7 leaves long-horizon behavior of the decomposed Q-values uncharacterized; whether the decomposition loss permits higher discounts with a slower schedule is an open question.","Because the platform's A/B conclusion framework uses permutation testing with support percentages rather than classical normality assumptions, the reported lifts are tied to that inferential choice; re-analyzing the same logs with different tests would show sensitivity."],"forward_implications":["Offline counterfactual exploration becomes possible: the actor can sample actions from non-serving policies during training, so learning no longer waits for online experiments on those policies.","Recommender policies can be compared and switched on static logs; the paper reports FAST-Q favoring a new mix of the three platform policies rather than any single one.","The balanced representation widens the spread of Q-values across counterfactual actions, which is evidence that the critic can actually distinguish alternatives instead of collapsing onto one narrow band.","Multi-objective Q-decomposition gives per-player, per-time priority weights, making recommendations explainable while keeping Q-values bounded; the paper reports that ablating it drops maximum achievable Q-values by about 40%.","On the live platform, the claims translate to at least 0.15% higher player returns, 2% higher lifetime value, 0.4% higher engagement, 2% higher dwell time, and at least 10% lower recommendation cost over the SOTA baselines."],"supporting_citations":[{"why":"It supplies the Gradient Reversal Layer and the domain-adversarial premise that invariant representations enable transfer across domains.","marker":"[18]"},{"why":"It transfers adversarial balanced representations to counterfactual treatment-outcome estimation over time, the template FAST-Q adapts.","marker":"[5]"},{"why":"It provides the TD3+BC base actor-critic and the behavior-cloning regularization term with the recommended $\\alpha=2.5$.","marker":"[13]"},{"why":"It contributes the double-Q and delayed-policy-update mechanisms that FAST-Q uses to limit Q-value overestimation.","marker":"[14]"},{"why":"It supplies Diffusion-QL, a principal SOTA baseline that FAST-Q compares against in offline and online experiments.","marker":"[39]"},{"why":"It provides the D4RL MuJoCo datasets used as the open benchmark for showing that balanced representations transfer to single-policy control tasks.","marker":"[10]"},{"why":"It represents prior offline-RL behavior-cloning approaches that clip out-of-distribution actions, the failure mode FAST-Q targets.","marker":"[15]"},{"why":"It supplies variational dropout, which FAST-Q uses to measure Q-value uncertainty and demonstrate that the balancing layer stabilizes predictions.","marker":"[16]"}],"fun_headline_variants":["Balanced state reps enable counterfactual actions in offline RL","Gradient reversal balances states for offline counterfactual actions","FAST-Q: Debiased states make offline RL explore counterfactuals","Offline RL learns from unchosen actions via balanced states","Counterfactual estimation in offline RL with balanced representations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that after gradient-reversal balancing the representation $\\Theta(\\beta(S_t))$ carries no information about which policy produced the data, so the critic's estimate equals the true counterfactual outcome; the paper assumes this invariance rather than validating it against ground-truth counterfactuals.","fun_headline_variants_meta":{"raw":{"variants":["Balanced state reps enable counterfactual actions in offline RL","Gradient reversal balances states for offline counterfactual actions","FAST-Q: Debiased states make offline RL explore counterfactuals","Offline RL learns from unchosen actions via balanced states","Counterfactual estimation in offline RL with balanced representations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1489,"prompt_tokens":1021,"completion_tokens":468,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":637,"completion_tokens_details":{"reasoning_tokens":384}},"tokens_in":637,"tokens_out":468,"duration_ms":4862,"temperature":1.0,"reasoning_tokens":384,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:05:08.554238+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a prospective experiment where players are randomly reassigned among the three policies, record the actual rewards for actions that FAST-Q would have judged counterfactual, and compare realized rewards with FAST-Q's predicted Q-values. If the prediction error is systematically larger for actions originating from non-serving policies than for actions from the serving policy, balancing has not removed the policy bias; equivalently, a held-out policy classifier that retains above-chance accuracy on the balanced representation would falsify the invariance.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the TD3+BC base actor-critic and the behavior-cloning regularization term with the recommended $\\alpha=2.5$."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It contributes the double-Q and delayed-policy-update mechanisms that FAST-Q uses to limit Q-value overestimation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies variational dropout, which FAST-Q uses to measure Q-value uncertainty and demonstrate that the balancing layer stabilizes predictions."}],"review_version":1}