{"id":"cbd5025a-85d9-44fa-afbb-a49e76d795f4","arxiv_id":"1908.04777","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Curriculum learning, room-aware action labels, and LinUCB exploration improve zero-shot performance on TextWorld cooking game families, reaching 72% and 68% of achievable points.","lead":"This paper trains agents to play never-before-seen text adventure games in the same family (cooking games from TextWorld) by using curriculum learning, map familiarization, and LinUCB exploration at test time. The best agent scores 72% and 68% on two unseen test sets, up from 14% for a random baseline and 50% to 54% for a mixed-training baseline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Curriculum vs. mixed comparison is confounded by unequal training compute: mixed uses 10M steps, curriculum up to 2M×6 tiers plus fine-tuning, so the 14-point gain may not be due to curriculum.","rationale":"I partially agree with the reader's verdict but identify a different weakest point. The reader's weakest assumption is the LinUCB linearity (Section 3.3). However, the DRRN Q-function is f(s,a)=h_s^T W h_a, which is exactly linear in h_s for a fixed action a, so the ridge-regression model is structurally compatible. The real issue with LinUCB is that the reported advantage over sampling is small (3 and 1 points) and may be within noise, and the confidence bounds are normalized with an ad-hoc 0.2 coefficient. But the more fundamental threat to the central claim is the compute mismatch in the curriculum experiment. Since the paper's first contribution and the abstract's main quantitative claim depend on curric > mixed, and the mixed baseline uses a different, likely smaller training budget, the causal interpretation is unproven. This warrants the conditional verdict: the authors should supply a matched-compute baseline or report actual step counts. I therefore recommend no change to the reader's CONDITIONAL verdict.","tokens_in":11064,"tokens_out":5611,"duration_ms":51475,"concrete_test":"Retrain the 'mixed go-room' baseline with exactly the same number of optimizer steps as the curriculum run (summing all tier stages and final fine-tuning), using the same epsilon reinitialization schedule (restart epsilon at 1 at each equivalent segment), and evaluate on Test 1 and Test 2. If mixed reaches or exceeds 64% on Test 1, the curriculum advantage is not established and the claim should be weakened. Also report significance tests (paired bootstrap or Wilcoxon) for the 3-point UCB-vs-sampling difference in Table 6.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that curriculum learning improves zero-shot performance rests on the comparison between 'curric' and 'mixed' in Table 3. Section 5.2 states that each curriculum stage decays epsilon over 'a maximum of two million steps', with six tiers plus a final fine-tuning stage, while the mixed ablation 'instead decay[s] over 10 million steps'. Thus the two conditions differ not only in the ordering of training data but also in total optimizer updates (potentially ≥12M vs. 10M) and in how epsilon is scheduled (restarted per tier vs. decayed once). Deep Q-learning results are highly sensitive to both total steps and exploration schedule, so the observed 64% vs. 50-54% gap cannot be attributed to curriculum structure without a matched-compute control. This is the most load-bearing concern because the abstract's first stated contribution is that curriculum learning leads to a better agent than learning with all games at once; if the compute confound is real, that contribution is unsupported, regardless of the LinUCB exploration details.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies zero-shot generalization in families of text-based adventure games generated by the TextWorld framework. The authors propose three techniques: curriculum learning over six difficulty tiers, conversion of instance-specific navigation knowledge into universal knowledge via random-go and go-room strategies, and LinUCB-based exploration at inference time. Experiments on 4,440 cooking games report improved task completion rates on two held-out test sets: curriculum learning over mixed training (64% vs 50–54% overall), go-room over go-cardinal, and LinUCB over greedy and sampling policies (72% vs 69% on Test 1). The central claim is that these simple human-inspired strategies improve the agent's ability to play never-before-seen games of the same theme.","tokens_in":11253,"tokens_out":4616,"duration_ms":45962,"significance":"If the results hold, the paper is a useful empirical contribution to the under-studied problem of zero-shot generalization in text-based games, and it demonstrates that simple, well-chosen inductive biases can beat both flat training and naive stochastic evaluation. The authors use a public benchmark, hold out games by type, select models on a dev set, and evaluate on genuinely unseen games, which is a strength. The paper also provides several ablations to isolate the effect of each proposed component. However, the two most load-bearing comparisons—curriculum versus mixed training, and LinUCB versus sampling—have problems that prevent the conclusions from being fully established: the curriculum comparison is confounded with training budget and epsilon schedule, and the LinUCB method rests on an unvalidated linearity assumption that may not hold for a nonlinear deep Q-network. The statistical reporting is also incomplete, with most tables lacking error bars or significance tests.","major_comments":[{"comment":"The headline comparison between 'curric' and 'mixed' is confounded with training budget and exploration schedule. §5.2 states that each curriculum stage decays epsilon 'across a maximum of two million steps' over six stages plus a final fine-tuning stage, while 'mix go-room' 'instead decay[s] over 10 million steps.' The curriculum condition therefore receives potentially up to 14M optimizer updates with epsilon reset at every stage, whereas the mixed condition receives 10M updates with a single decay. Because both total update count and epsilon schedule strongly influence deep Q-learning, the 14-point gap in Table 3 cannot be attributed solely to the ordering of training data. Please add a matched-compute control with identical total steps and epsilon schedule, or provide explicit evidence that the compute difference does not drive the result.","section":"§5.2, Tables 2–3"},{"comment":"The LinUCB evaluation procedure rests on the assumption that during evaluation Q_{s,a} = h_s^T θ_a + η_a with η_a R-sub-Gaussian, but h_s is produced by a CNN and the Q-values come from the nonlinear DRRN. The paper provides no validation of this linearity, so the confidence bounds used for action selection may be miscalibrated. Section 5.4 further introduces a normalization coefficient of 0.2 with no sensitivity analysis, leaving open the possibility that the reported LinUCB gains depend on this arbitrary constant. Since the improvement from LinUCB over sampling (72 vs 69 on Test 1) is the basis of the third contribution, please include fit diagnostics on held-out (h_s, Q) pairs, or explicitly reframe LinUCB as an unvalidated heuristic and investigate sensitivity to the 0.2 coefficient.","section":"§3.3, Table 6"},{"comment":"The main numerical claims lack statistical support. Tables 2–4 report single point estimates with no variance across training runs or evaluation episodes, and the confidence intervals in Table 6 are based on only 10 episodes per game, with per-tier differences that overlap (e.g., Test 1, tier 5: sampling 70±2 vs UCB 69±0). Without error bars or significance tests on the aggregate comparisons, the claimed superiority of curriculum over mixed training and of LinUCB over sampling is not established. Please report variance across at least three training seeds and test the significance of the overall differences.","section":"Tables 2–6"}],"minor_comments":[{"comment":"The table caption says 'Rows 1-6 show the breakdown of total scores and steps,' but the table contains only scores; please remove the reference to steps or add the step counts.","section":"§5.2, Table 3"},{"comment":"The conclusion states that bandit feedback is incorporated into 'both training and evaluation phases,' but Section 3.3 applies LinUCB only at evaluation time; the conclusion should be corrected to avoid overstating the method.","section":"§7"},{"comment":"The definition of the confidence-bound coefficient c_t is left as 'a normalization value related to V_t^a' with references to prior work; for reproducibility, please provide the explicit formula or a precise citation to the equation used.","section":"§3.3"},{"comment":"The terms 'universal knowledge' and 'instance knowledge' are used implicitly before being defined; please provide explicit definitions at first use.","section":"§3.2"},{"comment":"The row labels 'Test-1' and 'Test-2' are written with a hyphen, unlike the rest of the paper, which uses 'Test 1' and 'Test 2'; please make the notation consistent.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a legitimate empirical study on zero-shot generalization for families of TextWorld cooking games. The simple strategies — curriculum learning, rewriting go actions to room destinations, and LinUCB-style exploration at inference — do seem to improve task completion over reasonable baselines. But the headline curriculum result is confounded by unequal training compute, and the LinUCB improvement rests on an unvalidated linearity assumption. Worth serious review, but the authors need to run a matched-compute control and release code.\n\nWhat's new: most prior text-adventure work optimizes a single game; this one evaluates on never-before-seen games of a theme. That's the right setup. The tier breakdowns and the recency analysis (Table 5) are thoughtful. I believe the go-room action rewriting is a genuinely useful trick: converting instance-level direction info into universal room-transition info helps in larger maps. The zero-shot scores are well above random (72% vs 14%) and mixed-only training (50-54%).\n\nThe soft spots are real. The stress-test note is correct: Section 5.2 says curriculum decays epsilon over up to 2M steps per stage across six tiers plus fine-tuning, while 'mixed' decays over 10M steps. So the comparison does not match total optimizer steps or exploration schedule. The observed 64% vs 50-54% could partly be training compute, not curriculum ordering. The paper needs a matched-compute curriculum baseline. Without that, the abstract's first contribution is unsupported.\n\nLinUCB is the other place to push. The linearity assumption in Section 3.3 is stated but never checked. Q-values come from a nonlinear DRRN; the confidence bounds may be miscalibrated. The coefficient 0.2 is hand-chosen, and there's no sensitivity analysis. Table 6 shows some tier-level CIs overlap, and Table 2 lacks error bars. I'm not claiming the LinUCB result is false; it's just not established.\n\nOne more: the conclusion says bandit feedback helps 'both training and evaluation phases,' but LinUCB is only used at evaluation. That's a wording bug, not a fatal one.\n\nThis paper is for researchers working on generalization in RL for text games. It deserves a serious referee; the result is plausible and the benchmark setup is useful. Recommend acceptance after revision, with matched-compute control, significance/effect sizes, and a LinUCB robustness check.","headline":"A plausible empirical study on zero-shot text-game generalization; the curriculum result is confounded by training compute and LinUCB needs a validity check, but the paper deserves peer review.","tokens_in":11787,"tokens_out":2263,"would_cite":false,"duration_ms":22201,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that a deep Q-learning agent can play never-before-seen text-based cooking games, reaching 72% of possible score on one test set, by combining three human-like training strategies: curriculum learning, map…","keywords":["text-based games","deep reinforcement learning","curriculum learning","knowledge graph","LinUCB","zero-shot generalization","TextWorld","DRRN"],"falsifier":"A direct test is to take the trained DRRN agent, collect the encoded states and Q-values it produces during evaluation, and fit the linear model $Q = h_s^\\top \\theta_a + \\eta$ per action; if the residuals show systematic curvature or correlation with the state features beyond noise, the linearity assumption is violated. A behavioral falsification would be to replace LinUCB's confidence-bound exploration with pure random exploration over the same number of extra steps (e.g., 50 steps per episode), holding everything else fixed; if random exploration matches LinUCB's scores, the bandit's uncertainty-directed selection is not the cause of the improvement.","tokens_in":10830,"feed_emoji":"🍳","tokens_out":2985,"duration_ms":32310,"temperature":0.7,"pith_summary":"The paper asks whether an agent trained on a family of text-based adventure games can generalize to new, unseen instances of the same theme, rather than overfitting to a single game. It claims that three simple, human-inspired changes to a deep Q-learning recipe make this possible: training on easy games before hard ones, converting location-specific navigation knowledge into general room-type knowledge, and using a contextual bandit (LinUCB) to explore uncertain actions during evaluation. On the cooking-game test sets, the best configuration scores 72% and 68%, compared with 14% for random actions and 50–54% for training on all game types at once. The result matters because it suggests a path toward agents that can handle novel environments in open-world text tasks, closer to how people transfer cooking or navigation skills to a new kitchen.","feed_headline":"Agent learns to cook new recipes in unseen text-game houses","feed_subtitle":"Curriculum training, room-aware navigation, and bandit exploration lift zero-shot scores to 72%.","key_machinery":"The central object is the DRRN agent, which scores each action by the bilinear form $f(s,a) = h_s^\\top W h_a$, where $h_s$ is the CNN-encoded trajectory and $h_a$ is the LSTM-encoded action. Three mechanisms carry the argument: (1) a hand-designed curriculum over six game tiers that starts with single-room, one-ingredient games and progresses to twelve-room games with up to three ingredients; (2) a knowledge graph that supplies the destination room type for each navigation action, turning the instance-specific 'go east' into the universal 'go east to kitchen'; and (3) LinUCB applied at evaluation time, which solves a ridge-regression problem per action to estimate the covariance $V_a$ of seen state-action pairs and then selects the action maximizing $Q_{s,a} + c\\sqrt{h_s^\\top V_a^{-1} h_s}$. This bandit mechanism is what injects directed exploration without destructive random actions.","core_discovery":"The central claim is that deep Q-learning, originally designed for single video games, can be extended to zero-shot generalization across a family of text adventure games if the training and inference procedures are structured to separate universal knowledge from instance-specific knowledge. Concretely, the authors train a Deep Reinforcement Relevance Network (DRRN) on cooking games generated by the TextWorld engine, organizing games into six difficulty tiers and training sequentially (curriculum learning). They convert instance-level navigation actions like 'go east' into either a random direction ('go-random') or a room-destination-augmented action ('go room east to kitchen') using a knowledge graph built by random walking during early steps of each episode. At inference, instead of greedily picking the highest Q-value, they model each action's Q-value as a linear function of the encoded state and use LinUCB confidence bounds to select actions that explore rarely seen state-action pairs. The reported result is that the full recipe—curriculum, go-room augmentation, and LinUCB—achieves the highest aggregate scores on two held-out test sets of 222 unseen games each, with the gains most pronounced in the hardest twelve-room tier.","pith_inferences":["The go-room augmentation suggests a broader design principle: for any task with a spatial or structural component, making the action representation include the predicted next-state category may improve generalization, and this could be tested in navigation-oriented dialogue agents beyond text games.","The LinUCB gains, if they replicate, would indicate that the main bottleneck in zero-shot text-game play is not policy quality but confidence calibration over rarely seen states; a well-calibrated uncertainty estimate could be as valuable as more training data.","The reported recency effect—where the agent performs best on the tier most recently trained—implies a continual-learning failure mode; a natural extension would be to replay old-tier games during later training to preserve skills, which the authors only partially do with a final fine-tuning stage.","The assumption that the action set is fully known at inference limits the method to games with predefined admissible actions; future work could relax this by generating candidate actions from a language model, but the paper leaves this unexplored."],"forward_implications":["If the result holds, training on a curriculum of carefully ordered game tiers is a practical alternative to mixing all training data, improving zero-shot performance on the same family of tasks.","Representing actions by their semantic effect (destination room type) rather than their surface form ('go east') transfers better to new environments, suggesting that universal knowledge can be encoded in action representations.","Applying LinUCB-style confidence bounds at inference time, rather than during training, improves exploration in zero-shot settings and avoids the destructive random moves that damage game progress.","The method establishes a strong baseline on the TextWorld cooking competition's zero-shot evaluation, at 72% of achievable score on unseen cooking games."],"supporting_citations":[{"why":"Provides the TextWorld environment and the 4,440 cooking games used for all training and evaluation.","marker":"[Côté et al., 2018]"},{"why":"Introduces the DRRN architecture that the paper adapts for text game action scoring.","marker":"[He et al., 2016]"},{"why":"Establishes the DQN framework and replay-memory training that the paper extends to text and to families of games.","marker":"[Mnih et al., 2015]"},{"why":"Supplies the curriculum learning methodology used to order game tiers from simple to hard.","marker":"[Bengio et al., 2009]"},{"why":"Demonstrates LinUCB with DRNN-style models, providing the action-elimination precedent the paper adapts to evaluation-time exploration.","marker":"[Zahavy et al., 2018]"},{"why":"Provides the linear bandit confidence-bound construction used in the LinUCB upper bounds.","marker":"[Abbasi-yadkori et al., 2011]"},{"why":"Supplies the CNN trajectory encoder architecture and a prior text-game agent that the paper builds on and compares against.","marker":"[Yin and May, 2019]"}],"fun_headline_variants":["Learn to cook in unseen text houses: curriculum, maps, and bandits","Agent zero-shots into new text games using curricula and bandits","Three tricks for agents to cook in unseen text adventure houses","Bandits, curricula, and maps teach agents to cook in new text houses"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The LinUCB evaluation improvement rests on the assumption that the true Q-value for each action is linearly related to the encoded state plus sub-Gaussian noise, even though the actual Q-values come from a nonlinear neural network; if this linearity fails, the confidence bounds are miscalibrated and the reported exploration gains could be accidental.","fun_headline_variants_meta":{"raw":{"variants":["Learn to cook in unseen text houses: curriculum, maps, and bandits","Agent zero-shots into new text games using curricula and bandits","Three tricks for agents to cook in unseen text adventure houses","Bandits, curricula, and maps teach agents to cook in new text houses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000425,"raw_usage":{"total_tokens":2186,"prompt_tokens":959,"completion_tokens":1227,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":1162}},"tokens_in":575,"tokens_out":1227,"duration_ms":10113,"temperature":1.0,"reasoning_tokens":1162,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:32:37.023351+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test is to take the trained DRRN agent, collect the encoded states and Q-values it produces during evaluation, and fit the linear model $Q = h_s^\\top \\theta_a + \\eta$ per action; if the residuals show systematic curvature or correlation with the state features beyond noise, the linearity assumption is violated. A behavioral falsification would be to replace LinUCB's confidence-bound exploration with pure random exploration over the same number of extra steps (e.g., 50 steps per episode), holding everything else fixed; if random exploration matches LinUCB's scores, the bandit's uncertainty-directed selection is not the cause of the improvement.","supporting_citations":[{"cited_title":"Curriculum learning","cited_arxiv_id":null,"evidence_quote":"Supplies the curriculum learning methodology used to order game tiers from simple to hard."},{"cited_title":"Mankowitz, and Shie Mannor","cited_arxiv_id":null,"evidence_quote":"Demonstrates LinUCB with DRNN-style models, providing the action-elimination precedent the paper adapts to evaluation-time exploration."},{"cited_title":"Improved algorithms for lin- ear stochastic bandits","cited_arxiv_id":null,"evidence_quote":"Provides the linear bandit confidence-bound construction used in the LinUCB upper bounds."}],"review_version":1}