{"id":"8df94a9d-8844-46c1-aeb4-a93d7604e237","arxiv_id":"2607.08984","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":5,"one_line_summary":"Vanilla AlphaZero fails to preserve oracle-optimal trajectories in Connect Four and Chomp; AZAL with oracle policy supervision substantially raises oracle consistency, reaching perfect full-game play on Chomp 10×11.","lead":"AlphaZero-style agents play Connect Four and Chomp strongly but miss exact optimal lines; an auxiliary oracle policy loss (AZAL) largely closes the gap on Chomp and partially on Connect Four. The work is a diagnostic for when search-improved self-play fails to recover sparse long-horizon structure.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the reader's already-flagged checkpoint/search-budget caveat.","rationale":"The reader's weakest_assumption correctly isolates the softest point: representativeness of last-iteration checkpoints under a modest fixed search budget and three seeds when attributing residual non-oracle play to weak search-learning signal. The manuscript itself flags this in §7 and leaves checkpoint sensitivity and search-budget variation for future work. Multi-seed full-game and sampled-state tables, the multi-frame ablation, and the square-board sanity checks give independent support for the descriptive claim that vanilla AZ is strong but imperfect and that AZAL improves oracle consistency. No hidden algebraic assumption, coding inconsistency in the oracle definitions, or contradictory table entry appears that would invalidate the reported numbers. Therefore the CONDITIONAL verdict with high confidence on the measurements remains appropriate; no further downgrade or upgrade is warranted from this pass.","tokens_in":18936,"tokens_out":530,"duration_ms":15915,"concrete_test":"Re-evaluate the three final checkpoints of each Chomp 10×11 and Connect Four seed under doubled MCTS budget (1600/2000 sims) and also under the best validation-oracle checkpoint rather than last-iteration only; if Perfect/Match in Table 1 change by more than ~10 percentage points or FirstFail shifts substantially, the attribution of residual gaps to supervision strength rather than under-search/under-training would need revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest claim is an empirical diagnostic: under a fixed self-play+MCTS pipeline, vanilla AlphaZero produces strong but not oracle-consistent play on Connect Four and rectangular Chomp, while AZAL substantially raises oracle-match rates (perfect 60/60 full-game traces on Chomp 10×11; incomplete on 9×10 and Connect Four). Tables 1–2, the multi-seed traces, and the multi-frame negative control support that measurement. The causal gloss that residual gaps are primarily due to weak search-derived supervision is already caveated in §7 and Table 5 (last-iteration checkpoints, fixed 800–1000 sims, three seeds, no search-budget or checkpoint ablations). That is a real limitation on generality, but it does not undermine the reported measurements or the comparative AZAL result. I do not find an additional load-bearing internal inconsistency or measurement flaw that would overturn the claim as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper studies the gap between strong and perfect play for AlphaZero-style agents in two oracle-evaluable domains: Connect Four (partisan, exact game-theoretic scores via Pons’s solver) and Chomp (impartial, Grundy-number structure). Under a unified self-play+MCTS pipeline it compares vanilla AlphaZero, a multi-frame representation ablation (Chomp only), and AZAL, which adds a sparse oracle-derived auxiliary policy loss while leaving search and value targets unchanged. Multi-seed full-game traces (60 traces, 3 seeds) and random-start sampled-state evaluations show that vanilla AlphaZero produces strong but not oracle-consistent play—failing to preserve optimal early continuations in Connect Four and the g≠0→g=0 invariant on rectangular Chomp—while multi-frame inputs do not close the Chomp gap. AZAL substantially raises oracle-match rates, reaching perfect full-game consistency on Chomp 10×11 (60/60), high but incomplete consistency on Chomp 9×10, and improved but still imperfect play on Connect Four.","tokens_in":19188,"tokens_out":1141,"duration_ms":28853,"significance":"If the reported measurements hold, the paper supplies a clear, falsifiable diagnostic that superhuman-style self-play need not recover exact optimality in sparsely structured games, and that oracle-evaluable domains (Connect Four, Chomp) are useful for separating those notions move-by-move. Strengths include external oracles independent of the network, multi-seed aggregate tables (Tables 1–2, 4), annotated traces, a multi-frame negative control, losing-but-won diagnostics (Table 3), and publicly linked code. The contribution is primarily empirical and methodological rather than a new general algorithm; its value is in quantifying a known concern (strong vs perfect play) under a controlled pipeline and showing that auxiliary oracle supervision helps unevenly across domains. That is a solid, publishable diagnostic contribution for the RL/game-AI literature, provided causal language about the search-learning bottleneck remains carefully scoped.","major_comments":[{"comment":"§7 and Table 5: Contribution 3 and the cross-domain discussion (§6.4) treat weak search-derived supervision as a plausible bottleneck for residual imperfect play. That interpretation rests on last-iteration checkpoints under a fixed modest MCTS budget (800–1000 sims), fixed residual architecture, λ_aux=1.0, and three training seeds, with no search-budget, architecture, λ_aux, or checkpoint-selection ablations. The manuscript already flags this limitation; either add a minimal ablation (e.g., higher-budget or best-of-checkpoints evaluation on one board) or further restrict the claim so residual gaps are reported strictly as fixed-pipeline diagnostics rather than evidence that the search-learning signal is the primary cause.","section":null},{"comment":"§5–§6 and Tables 1–2: “Strong play” is asserted for vanilla AlphaZero but is not quantified against fixed external opponents or win-rate baselines—only via training curves, square-board sanity checks, and oracle-match rates. Because Table 3 shows non-trivial losing-but-won rates (opponent failures), self-play strength is not fully characterized. A short head-to-head or fixed-opponent win-rate summary for vanilla vs AZAL would make the strong-vs-perfect distinction load-bearing rather than largely qualitative.","section":null}],"minor_comments":[{"comment":"Author names and several section headings in the source show spacing artifacts (“T ejas Ram”, “V anilla AlphaZero”, “T race-level metrics”); clean these in the camera-ready text.","section":null},{"comment":"Figure 3 panels are dense; ensure filled vs hollow markers and player identity remain legible in print, and state explicitly in the caption that markers are greedy deterministic rollouts, not the multi-seed aggregates of Table 1.","section":null},{"comment":"Notation for the auxiliary term varies (L_aux, p_move_loss, pmove); standardize to one symbol in the main text and appendix.","section":null},{"comment":"Table 4 N/A entries for Chomp second-player under AZAL 10×11 are correct given g=0 labeling, but a one-sentence reminder in the table caption would help readers unfamiliar with impartial games.","section":null},{"comment":"Related work cites Zhou & Riis and Riis (multi-frame) appropriately; a brief note on how AZAL differs from pure imitation or supervised fine-tuning of an oracle policy would further clarify novelty.","section":null},{"comment":"Appendix E code link is anonymous; ensure the final version points to a stable repository with seeds and evaluation scripts matching Tables 1–2.","section":null}],"recommendation":"minor_revision","confidential_remarks":"Fit is appropriate for a solid empirical/diagnostic ML or game-AI venue. Novelty is incremental (AlphaZero + auxiliary oracle loss + multi-frame ablation) but the measurement design is careful and the limitations are largely self-reported. I would not reject on scope; minor revision is enough if the authors tighten causal language and add a small strength or budget check. No integrity concerns from the manuscript as presented."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful takeaway is simple. Under one self-play+MCTS setup, vanilla AlphaZero plays well on Connect Four and Chomp but does not stay on oracle-optimal trajectories; multi-frame history does not fix rectangular Chomp; a light oracle policy auxiliary (AZAL) raises match rates a lot, including 60/60 perfect full-game traces on Chomp 10×11, while still leaving residual gaps on 9×10 and Connect Four.\n\nWhat is new is the unified move-level protocol against external oracles (Pons solver; recursive Grundy), the multi-seed aggregates (60 traces, three seeds), sampled-state tables, FirstFail/Chain metrics, the multi-frame negative control, and the AZAL comparison. That package is cleaner than another win-rate plot. The square-board sanity checks, player-specific match rates, losing-but-won diagnostics, and annotated traces make the strong-vs-perfect distinction concrete rather than rhetorical. Code is linked; hyperparameters are tabled. Citations sit in the right line (Silver, Anthony, Zhou & Riis, Riis, Trudeau & Bowling) without pretending to invent the AlphaZero loop.\n\nSoft spots are real but already mostly owned. Evaluation is last-iteration only, search budget is fixed (800–1000 sims), boards are small, and AZAL uses the oracle by design—so the paper measures recovery under stronger supervision, not pure self-play discovery of perfect play. The gloss that residual gaps are mainly “weak search-learning signal” is plausible from the oscillations and target-corruption table, but not isolated; §7 says so. Multi-frame is Chomp-only. None of that overturns the reported numbers.\n\nThis is for people who care about exact optimality diagnostics in combinatorial RL, not for someone hunting a general AlphaZero theory fix. I would send it to peer review: the measurements are sharp enough to deserve referee time, with the usual ask for search/checkpoint ablations and clearer scope. Worth engaging if that is your lane.","headline":"Solid empirical diagnostic: vanilla AlphaZero is strong but not oracle-consistent on Connect Four and rectangular Chomp; AZAL closes much of the gap under a fixed pipeline, with the main soft spot being causal attribution rather than the measurements.","tokens_in":19796,"tokens_out":538,"would_cite":true,"duration_ms":5274,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Vanilla AlphaZero plays strongly but fails to preserve perfect-play trajectories; oracle auxiliary supervision closes much of the gap.","keywords":["AlphaZero","perfect play","Connect Four","Chomp","auxiliary loss","Monte Carlo Tree Search","oracle consistency","impartial games"],"falsifier":"Retrain vanilla and AZAL agents with much larger MCTS budgets and evaluate multiple intermediate checkpoints on the same multi-seed full-game and sampled-state oracle metrics; if vanilla then matches AZAL consistency (or AZAL’s advantage disappears), weak search-learning signal is not the primary bottleneck.","tokens_in":19824,"feed_emoji":"♟️","tokens_out":987,"duration_ms":28776,"temperature":0.7,"pith_summary":"AlphaZero-style agents can reach high practical performance through self-play guided by tree search, yet strong play is not the same as choosing an optimal move at every state. This paper measures that gap on two games where exact answers are known: Connect Four, with full game-theoretic scores, and Chomp, whose winning play is defined by restoring a zero Grundy number. Under a shared self-play and search pipeline, vanilla AlphaZero produces solid policies but repeatedly leaves the optimal line—losing early value control in Connect Four and failing to keep the g=0 invariant on rectangular Chomp boards. Stacking recent board frames does not fix the Chomp failure. Adding a sparse oracle-derived policy loss (AZAL) sharply raises oracle consistency on multi-seed full-game traces and random sampled states, reaching perfect full-game consistency on Chomp 10×11, high but incomplete consistency on Chomp 9×10, and improved but still imperfect play on Connect Four. The result matters for anyone who needs to know whether search-based learning recovered exact structure, not merely good win rates.","feed_headline":"AlphaZero plays strongly but misses perfect trajectories","feed_subtitle":"Oracle checks on Connect Four and Chomp show the gap; sparse auxiliary loss shrinks it, sometimes to full-game perfection.","key_machinery":"AlphaZero Auxiliary Loss (AZAL): the usual policy-plus-value objective is augmented by a cross-entropy term against a softened distribution over oracle-optimal legal moves. Self-play, MCTS, and value targets stay the same; only the training signal is strengthened so the policy is biased toward invariant-preserving or value-preserving actions.","core_discovery":"Vanilla AlphaZero learns strong self-play policies in Connect Four and Chomp without routinely recovering oracle-consistent play: it fails to maintain the optimal line in Connect Four and fails to consistently restore the g=0 invariant in Chomp. Multi-frame inputs alone do not remove the rectangular-Chomp gap. AZAL, which adds sparse oracle-derived policy supervision while leaving self-play and search unchanged, substantially improves oracle consistency across multi-seed full-game traces and sampled states—perfect on Chomp 10×11 full games, high but incomplete on Chomp 9×10, and improved but imperfect on Connect Four.","pith_inferences":["Similar early-error gaps may appear in other sparse-reward planning settings where “good enough” self-play masks rare decisive mistakes.","AZAL’s drop from full-game perfection to imperfect sampled-state match suggests auxiliary labels should cover off-trajectory states, not only opening rollouts.","Systematically varying search budget, λ_aux, and checkpoint selection would separate under-search from true supervision weakness.","Invariant-style constraints that do not require a full exact solver would extend the same idea to games where oracles do not scale."],"forward_implications":["Win rates and smooth training curves can hide early trajectory failures that only move-by-move oracle checks reveal.","Short multi-frame history is not a general fix when a game’s structure does not expose simple local invariants.","Sparse oracle policy labels recover impartial invariant play more readily than deep partisan tactical play.","Standard search-improved self-play targets can be too weak to lock in sparse long-horizon structure required by perfect play.","Closing the remaining gap may need search or training changes that do not depend on an external oracle."],"fun_headline_variants":["AlphaZero strong yet misses perfect trajectories in Connect Four and Chomp","Vanilla AlphaZero fails exact optimal lines; AZAL lifts oracle consistency","Sparse auxiliary loss shrinks AlphaZero's gap to oracle-perfect play","AlphaZero restores g=0 poorly in Chomp; AZAL nears full-game perfection","AZAL boosts AlphaZero oracle match without changing self-play or search"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The residual gap is blamed mainly on weak search-derived supervision, treating last-iteration checkpoints under a fixed modest search budget, fixed network, and three seeds as representative of what the standard pipeline can achieve.","fun_headline_variants_meta":{"raw":{"variants":["AlphaZero strong yet misses perfect trajectories in Connect Four and Chomp","Vanilla AlphaZero fails exact optimal lines; AZAL lifts oracle consistency","Sparse auxiliary loss shrinks AlphaZero's gap to oracle-perfect play","AlphaZero restores g=0 poorly in Chomp; AZAL nears full-game perfection","AZAL boosts AlphaZero oracle match without changing self-play or search"]},"model":"grok-4.5","effort":"low","cost_usd":0.005844,"raw_usage":{"total_tokens":1606,"prompt_tokens":852,"num_sources_used":0,"completion_tokens":102,"cost_in_usd_ticks":58440000,"prompt_tokens_details":{"text_tokens":852,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":652,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":852,"tokens_out":102,"duration_ms":7100,"temperature":1.0,"reasoning_tokens":652,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T01:13:27.664055+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Retrain vanilla and AZAL agents with much larger MCTS budgets and evaluate multiple intermediate checkpoints on the same multi-seed full-game and sampled-state oracle metrics; if vanilla then matches AZAL consistency (or AZAL’s advantage disappears), weak search-learning signal is not the primary bottleneck.","supporting_citations":[],"review_version":1}