{"id":"0df15d16-a5a7-430c-bac8-a6856daa6842","arxiv_id":"2412.11417","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An RL opponent probes a curling decision tree for weaknesses, an LLM rewrites the tree, and the loop produces a top-ranked curling AI on the Jidi platform.","lead":"The paper proposes an automated loop in which reinforcement learning finds weaknesses in a curling game decision tree and a large language model rewrites the tree to fix them. The resulting AI ranked first among 34 entries on the Jidi curling platform, suggesting LLMs can help automate strategy improvement.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The RL component's causal role is unverified: the LLM Coder alone already enhanced the initial tree, and no LLM-only baseline or repeated evaluation is reported.","rationale":"The paper's own Figure 5 and Section V-B present direct evidence that the LLM alone adds tactics; the central novelty is the RL-guided loop. Without an ablation, the observed monotone leaderboard improvement cannot be assigned to the RL component. This is an internal attribution gap, not a disagreement with outside consensus. The available evidence—Table I and Figure 7—does not include variance or paired comparisons, so statistical significance cannot be assessed. The reader's weakest assumption about adversarial coverage is a real sub-issue, but it presupposes that the RL component is doing causal work; my proposed ablation tests that first. If the RL component is entirely superfluous, coverage is moot. If the ablation supports attribution, then the coverage test becomes the next necessary check, for example by measuring the diversity of strategies the RL policy uses against the tree. The requested additions—ablation, repeated runs, and error bars—are concrete and feasible because the code is reported as available. I therefore keep the conditional verdict: the claim is plausible but currently under-supported.","tokens_in":15098,"tokens_out":3472,"duration_ms":33653,"concrete_test":"Run a controlled ablation from the same initial tree and same LLM: (B) full RL-LLM-DT as described; (C) same critic prompt but with the lose-trajectory field Itrace removed; (D) critic prompt restricted to randomly sampled trajectories not produced by an RL-trained policy. Generate at least five independent runs per condition and evaluate every resulting tree in a round-robin tournament against the same 34 Jidi opponents with at least 1000 games per pairing. If condition C reaches the same mean score as B within one standard error, the RL debugger is not load-bearing and the paper's central claim fails; if B significantly outperforms C, the attribution is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the RL debugger's losing trajectories are what enable iterative improvement. Section V-B documents that the LLM Coder, without any RL feedback, 'subtly enhances the decision tree by incorporating a protective operation' not present in the initial tactic. Table I then reports scores 0.47/0.90/0.93 across versions, but with no LLM-only baseline, no repeated runs, no error bars, and no game count behind the Jidi leaderboard score. The observed improvement is therefore equally compatible with an LLM that is simply being re-prompted with its own previously generated code and with prior curling knowledge; the RL-identified failure trajectories may be superfluous. If the RL component is superfluous, the claimed contribution of this paper—an automatic RL-guided decision-tree refinement loop—is not demonstrated. The reader's coverage concern about Section IV-C is also valid: a single leaderboard rank does not establish that the RL adversary explored the relevant weakness space, because no coverage or diversity metric is reported. But the more fundamental missing control is the attribution check: whether RL feedback changes the LLM's output at all.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RL-LLM-DT, an iterative framework for refining decision-tree policies in a curling game. An RL agent is trained against the current decision tree to collect losing trajectories; an LLM critic analyzes these trajectories and proposes an improved tactic; an LLM coder implements the tactic as Python code. The loop terminates when the RL debugger cannot find further flaws or the LLM fails to propose an improvement. The authors report that the third iteration of the decision tree ranks first on the Jidi platform with a score of 0.93, outperforming a human-designed decision tree from their prior work (score 0.80).","tokens_in":15350,"tokens_out":4121,"duration_ms":37768,"significance":"If the causal role of the RL feedback were established, the framework would be a meaningful step toward automating decision-tree strategy improvement by combining LLM prior knowledge with targeted adversarial search. The paper is clearly structured, the code is released, and the evaluation is performed on an external platform against 34 submitted AIs, which is a strength relative to many internal-gym evaluations. The resulting final tree is more complex than the human-designed version and includes strategic distinctions such as last-throw behavior and obstacle protection. However, the current empirical evidence does not isolate the contribution of the RL feedback loop from the LLM's own prior knowledge, and the leaderboard-based claim lacks statistical support. These issues are load-bearing for the central claim that the RL-guided iterative process is what drives the improvement.","major_comments":[{"comment":"The causal role of the RL feedback is not established. Section V-B explicitly states that the LLM Coder, without any RL feedback, already added a protective operation not present in the initial tactic. Table I then shows scores of 0.47, 0.90, and 0.93 for Tree I, Tree II, and Tree III, but there is no baseline in which the LLM is re-prompted without the lose trajectories. The observed improvement is compatible with the LLM simply being re-prompted with its own previous code and the game rules. To support the central claim that the RL Debugger's losing trajectories drive the refinement, the authors should run an LLM-only control (identical prompts but omitting Itrace) and compare the generated tree's structure and leaderboard score.","section":"V-B and Table I"},{"comment":"The leaderboard scores are reported as single point values without error bars, game counts, or a description of the opponent pool and evaluation protocol. The central claim that the final tree 'ranks first among 34 curling AIs' rests on a single online submission, which could be sensitive to the particular set of opponents, game randomness, and the leaderboard's updating schedule. The authors should report multiple independent evaluations or submissions, with at least the mean and standard deviation of the score, the number of games per evaluation, and the date or version of the leaderboard used.","section":"V-A and Table I"},{"comment":"The RL Debugger's coverage of the decision tree's weakness space is unverified. The method assumes that the losing trajectories collected by the RL-trained neural policy expose the most important flaws of the current tree, but Figure 7 only shows unlabeled win-rate curves, with no axes labels, no game counts, and no indication of how many distinct losing states or strategies were encountered. Without a coverage or diversity metric for the trajectories, the termination condition 'RL can't find any flaw' (Section IV-A) is not verifiable, and the final tree's robustness against a broad set of opponents is not established.","section":"IV-C and Figure 7"},{"comment":"The stopping criterion is under-specified and the final iteration is not fully reported. Section V-D states that the loop terminates when the LLM Critic fails to improve the tree, and Figure 10 shows that Tree 3 and Tree 4 are semantically identical except for comments. However, Table I does not include a score for Tree 4, and the paper does not explain why Tree III, rather than a later version, is the reported final result. The stopping rule also lacks a performance-regression check: the framework stops when the LLM cannot generate different code, but it does not verify that the new tree is at least as strong as the previous one before proceeding.","section":"V-D and Figure 10"}],"minor_comments":[{"comment":"The winning-rate figure has no labeled axes, no legend, and no explicit indication of the x-axis unit (episodes, steps, or wall-clock time). The three curves presumably correspond to the three tree versions, but this should be stated in the caption and the figure itself.","section":"Figure 7"},{"comment":"The heading 'Markov Decisioin Process' contains a typo; it should read 'Markov Decision Process.'","section":"III-B1"},{"comment":"The expectation in Eq. (2) is ambiguous because the distribution over the initial state s0 is not specified, and the notation in Eq. (3) for J(π) does not match the state-dependent definition in Eq. (2). Clarify that Eq. (2) conditions on s0 = s and that Eq. (3) averages over an initial-state distribution.","section":"III-B1, Eq. (2)"},{"comment":"The prompt text contains grammatical errors, for example 'The first game end begins' and 'Please summarize why the code cause the result.' These should be corrected for clarity.","section":"Figure 8"},{"comment":"The code uses the threshold `significant_distance` without defining its value or explaining how it is chosen; this parameter should be documented in Section IV or an appendix for reproducibility.","section":"Figure 10"},{"comment":"The abstract and conclusion repeat 'ranks first among 34 curling AIs' without specifying the evaluation date or the leaderboard version; a timestamp or archived snapshot would make the claim reproducible.","section":"V-A"}],"recommendation":"major_revision","confidential_remarks":"The paper reads more like a workshop or conference presentation in its current form. The framework is interesting and the external evaluation is a positive feature, but the missing LLM-only ablation and the lack of statistical support for the leaderboard claim are the key barriers to publication in a serious journal. I would encourage the authors to run the suggested control and repeated evaluations; if the LLM-only baseline achieves similar scores, the paper's contribution would need to be reframed around the full pipeline's reliability rather than the necessity of the RL component. The comparison against their own prior human-designed tree is acceptable but should be positioned as a baseline from their own earlier work, not as an external human expert benchmark."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: the core idea is worth a glance, but the central claim is not yet supported. The paper describes a loop where RL trains an opponent to beat a decision tree, the LLM critic reads the losing trajectories and proposes a better tree, and the LLM coder writes the code. On the Jidi curling leaderboard, the tree climbs from score 0.47 to 0.93 and rank 1 among 34. If the loop really works, it's a nice automated alternative to human strategy tuning. But the evidence doesn't show that the RL feedback is doing the work.\n\nWhat's new: prior work [9] used the same RL-plus-human refinement on this game; the new bit is using an LLM to do the critique and code generation. The framework is clearly described, and the authors honestly report that the loop terminated when the LLM produced semantically identical code for Tree 3 and Tree 4, not because RL failed to find flaws. They also release the code and compare against their own human-designed tree, which scores lower (0.80).\n\nThe soft spots are real. Section V-B shows the LLM Coder already enhanced the initial tree by adding a protective operation that wasn't in the tactic description. That means the LLM alone, without any RL feedback, improves the strategy. The paper reports no LLM-only baseline: no run where the same LLM is re-prompted with its own previous code and maybe a simple \"make it better\" instruction, no repeated runs, no error bars, no game counts behind the leaderboard scores. So the 0.47 to 0.93 jump is equally compatible with an LLM that's just getting better at curling through re-prompting. The RL debugger's causal role is unverified.\n\nThe coverage concern in the stress-test note is valid but secondary. The RL policy could be exploiting one narrow weakness, and there's no diversity metric or evidence that the collected trajectories span the important failure modes. The authors only report that the win rate stabilizes during training, which doesn't tell you what the policy learned.\n\nThe paper also leaves out the PPO hyperparameters, action discretization resolution, the 'significant_distance' threshold, and how many games the Jidi evaluation used. Figure 7 is unlabeled in the rendering. These are fixable, but as submitted the empirical foundation is thin.\n\nMy take: this is a plausible workshop-to-conference idea that hasn't been stress-tested. The missing LLM-only ablation is the thing to demand. If the authors add that, plus repeated runs with error bars and a note on RL trajectory coverage, the claim might hold. As it stands, I wouldn't cite it for the RL-guided aspect, but I'd send it to a serious referee with major-revision expectations, because the framework is coherent and the external leaderboard gives it some grounding.","headline":"A plausible RL+LLM decision-tree refinement loop, but the paper never shows the RL feedback is what drives the gains.","tokens_in":15871,"tokens_out":3661,"would_cite":false,"duration_ms":31665,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Repeatedly training an RL opponent to beat a decision tree and having an LLM repair the tree's losing plays can produce a top-ranked, interpretable game strategy without human intervention, as demonstrated in curling.","keywords":["decision tree","large language models","reinforcement learning","game AI","curling","strategy refinement","interpretability","PPO"],"falsifier":"Evaluate the final tree against a set of fresh RL opponents trained with different random seeds, reward functions, or architectures: if any of them beats the tree at a rate well above the near-50% level that would indicate the loop truly closed, the claim that the iterative process converged to a robust strategy is undercut. A second check is whether the RL debugger's final trained policy is actually unable to beat the tree -- if the loop terminated because the LLM stopped proposing changes rather than because the adversary failed, the stopping rule did not do what it claims.","tokens_in":14920,"feed_emoji":"🥌","tokens_out":5416,"duration_ms":44648,"temperature":0.7,"pith_summary":"This paper claims that a fully automatic loop can strengthen an interpretable decision-tree strategy in a two-player zero-sum game, using no human expert input after the first rough draft. The loop works by training a reinforcement-learning agent to beat the current tree, feeding the recorded losing games to a large language model, and having the model propose and code an improved tree. The authors demonstrate the loop on a curling game, where three refinement cycles produce a decision tree that ranks first among 34 submitted AIs and scores 0.93 against unseen opponents, ahead of a human-designed tree's 0.80. The result matters because it suggests that LLMs can substitute for human strategists in the iterative \"find the bug, fix the strategy\" part of game AI development.","feed_headline":"LLM and RL loop auto-upgrades a curling bot to rank first","feed_subtitle":"No human expert needed: three automated cycles turn a simple tree into the top scorer among 34 curling AIs.","key_machinery":"The load-bearing mechanism is the RL-LLM-DT loop, composed of three modules: the LLM Coder, which turns a tactic description into executable decision-tree Python code; the RL Debugger, which trains a distributed-PPO neural policy as an adversary against that code and records the tree's losing games; and the LLM Critic, which receives the game rules, the tactic, the code, and the losing trajectories and returns a diagnosed, improved tactic. The loop works because the RL agent is a targeted flaw detector -- it searches specifically for ways to beat the current tree -- and the LLM is a strategy synthesizer that converts those concrete failure traces into new conditional rules rather than just tweaking parameters. The paper's key evidence that the mechanism is doing the work is the win-rate curve in Figure 7, where later tree versions require noticeably longer RL training before a neural model can beat them.","core_discovery":"The central claim is that a decision tree can be iteratively refined into a top-performing, interpretable game policy by alternating two roles: a reinforcement-learning \"debugger\" that trains a neural policy specifically to defeat the current tree and collect the trajectories where the tree loses, and a large language model \"critic\" that reads those trajectories, diagnoses the tactical flaws, and writes out an improved tree as executable Python code. The paper calls this loop RL-LLM-DT, and reports that in the Jidi curling environment each cycle makes the tree harder for a freshly trained RL opponent to beat, until the third generated tree reaches rank 1 among 34 AIs and a platform evaluation score of 0.93, surpassing the human-designed tree (0.80) from the authors' earlier work. The termination condition is either that the RL debugger can no longer find a flaw or that the LLM critic can no longer propose a meaningfully different improvement; in the reported runs the loop stopped when the LLM produced only cosmetically different code.","pith_inferences":["A natural stress test is whether the RL Debugger's coverage determines the ceiling: if the trained opponent exploits only a narrow style of play, the LLM Critic is blind to other weaknesses, so the final tree's robustness to diverse unseen opponents is not guaranteed by the rank-1 result alone.","The method might transfer to non-game settings where policies are rule-based and failure traces are available, such as robot control or automated negotiation, with the LLM proposing condition changes rather than full code rewrites.","One testable prediction is that the loop's improvement rate should degrade if the RL opponent's exploration is restricted (fewer seeds, less entropy), because the critic would receive fewer distinct failure modes to analyze.","The comparison to the human-designed tree is suggestive but not controlled: the human tree came from earlier work with a different starting point, so a fair test would run the same loop seeded with the human tree and see whether it also improves."],"forward_implications":["If the loop is correct, interpretable decision-tree policies can reach competitive levels in strategy games without a human expert iterating on them manually.","The same loop could be applied to other two-player zero-sum games, such as chess, card games, or StarCraft, by swapping in the relevant environment rules and interface definitions.","The termination condition (RL finds no flaw or LLM cannot improve) gives a practical stopping rule for automatic strategy refinement, and the authors observe that it fired when the LLM Critic's suggestions became semantically empty.","Because the final artifact is a decision tree, the resulting AI remains inspectable and explainable, unlike the neural opponent used to probe it.","The approach suggests that RL evaluation and LLM code generation are complementary: RL supplies concrete counterexamples, LLM supplies the repair."],"supporting_citations":[{"why":"The authors' former work on RL-revised decision trees; supplies the human-designed baseline tree and the original idea of training an RL agent to expose a tree's flaws.","marker":"[9]"},{"why":"Proximal policy optimization, the RL algorithm the RL Debugger uses in distributed form to train the adversarial neural policy.","marker":"[31]"},{"why":"Eureka, a prior system where a coding LLM improves policies through iterative code generation; motivates the LLM Coder and Critic design.","marker":"[29]"},{"why":"CodeAct, which shows that executable Python code is an effective action space for LLM agents; supports the choice to have the LLM output decision-tree code.","marker":"[28]"},{"why":"Emergent abilities of large language models; grounds the assumption that an LLM can perform the trajectory analysis and code generation the loop requires.","marker":"[10]"}],"fun_headline_variants":["RL finds flaws, LLM fixes them: curling bot tops leaderboard","No human tuning: RL-LLM loop refines tree to rank 1","Curling AI self-improves via RL-LLM to beat 34 rivals","Automated decision tree upgrade: RL-LLM makes champion","RL-LLM loop turns tree into top curling AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole improvement loop assumes that the RL-trained opponent is a broad enough adversary that the losing games it produces expose the decision tree's most important weaknesses; if it exploits only a narrow style of play, the LLM critic never sees the other failures and the final tree's robustness is unverified.","fun_headline_variants_meta":{"raw":{"variants":["RL finds flaws, LLM fixes them: curling bot tops leaderboard","No human tuning: RL-LLM loop refines tree to rank 1","Curling AI self-improves via RL-LLM to beat 34 rivals","Automated decision tree upgrade: RL-LLM makes champion","RL-LLM loop turns tree into top curling AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000648,"raw_usage":{"total_tokens":3037,"prompt_tokens":1072,"completion_tokens":1965,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":688,"completion_tokens_details":{"reasoning_tokens":1870}},"tokens_in":688,"tokens_out":1965,"duration_ms":11934,"temperature":1.0,"reasoning_tokens":1870,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:57:01.076333+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the final tree against a set of fresh RL opponents trained with different random seeds, reward functions, or architectures: if any of them beats the tree at a rate well above the near-50% level that would indicate the loop truly closed, the claim that the iterative process converged to a robust strategy is undercut. A second check is whether the RL debugger's final trained policy is actually unable to beat the tree -- if the loop terminated because the LLM stopped proposing changes rather than because the adversary failed, the stopping rule did not do what it claims.","supporting_citations":[{"cited_title":"Mastering curling with rl-revised decision tree,","cited_arxiv_id":null,"evidence_quote":"The authors' former work on RL-revised decision trees; supplies the human-designed baseline tree and the original idea of training an RL agent to expose a tree's flaws."}],"review_version":1}