{"id":"d3bf7ffe-a5c1-4356-85b0-1f3ca13666db","arxiv_id":"2607.12924","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A DQN with Datalog action pruning and projected gradient ascent over parameters beats PAMDP baselines in return and sample efficiency.","lead":"KGRL is a reinforcement-learning method that consults a Datalog rule base at every step to prune impossible actions and constrain numeric parameters, then uses gradient ascent on the value network to refine the chosen parameter. On four standard PAMDP benchmarks it reports better sample efficiency and final return than five baselines, although handcrafted rules encode much of the task knowledge.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The handcrafted Datalog rules in each benchmark encode near-optimal action/parameter restrictions; the reported gains may reflect the knowledge content, not the KGRL algorithm itself.","rationale":"The reader's weakest assumption is that the handcrafted Datalog rules are correct and prune suboptimal decisions without pruning optimal ones. My analysis confirms this is the load-bearing point: the benchmarks are crafted so that simple, near-optimal rules are available, and the ablations show that the knowledge base is the dominant contributor to the performance difference. The paper is transparent about this and explicitly lists the limitation in §6, but the empirical evaluation does not isolate the algorithm's contribution from the knowledge's contribution. A comparison against a baseline also equipped with the same knowledge masks would directly test whether the proposed gradient-guided parameter refinement loop adds value beyond the handcrafted rules. Since the reader already reached a CONDITIONAL verdict and identified the same concern, there is no need to change the verdict; the concern reinforces the need for additional experiments and more tempered claims.","tokens_in":15715,"tokens_out":8170,"duration_ms":85701,"concrete_test":"Retrain a DQN baseline with the same Datalog-derived action mask and feasible-parameter intervals applied as hard constraints during both training and evaluation, but with a one-shot parameter estimator (e.g., the mean of the feasible interval) instead of PARAMOPT. Compare its AULC against full KGRL in Table 1. If this knowledge-masked DQN attains AULC comparable to or better than KGRL, the reported gains are due to the handcrafted rules, not the gradient-guided parameter refinement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim (KGRL outperforms all baselines) rests on a comparison in which KGRL is given handcrafted, per-benchmark Datalog rules that in several cases encode near-optimal decision rules. In HardMoveX (§5.1) the rules 'prune actions that point away from the goal in a larger angle than 90°, and cap parameters based on the distance to the goal'; in HardGoal they 'prune ... any action that is blocked by the keeper.' These are strong, correct priors that the five baselines do not receive. Table 2 shows the knowledge base prunes ~50% of actions in HardMoveX and ~34% in HardGoal, and the ablations in Table 1 show that removing the knowledge base at evaluation time drops AULC dramatically (e.g., HardMove4 from -1.64 to -4.33). This indicates the AULC advantage is largely attributable to the injected rules, not to KGRL's learning or parameter-refinement mechanics. The paper acknowledges in §6 that 'wrong rules can harm the performance' and states that KGRL can operate with learned knowledge bases, but it provides no experiment with learned, noisy, or imperfect rules; nor does it compare against a baseline that receives the same knowledge. Consequently, the load-bearing condition — that the encoded knowledge is incomplete yet correct, and that the algorithm's contribution is separable — is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Knowledge- and Gradient-Guided Reinforcement Learning (KGRL), a DQN extension for Parametrized Action MDPs. KGRL evaluates a handcrafted Datalog knowledge base at each state to prune non-applicable actions and constrain feasible parameter regions, and uses PARAMOPT, a projected gradient-ascent loop over the Q-function, to refine parameter estimates. The authors formalize knowledge-constrained PAMDPs, provide an algorithm with explanation traces, and evaluate on HardMove4/8/10, CatchPoint, HardGoal, and Platform against QPAMDP, PA-DDPG, P-DQN, MP-DQN, and HyAR. They report that KGRL achieves the highest average AULC across all domains and seeds, and that ablations show the knowledge base matters at deployment while gradient guidance has mixed or negative effects during training.","tokens_in":16005,"tokens_out":4219,"duration_ms":45723,"significance":"If the empirical claims are established, the paper contributes a useful neuro-symbolic extension to PAMDP RL: a formal integration of incomplete Datalog knowledge with action pruning and parameter constraints, a practical gradient-based parameter refinement, public code, and a demonstration of local procedural explanation traces. The experimental design (eight seeds, 16 evaluation runs, standard benchmarks, ablations) is a strength. However, the central claim that KGRL outperforms all baselines is not yet cleanly separated from the handcrafted, near-optimal knowledge injected into KGRL, and the paper's own ablations undermine the training-time benefit of the gradient component. These issues must be addressed before the headline claims are acceptable.","major_comments":[{"comment":"RQ2 asks whether gradient-guided parameter refinement increases sample efficiency and performance compared with one-shot estimators. This is contradicted by the paper's own Table 1: KGRL (train nogg) outperforms or matches full KGRL on HardMove4 (-1.43 vs -1.64), HardMove8 (-1.34 vs -1.44), and CatchPoint (-1.69 vs -1.70), and is essentially tied on HardGoal and Platform. The Discussion (§6) concedes that \"integrating the gradient-guidance during training has a negative effect.\" Since the Abstract and RQ2 claim a training-time sample-efficiency gain, the current evidence supports at most a deployment-time benefit. Please reconcile the claims and clarify whether the nogg ablation also removes PARAMOPT from the target computation in Alg. 1 lines 18-21.","section":"§1 RQ2; §5.2 Table 1; §6"},{"comment":"The handcrafted knowledge bases encode strong, near-optimal task knowledge: HardMoveX rules prune actions pointing away from the goal by >90° and cap parameters by distance; HardGoal rules prune segments blocked by the keeper. Table 2 shows ~49% action pruning in HardMoveX, and Table 1 shows that removing the knowledge base at evaluation degrades AULC dramatically (e.g., HardMove4 from -1.64 to -4.33). No baseline receives equivalent knowledge, and, despite §6 claiming that KGRL can operate with learned knowledge bases, no experiment with learned, noisy, or imperfect rules is presented. The headline advantage may therefore reflect the injected rules rather than the KGRL mechanism itself. Please add baselines that receive the same action/parameter masks, or a sensitivity study over corrupted/learned rules, to establish the separable contribution of the algorithm.","section":"§5.1; Table 2; Table 1; §6"},{"comment":"The AULC comparison reports only means ± one standard deviation over eight seeds, with no significance tests, paired comparisons, or confidence intervals. In the Platform row, the best baseline MP-DQN (0.19±0.06) overlaps with KGRL (0.22±0.02), and similar overlap may exist for other rows. The claim in §5.2 that KGRL \"consistently achieves the highest average AULC\" needs statistical support across the eight seeds, or at least a clear statement of effect sizes and variability.","section":"§5.2; Table 1"}],"minor_comments":[{"comment":"Eq. (15) writes y = r + γ(1-d') arg max_{a'} max_ψ Q; the target should be the maximum value, not an arg max action. Algorithm 1 line 21 is correct, so this appears to be a typo in the displayed equation.","section":"§4.1, Eq. (15)"},{"comment":"Typo: \"algortihm\" should be \"algorithm.\"","section":"§5.2"},{"comment":"The learning curves show only HardMove4, CatchPoint, HardGoal, and Platform; adding HardMove8 and HardMove10 would make the cross-domain claim fully visible.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: this is a solid, honest paper with a real but narrow contribution. KGRL combines Datalog-based action pruning and parameter constraints with projected gradient ascent on the Q-function for PAMDPs. That combination is new relative to the cited literature, and the explanation-trace side effect is a nice bonus. The empirical setup is careful: eight seeds, 16 evaluation runs per checkpoint, standard benchmarks, ablations, and a link to code. Full KGRL does beat the five baselines on AULC across all domains.\n\nThe soft spots are real. The ablations undercut the headline claim about gradient guidance: on HardMove4, HardMove8 and CatchPoint, removing gradient guidance during training improves or matches full KGRL. The paper itself admits this in §6, saying gradient guidance during training has a negative effect. So the abstract's 'gradient-guided parameter refinement' contribution is more about deployment-time optimization, not training efficiency. That should be stated more carefully.\n\nThe bigger issue is the knowledge base. The handcrafted rules for each benchmark encode strong near-optimal heuristics: pruning actions pointing more than 90° away from the goal, capping parameters by distance to goal, pruning blocked shots. Table 2 shows 49% action pruning in HardMove and 34% in HardGoal. Ablating the knowledge base at evaluation collapses performance (-1.64 to -4.33 on HardMove4). So the comparison against baselines measures human-provided domain knowledge, not just the algorithm. The paper acknowledges this risk and mentions learned knowledge bases as future work, but provides no experiment with noisy or learned rules, nor an ablation where baselines receive the same knowledge. That leaves the load-bearing condition -- incomplete but correct knowledge -- only partially established.\n\nMinor notes: no statistical significance tests beyond standard deviations; hyperparameters and rule sets should be released with a version hash; the PARAMOPT non-convexity caveat is acknowledged.\n\nWho this is for: researchers working on neuro-symbolic RL, hybrid action spaces, or explainable RL. They'd get a useful baseline and a clearly stated open problem. Despite the overclaims, the paper deserves a serious referee -- strict but not dismissive. I'd recommend major revision, not rejection.","headline":"Useful neuro-symbolic RL for parametrized action spaces, but the gains largely come from handcrafted per-domain rules; the gradient-guided component's benefit is not supported by their own ablations.","tokens_in":16510,"tokens_out":1767,"would_cite":true,"duration_ms":19313,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By pruning the decision space with a Datalog knowledge base and refining parameters with projected gradient ascent on the Q-function, KGRL achieves higher returns and better sample efficiency than five PAMDP baselines.","keywords":["reinforcement learning","parametrized action MDP","neuro-symbolic reasoning","Datalog","action pruning","gradient-guided parameter optimization","sample efficiency","explainable reinforcement learning"],"falsifier":"Take a PAMDP where the rules are deliberately corrupted to prune a random subset of actions with the same pruning rate as the true rules. If KGRL with random rules matches the AULC of KGRL with true rules, the reported advantage could be due to action-space reduction rather than semantic guidance; if true rules clearly outperform random rules of equal pruning rate, the knowledge content is doing the causal work.","tokens_in":15544,"feed_emoji":"🧠","tokens_out":5738,"duration_ms":55949,"temperature":0.7,"pith_summary":"This paper tries to show that explicit but incomplete symbolic domain knowledge can make reinforcement learning in parametrized action spaces dramatically more sample-efficient. In a parametrized action MDP (PAMDP), each decision is a discrete action plus continuous parameters, and existing methods estimate parameters with a single forward pass. KGRL instead evaluates a Datalog knowledge base at every step to prune actions that are known to be inapplicable and to constrain parameters to feasible ranges, then refines parameter guesses by projected gradient ascent on the Q-function. The authors argue this guides exploration toward feasible decisions and leads to higher episodic returns with lower variance than state-of-the-art baselines on four benchmark domains. A sympathetic reader would take away that symbolic rules and gradient-based parameter search are complementary, and that even incomplete rules can substantially improve both training and deployment.","feed_headline":"Rule-guided RL beats five baselines on parametrized-action tasks","feed_subtitle":"Symbolic rules prune invalid actions and a gradient loop tunes parameters, boosting sample efficiency and returns.","key_machinery":"The central object is the knowledge-constrained PAMDP, a PAMDP enriched with a Datalog knowledge base K and an abstraction function α mapping continuous states to symbolic facts. From these, the set of applicable actions and, for each action, a feasible parameter region are derived as hard constraints. The PARAMOPT loop — projected gradient ascent on the differentiable Q-function with respect to the parameters, projecting back into the feasible region — is the mechanism that turns one-shot parameter estimates into iteratively refined ones. KGRL is the DQN-based algorithm that couples rule-based action pruning with this parameter refinement at every step.","core_discovery":"KGRL extends DQN with two coupled mechanisms: a Datalog knowledge base that, together with a symbolic abstraction of the state, computes the set of applicable actions and the feasible parameter region for each action; and PARAMOPT, a projected gradient ascent loop that optimizes the action's parameter by ascending the action-value function's gradient within the feasible region. At decision time, KGRL evaluates the rule base, prunes non-applicable actions, runs PARAMOPT for each remaining action, and greedily selects the action-parameter pair with the highest Q-value. The paper reports that across HardMoveX, CatchPoint, HardGoal, and Platform, KGRL achieves the highest average Area Under the","pith_inferences":["The paper's own ablations imply that enabling gradient-guided refinement only after the Q-function is well fitted could preserve the deployment benefit while avoiding the training-time noise it observes.","If the knowledge base were learned from data or demonstrations instead of handcrafted, the same algorithm could transfer to domains where experts cannot articulate rules; the hard-pruning design would then need softening to tolerate rule errors.","Systematically corrupting the rule base (e.g., randomly flipping a fraction of pruning rules) and measuring AULC would isolate whether the gains come from the semantic content of the knowledge or merely from shrinking the action space.","The explanation traces suggest a post-hoc aggregation step could produce global explanations — e.g., which rules fire most often in which state regions — turning local justifications into policy-level insight."],"forward_implications":["If KGRL's claims hold, RL agents can safely embed incomplete expert rules as hard constraints without losing the ability to learn the unspecified part of the policy.","Sample efficiency in PAMDPs improves because exploration is restricted to feasible action-parameter regions, so fewer environment interactions are wasted on clearly invalid decisions.","Gradient-based parameter refinement at decision time can substitute for learned one-shot parameter estimators, yielding better parameters with the same Q-network.","The approach produces local procedural explanations as a byproduct of the decision mechanism, which is relevant for safety-critical deployment.","Knowledge guidance matters most at deployment, suggesting that rule bases act as a hard feasibility filter even when the policy is already trained."],"fun_headline_variants":["Prune with rules, tune with gradients: KGRL for PAMDPs","KGRL: Datalog rules prune actions, gradient ascent tunes parameters","Rule-guided RL prunes invalid actions, gradient loop refines parameters for PAMDPs","Datalog rules cut action space, gradient ascent tunes parameters for RL","Combine symbolic rules and gradient ascent to lift PAMDP sample efficiency"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the handcrafted Datalog rules and the symbolic abstraction are correct enough to prune only suboptimal decisions; a wrong rule that prunes an optimal action or parameter range directly harms the agent's performance.","fun_headline_variants_meta":{"raw":{"variants":["Prune with rules, tune with gradients: KGRL for PAMDPs","KGRL: Datalog rules prune actions, gradient ascent tunes parameters","Rule-guided RL prunes invalid actions, gradient loop refines parameters for PAMDPs","Datalog rules cut action space, gradient ascent tunes parameters for RL","Combine symbolic rules and gradient ascent to lift PAMDP sample efficiency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001126,"raw_usage":{"total_tokens":4539,"prompt_tokens":783,"completion_tokens":3756,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":3655}},"tokens_in":527,"tokens_out":3756,"duration_ms":26749,"temperature":1.0,"reasoning_tokens":3655,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T06:11:43.631442+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a PAMDP where the rules are deliberately corrupted to prune a random subset of actions with the same pruning rate as the true rules. If KGRL with random rules matches the AULC of KGRL with true rules, the reported advantage could be due to action-space reduction rather than semantic guidance; if true rules clearly outperform random rules of equal pruning rate, the knowledge content is doing the causal work.","supporting_citations":[],"review_version":2}