{"id":"2c3ade9e-dd49-4e84-aa5a-2aa4a779c507","arxiv_id":"2412.19578","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper claims that trust-region-triggered clipping policy optimization with a scaled dot-product graph attention encoder improves RL-based causal discovery on synthetic and benchmark datasets.","lead":"This paper combines a graph attention encoder with a modified policy gradient method for reinforcement-learning-based causal discovery, and reports accuracy gains on synthetic and benchmark datasets. The paper is worth a look because causal discovery from observational data is broadly useful, but the headline claim is undercut by the paper's own experimental results and choices.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TRC's per-edge KL-gated clipping does not bound the joint likelihood ratio used in Algorithm 3's surrogate loss, so the claimed trust-region guarantee and aggregate-deviation fix are unsupported.","rationale":"The reader's weakest assumption is exactly the most load-bearing point: TRC's per-subaction KL gate is asserted to keep the joint policy inside a trust region, but the paper's own Algorithm 3 multiplies the per-edge ratios, and Eq. 19 gives no bound on that product. Since the paper's headline contribution is a trust-region-navigated clipping rule that 'guarantees' steadiness, an unproven and in fact violated aggregate bound directly undermines the central claim. This is a correctness risk, not merely a disagreement with consensus: the proposed algorithm's distinctive mechanism does not deliver its stated guarantee. The other concerns raised by the reader (PPO excluded from tables, hyperparameters tuned on the benchmark, GP results naming PSR-BIC as best, and the Algorithm 4/Table III inconsistencies) weaken confidence in the empirical support, but the trust-region gap is the fundamental issue.","tokens_in":17569,"tokens_out":4961,"duration_ms":44159,"concrete_test":"Analyze one TRC update step under Eq. 19 with all subactions satisfying D_KL < sigma but ratio 3, e.g., b_{ij}=0.01 and pi_{ij}=0.03 for every edge in a 12-node graph. Algorithm 3 will leave every ratio unclipped, giving a joint ratio of 3^132 in the surrogate loss; compute the resulting update and compare it with the PPO clipping bound and with a true KL trust-region step. If the joint ratio exceeds (1+epsilon)^{n(n-1)} or the KL divergence after update is orders of magnitude above sigma, the trust-region claim is falsified. A complementary rerun should also include standard PPO with the same per-subaction factorization and hyperparameters on SynTReN and CYTO, since the paper omits PPO from the reported SHD comparisons.","verdict_should_be":"REJECT","load_bearing_attack":"The load-bearing claim is that TRC (Section V-C, Algorithm 3, Eq. 19) prevents the 'aggregate deviation' of PPO and keeps updates inside a trust region. The mechanism does not do this. Algorithm 3 line 9 defines the surrogate loss as L_theta = A'_t * Π_{i,j} ratio^{i,j}, i.e., the joint probability ratio over all subactions. Eq. 19 clips each ratio only when that edge's own D_KL exceeds sigma, and otherwise leaves it unchanged. Two consequences follow. First, a per-edge KL below sigma does not bound the per-edge likelihood ratio: for independent Bernoulli policies, D_KL can be arbitrarily small while q/p is large whenever the old probability p is small, so ratios well outside [1-epsilon,1+epsilon] are never clipped. Second, even if every ratio were clipped into [1-epsilon,1+epsilon], the product over n(n-1) subactions is bounded only by (1+epsilon)^{n(n-1)}; for n=12 and epsilon=0.2 this is roughly 3.5e10. Thus the aggregate deviation the paper attributes to PPO is still present in TRC. The paper provides no bound on the joint ratio or on D_KL(b,pi_theta|S) under Eq. 19; Fig. 7 reports only clipping frequencies, which do not measure trust-region satisfaction. The central claimed improvement in optimization robustness therefore rests on an unverified mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes Trust Region-navigated Clipping policy optimization (TRC) for RL-based causal discovery, along with a scaled dot-product graph attention encoder (SDGAT). The authors argue that REINFORCE is prone to local convergence, TRPO is computationally expensive, and PPO suffers from aggregate deviation because the joint likelihood ratio is a product of many per-edge ratios; TRC replaces PPO's ratio trigger by a per-edge KL-divergence trigger. Experiments on synthetic linear-Gaussian, LiNGAM, quadratic, GP, SynTReN, and CYTO datasets compare TRC-BIC/TRC-BIC2, PSR, REINFORCE, and non-RL baselines. The central claims are that TRC is more robust and faster than PPO/REINFORCE/PSR and that SDGAT improves causal encoding.","tokens_in":17810,"tokens_out":5617,"duration_ms":45151,"significance":"The problem is important: RL-based causal discovery needs stable policy optimization for high-dimensional binary action spaces. The paper's identification of aggregate deviation in PPO's product ratio is valid and interesting. The SDGAT encoder is a reasonable extension of GAT, and the experimental campaign is broad, covering both synthetic and real benchmarks. The paper ships explicit pseudocode for all algorithms and attempts to justify every design choice. However, the load-bearing theoretical mechanism for TRC is not established: as defined, it does not bound the joint ratio or the joint KL divergence, so the claimed improvement over PPO is unverified. The contradictory GP result and the tuning of hyperparameters on evaluation datasets further weaken the empirical claims.","major_comments":[{"comment":"The central robustness claim is unsupported. Eq. (19) clips the per-edge likelihood ratio only when that edge's own D_KL exceeds sigma. For independent Bernoulli subpolicies, D_KL can be arbitrarily small while the likelihood ratio q/p is large (whenever the old probability p is small), so ratios far outside [1-epsilon, 1+epsilon] are never clipped. Moreover, even if every ratio were clipped, the surrogate in Algorithm 3 line 9 uses the product over all subactions; with n=12 and epsilon=0.2, (1+epsilon)^{n(n-1)} is roughly 3.5e10, so the aggregate deviation the authors attribute to PPO remains present in TRC. No bound is provided on the joint ratio or on D_KL(b, pi_theta | S), and Fig. 7 reports only clipping frequencies, which do not measure trust-region satisfaction. The stated guarantee that TRC 'stays safe in the KL bounds' (Section VI-A) is therefore not established.","section":"V-C, Algorithm 3, Eq. (19)"},{"comment":"PPO is excluded from all quantitative comparisons after being described as 'miserable' (SHD over 35). The paper therefore never reports a head-to-head TRC-versus-PPO comparison on the same benchmark; the abstract and contribution claims that TRC outperforms PPO are not supported by any table or figure. At minimum, a table with PPO results and variance across seeds is required before such claims can be evaluated.","section":"VI-A, Fig. 7"},{"comment":"The GP experiment directly contradicts the paper's general claim. The text states that 'PSR would deliver the best result when combined with BIC, whilst REINFORCE still lag behind the other two RL approaches.' This means on one of the four synthetic settings, the proposed TRC does not outperform PSR. The abstract and introduction claim TRC outperforms 'former RL methods' without qualification. This internal inconsistency must be addressed by either revising the claim or explaining this result.","section":"VI-B, Fig. 9"},{"comment":"The (epsilon, delta) pairs are selected by grid search on the CYTO dataset, and the text says 'the choice of (epsilon-delta) pair for synthetic dataset is obtained likewise,' i.e., tuned on the same datasets later used for final SHD reporting. This selection on the test sets inflates the reported performance and makes comparisons with fixed-default baselines unfair. The paper should use a validation-set split or report the sensitivity of the final results to these hyperparameters.","section":"Appendix A, Section VI"},{"comment":"The penalty schedule is internally inconsistent and non-reproducible. Table III lists Lambda_1 = 0 and BIC_u = -1, while the text says lambda_1 is increased with upper bound Lambda_1 and lambda_2 is increased with upper bound BIC_u. With BIC_u = -1, the update lambda_2 <- min(lambda_2 + Delta_2, BIC_u) drives lambda_2 negative, which turns the acyclicity penalty into a reward for cycles; Lambda_1 = 0 prevents lambda_1 from ever becoming positive. In addition, Algorithm 4 requires BIC0, but Table III does not list it. These issues must be corrected and the actual values used in the experiments reported.","section":"Algorithm 4, Table III"}],"minor_comments":[{"comment":"The title and abstract use 'Casual Discovery' where 'Causal Discovery' is meant; also 'without priori neighbourhood information' should be 'without a priori neighbourhood information' in the abstract and Section IV-A.","section":"Title and Abstract"},{"comment":"The caption for Fig. 10 says the CYTO dataset has '14 nodes,' while the text states the graph has 11 nodes and 17 edges; please reconcile this discrepancy.","section":"Section VI-C, Fig. 10"},{"comment":"Equation (5) uses the notation '[X At]_i,j' without defining it; the regression estimate for Xi,j should be spelled out.","section":"Equation (5)"},{"comment":"The advantage conventions differ between Algorithm 1 line 4 (Rt - Rm - V_omega) and Algorithm 2 line 2 (Rt + Rm - V_omega); this sign inconsistency should be checked and corrected.","section":"Section V, Algorithms 1 and 2"},{"comment":"The claim that TRC 'outperforms former RL methods' is not placed in the context of recent RL-based causal discovery approaches such as CORL or DAG-Actor; the scope of the comparison should be stated.","section":"Section VI-A, Fig. 7 and related text"}],"recommendation":"reject","confidential_remarks":"The paper has several internal inconsistencies that suggest it is not yet ready for journal review. The most serious is the unsupported trust-region claim in Section V-C, which is the core algorithmic contribution; the GP result in Section VI-B directly contradicts the main claim; and the tuning of hyperparameters on test datasets in Appendix A undermines the reported comparisons. These are load-bearing issues that would require a redesign of the method and a re-run of the experiments, not just local edits."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is an incremental combination of known ingredients — trust-region-triggered clipping (similar to TPPO), scaled dot-product attention replacing GAT's additive attention, and prioritized replay — applied to RL-based causal discovery. That combination is new, and the authors test it on several synthetic and real benchmarks, provide pseudocode, and are honest about scaling limits and prior work. The SDGAT encoder is straightforward but sensible, and the empirical comparisons against classic methods like PC, GES, NOTEARS, and DAG-GNN are useful.\n\nThe problem is the load-bearing theoretical claim. The paper argues that PPO's per-ratio clipping causes \"exponentially-large aggregate deviation\" when multiplying over many subactions, and that TRC fixes this by clipping only when per-edge KL exceeds a threshold. As written, this does not work. Per-edge KL can be arbitrarily small while the likelihood ratio is large — for Bernoulli policies, a tiny old probability p makes q/p huge even when KL is near zero. And even if every ratio were clipped into [1-ε,1+ε], the product over n(n-1) subactions is bounded by (1+ε)^(n(n-1)); for n=12 and ε=0.2 that is on the order of 10^10. So the aggregate deviation TRC is meant to prevent is still present. Figure 7 reports clipping frequencies, not trust-region satisfaction, so it does not fill the gap.\n\nThe empirical side also has issues. PPO is dropped from the main comparisons after poor results, without its learning curves or final metrics shown alongside the others. The GP experiment (Fig. 9) explicitly names PSR-BIC as the best method, which contradicts the introduction and abstract's claim that TRC outperforms PSR. Hyperparameters (ε, δ) are tuned on the same datasets used for final reporting, which gives the headline numbers a fitting component. Algorithm 4 and Table III give inconsistent descriptions of the penalty schedule — Algorithm 4 has λ1 multiplied by Δ1 and λ2 incremented by Δ2, while Table III labels Δ1 as additive and Δ2 as multiplicative. BIC0 in Algorithm 4 is never defined. No code is provided.\n\nThis is not a shoddy paper; the experiments are documented and the authors engage with related work, including citing Wang et al.'s similar trigger algorithm. But the central mechanism does not do what is claimed, and the authors' own data contradict the main narrative. The right outcome is major revision, not acceptance as is. If this crosses my desk, I would send it to a referee who knows policy gradient theory cold, with a specific instruction to scrutinize Equation 19 and the product-ratio bound. A serious referee could push the authors to either prove a real trust-region guarantee or reframe the paper as a heuristic with compelling empirical support.","headline":"A reasonable incremental RL-for-causal-discovery package with real empirical work, but the central trust-region claim is mathematically unsupported and the paper's own results undercut its headline.","tokens_in":18367,"tokens_out":2142,"would_cite":false,"duration_ms":21631,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that per-edge trust-region clipping makes reinforcement-learning-based causal discovery both steadier and more accurate, and that a scaled dot-product graph attention encoder adds further gains, with best structural…","keywords":["causal discovery","reinforcement learning","policy optimization","trust region","graph attention network","directed acyclic graph","Bayesian information criterion","prioritized experience replay"],"falsifier":"Run TRC on a 12-node linear-Gaussian dataset and record the joint KL divergence between old and new policies at every update; if the joint KL regularly exceeds the intended trust-region bound while per-edge clipping is active, the central mechanism is not doing its claimed work.","tokens_in":17318,"feed_emoji":"🧠","tokens_out":8058,"duration_ms":67271,"temperature":0.7,"pith_summary":"The paper tries to establish that reinforcement-learning-based causal discovery, which searches for the best-scored directed acyclic graph by sampling adjacency matrices, can be made more accurate and stable with a per-edge clipping rule. The trouble it addresses is that the graph action is a product of many independent edge decisions, so PPO's fixed global clipping lets small per-edge deviations multiply into a large aggregate deviation. The proposed TRC rule clips an edge's update only when that edge's own statistical distance from the old policy crosses a threshold, leaving other edges free. The paper pairs this with SDGAT, a scaled dot-product graph attention encoder that extracts variable features without prior graph information. On synthetic, SynTReN, and CYTO data, TRC is reported to beat REINFORCE, prioritized-sampling REINFORCE, PPO, and non-RL baselines, with the best structural Hamming distances, a count of edge edits needed to match the true graph.","feed_headline":"Trust-region clipping sharpens RL causal discovery","feed_subtitle":"Gating each edge by its own drift cuts SHD to 35 on SynTReN and 9 on CYTO.","key_machinery":"The load-bearing mechanism is the KL-gated clipping rule of TRC, built on the factorization of the graph policy into independent Bernoulli subpolicies, one per directed edge. For each edge, the likelihood ratio is clipped only if the per-edge KL divergence between the new and old policies exceeds a threshold, and otherwise it is retained exactly. The paper argues this gives first-order efficiency with trust-region safety, avoiding both TRPO's cost and PPO's aggregate drift. The supporting encoder, SDGAT, uses scaled dot-product attention in a two-level multi-head design to extract variable features without requiring prior neighborhood information, which the paper claims is better suited to causal discovery than GAT's additive attention.","core_discovery":"On the paper's own terms, the central discovery is that the failure of PPO in causal discovery comes from aggregate constraint deviation: the joint likelihood ratio is the product of $n(n-1)$ per-edge ratios, so even trivial per-edge violations compound. TRC replaces the ratio-triggered clipping of PPO with a trust-region-triggered clipping: each per-edge ratio is clipped to a fixed interval around 1 only when that edge's own KL divergence crosses a threshold, and is left untouched otherwise. This is claimed to keep the joint policy near the old policy while preserving the exploratory freedom of edges that have not drifted. With the SDGAT encoder replacing GAT's additive attention by scaled dot-product attention, TRC-BIC and TRC-BIC2 are reported to obtain the lowest structural Hamming distances on the SynTReN pseudo-real datasets and the CYTO protein-signaling dataset among all compared methods.","pith_inferences":["Beyond the paper, one can directly test whether the per-edge gating actually controls the joint policy by computing the aggregate KL divergence during TRC training; the paper reports clipping rates but not joint KL values, so this is a concrete way to verify the mechanism.","If TRC works as claimed, the same per-edge clipping rule could transfer to other combinatorial generators, such as molecule or architecture generation, where the action is a product of many independent choices rather than Bernoulli edges.","The paper leaves implicit that SDGAT could serve as a general structure-agnostic attention encoder, since its experiments only cover causal discovery and the authors themselves note that transductive and inductive performance remains untested.","A more principled schedule for the clipping threshold and the trust-region threshold could replace the grid search reported in the appendix, provided a closed-form relation between per-edge KL and joint-policy KL is derived."],"forward_implications":["On the 12-node linear-Gaussian and LiNGAM settings, TRC converges to a batch negative reward around -2.35 and stops fluctuating earlier than REINFORCE or PSR, with PPO excluded because its SHD exceeded 35.","On nonlinear quadratic data, TRC produces graphs with SHD at most 1, effectively recovering the true graph.","On SynTReN, TRC-BIC and TRC-BIC2 reach SHD 35.0 and 34.9, the best among all compared methods, and on CYTO they reach SHD 9 and 10.","With TRC-BIC on CYTO, the SDGAT encoder converges to a reward of -5.48 with standard deviation 0.27 and better final metrics than GAT and Transformer encoders.","The TRC idea is claimed to generalize to other high-dimensional combinatorial optimization problems whose actions decompose into many independent subactions."],"supporting_citations":[{"why":"Establishes the RL encoder-decoder formulation and the RL-BIC/RL-BIC2 baselines that this paper extends.","marker":"[6]"},{"why":"Provides the PPO clipping algorithm whose aggregate deviation TRC is designed to fix.","marker":"[11]"},{"why":"Supplies the GAT encoder with additive attention that SDGAT modifies and outperforms.","marker":"[12]"},{"why":"Gives the trace acyclicity penalty and the Lagrangian penalty schedule used in the reward.","marker":"[15]"},{"why":"Defines the additive noise model settings and identifiability assumptions under which the synthetic experiments are run.","marker":"[17]"},{"why":"Provides the rank-based prioritized replay used by both PSR and TRC.","marker":"[41]"},{"why":"Supplies the CYTO protein-signaling benchmark on which TRC reports the lowest SHD.","marker":"[46]"},{"why":"Supplies the SynTReN pseudo-real gene-expression datasets on which TRC reports the best SHD.","marker":"[47]"}],"fun_headline_variants":["Trust-region clipping fixes PPO's causal search flaw","Per-edge clipping prevents PPO constraint drift","New clipping rule fixes PPO in causal discovery","Trust-region clipping beats PPO for causal search"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes, without proof, that gating each edge's update by that edge's own statistical distance from its old policy keeps the whole product-of-edges policy inside a trust region; if this per-edge gate does not control the joint deviation, TRC loses its claimed advantage over PPO.","fun_headline_variants_meta":{"raw":{"variants":["Trust-region clipping fixes PPO's causal search flaw","Per-edge clipping prevents PPO constraint drift","New clipping rule fixes PPO in causal discovery","Trust-region clipping beats PPO for causal search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001288,"raw_usage":{"total_tokens":5273,"prompt_tokens":970,"completion_tokens":4303,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":4242}},"tokens_in":586,"tokens_out":4303,"duration_ms":34109,"temperature":1.0,"reasoning_tokens":4242,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:12:07.769900+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run TRC on a 12-node linear-Gaussian dataset and record the joint KL divergence between old and new policies at every update; if the joint KL regularly exceeds the intended trust-region bound while per-edge clipping is active, the central mechanism is not doing its claimed work.","supporting_citations":[{"cited_title":"Causal discovery with reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Establishes the RL encoder-decoder formulation and the RL-BIC/RL-BIC2 baselines that this paper extends."},{"cited_title":"Dags with NO TEARS: continuous optimization for structure learning,","cited_arxiv_id":null,"evidence_quote":"Gives the trace acyclicity penalty and the Lagrangian penalty schedule used in the reward."},{"cited_title":"Causal discovery with continuous additive noise models,","cited_arxiv_id":null,"evidence_quote":"Defines the additive noise model settings and identifiability assumptions under which the synthetic experiments are run."},{"cited_title":"Prioritized Experience Replay,","cited_arxiv_id":null,"evidence_quote":"Provides the rank-based prioritized replay used by both PSR and TRC."},{"cited_title":"Causal Protein-Signaling Networks Derived from Multiparam- eter Single-Cell Data,","cited_arxiv_id":null,"evidence_quote":"Supplies the CYTO protein-signaling benchmark on which TRC reports the lowest SHD."},{"cited_title":"Syntren: a generator of synthetic gene expression data for design and analysis of structure learning algorithms,","cited_arxiv_id":null,"evidence_quote":"Supplies the SynTReN pseudo-real gene-expression datasets on which TRC reports the best SHD."}],"review_version":1}