{"id":"6a47195e-a657-4f33-ac12-c78d5d801976","arxiv_id":"2504.16879","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A loss function built from differentiable reachable-set bounds lets neural control policies be trained to satisfy reach-avoid and invariance specifications, so a lightweight verifier can re-check them at run time.","lead":"This paper trains robot control policies with a safety check built into every learning step, using a fast approximation of all possible trajectories to steer the policy toward safe behavior. The result is a control policy whose safety can be re-checked quickly at run time, demonstrated on simulated unicycle and quadrotor models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported loss weights in Eq. (10) would push reachable sets into obstacles and away from the goal, contradicting the paper's main result; need code check.","rationale":"The reader's verdict correctly flags a likely sign error in the obstacle-overlap weight and recommends conditional acceptance. My stress-test confirms that this is the single most load-bearing concern, and I elevate it from a side note to the central issue. The paper's main claim is empirical: the proposed loss, with the stated weights, yields policies whose CROWN reachable sets satisfy reach-avoid. Eq. (5) and Eq. (10) together imply an objective that rewards obstacle overlap and punishes goal overlap, which is the opposite of the intended behavior. The internal contradiction between Section IV-C's text (wgoal overlap = 15) and Eq. (11) (w_overlap_goal = -15) reinforces that the signs are misreported. This is not a matter of tuning or conservative verification; it is a potential foundational inconsistency in the method as described. A straightforward code check can settle whether this is a typographical error. If the code uses different signs, the paper needs a correction and the experiments must be re-run with the corrected weights. If the code matches the paper, the central result is unreproducible. Either way, the paper cannot be accepted as is. Because the issue is potentially fixable and the underlying idea of verification-in-the-loop training is plausible, CONDITIONAL remains the appropriate verdict, but the condition must explicitly include resolving this sign inconsistency.","tokens_in":9427,"tokens_out":4266,"duration_ms":39808,"concrete_test":"Inspect the public GitHub repository's unicycle training script and extract the exact loss weights used in the implementation. Then re-run training for 20,000 epochs with the weights exactly as printed in Eq. (10) (w_overlap_danger = -15, w_overlap_goal = 20) and check whether the final CROWN reachable sets avoid obstacles and reach the goal. If the implemented signs differ from the paper, the paper misstates the method; if they match, the reported avoidance result is unreproducible as written.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central experimental claim in Section IV-A is that the proposed loss trains a NN policy whose CROWN reachable sets avoid obstacles and reach the goal (Fig. 2c). For this to be true, the loss must encode those objectives. Eq. (5) defines L_overlap as the volume of intersection between the reachable box and a target box (goal or obstacle). Eq. (10) then sets w_overlap_danger = -15 and w_overlap_goal = +20. Minimizing -15 * L_overlap_obs encourages large overlap with obstacles, and minimizing +20 * L_overlap_goal encourages small overlap with the goal. Both signs are opposite to the stated intent. The inconsistency is not isolated: Section IV-C lists 'wobstacle overlap = 20, wgoal overlap = 15' but Eq. (11) says w_overlap_goal = -15, an internal contradiction. If Eq. (10) is implemented as written, the training objective would actively oppose the reported behavior, so the paper's main evidence is unsupported as stated. This is more load-bearing than the tinv sensitivity because it undermines the method's mechanism, not just its robustness.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a verification-in-the-loop training method for neural network control policies. At each training iteration, differentiable CROWN reachable-set over-approximations are used to define loss terms that penalize overlap with obstacles, encourage goal reaching, minimize reachable-set volume, and encourage late-horizon forward invariance. The closed-loop bounds are computed iteratively over a finite horizon and are used both as training signals and, after training, as verification certificates. Experiments on a unicycle model and a 6D quadrotor model report reach-avoid behavior and an apparent forward invariant set, along with a qualitative comparison against a numerical-gradient baseline.","tokens_in":9726,"tokens_out":6279,"duration_ms":54134,"significance":"If the reported results hold, the contribution is useful: it provides a lightweight mechanism to bias policy search toward policies that are amenable to fast, sound verification, and the use of sound CROWN bounds means a verified over-approximation is a genuine safety certificate for the true dynamics. The paper releases its code and includes a comparison to a prior numerical-gradient approach. However, the evidence is currently weakened by an apparent sign inconsistency in the main loss equation, the lack of statistical evaluation, and the hand-picked invariance start time. These issues must be resolved before the central claims are established.","major_comments":[{"comment":"The loss weights in Eq. (10) contradict the stated objectives. Eq. (5) defines L_overlap as the volume of intersection between the reachable box and a target region. With w_overlap_danger = -15, minimizing -15 * L_overlap_obs encourages the reachable set to intersect obstacles, and with w_overlap_goal = +20, minimizing +20 * L_overlap_goal discourages intersection with the goal. Both signs are opposite to the intended reach-avoid objective. Furthermore, Section IV-C states \"wobstacle overlap = 20, wgoal overlap = 15\" while Eq. (11) reports w_overlap_goal = -15 and omits the obstacle term entirely. Because the training objective is the mechanism by which the reported policies are obtained, this inconsistency must be resolved by checking the released code and correcting either the formulas or the reported weights; as printed, the main experimental claim is unsupported.","section":"Section IV-A, Eq. (10) and Eq. (5)"},{"comment":"The reported invariant set is a direct consequence of the choice t_inv = 22. Linv penalizes ||\\bar{x}_{t+1} - \\bar{x}_t|| for all t >= t_inv, so the objective explicitly encourages \\bar{R}_{23} to coincide with \\bar{R}_{22}. The CROWN subset check is a valid sound verification, but the demonstration does not establish the method's robustness: no ablation over t_inv, no multiple seeds, and no evidence that an invariant set is found when t_inv is not chosen to match the final invariant set. The claim that the method \"can lead to an invariant set\" would be considerably strengthened by reporting success rates across t_inv values and random initializations.","section":"Section IV-C, Eq. (8) and the invariance experiment"},{"comment":"The results consist of single training runs and single trajectory visualizations, without seeds, error bars, or quantitative success metrics. The comparison in Section IV-A with the numerical-gradient baseline is qualitative (Fig. 2), and the text does not report how often each method produces a policy that passes the CROWN verification. Since the central assertion is that the proposed loss reliably yields verifiable policies, at least success-rate statistics over multiple random seeds and quantitative measures of reachable-set volume and obstacle clearance should be reported.","section":"Section IV, overall experimental design"}],"minor_comments":[{"comment":"The two terms printed as ||\\bar{x}_{t+1} - \\bar{x}_t||^2 appear identical; presumably one should apply to the lower bound \\underline{x}_t and the other to the upper bound \\bar{x}_t.","section":"Eq. (8)"},{"comment":"The phrase \"penalizes the volume of (positional) reachable sets that are not within a region\" is inconsistent with the displayed formula, which computes the volume of the intersection with the region; please clarify the intended penalization.","section":"Section III-A, Eq. (5) and surrounding text"},{"comment":"The phrase \"true reachable sets according to the sampled trajectories\" should read \"sampled trajectories\" or \"empirical reachable sets,\" since finitely many samples do not determine a true reachable set.","section":"Section IV-B, last paragraph"},{"comment":"The call CROWN(f, \\bar{R}_t) should specify that the controller parameters are \\theta_i, and the notation for the set of avoid sets A0:A in the input list is undefined.","section":"Algorithm 1, line 6"}],"recommendation":"major_revision","confidential_remarks":"The sign inconsistency in Eq. (10) is the most important issue; I would ask the editor to require the authors to confirm the implementation against the released code before acceptance. The novelty relative to [33]-[35] is incremental, but the paper's emphasis on using loss terms to obtain tight bounds for lightweight verification is reasonable if the experiments are made rigorous."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is sound and worth taking seriously: train a neural control policy using differentiable CROWN reachable sets in the loss, so that the final policy is verifiable by the same fast relaxation. The additions of the volume-tightness term and the invariance term are genuinely new relative to [33]–[35], and the no-reward formulation makes the approach simpler to reason about. The code is provided, which is a real asset. The unicycle and quadrotor examples are illustrative, and producing an invariant box via CROWN during training is a neat result even if it is partly shaped by the loss.\n\nThe soft spots are significant but mostly fixable. The biggest problem is the sign of the overlap weights. Equation (10) sets w_overlap_danger = -15 and w_overlap_goal = +20. Since L_overlap in Eq. (5) is the intersection volume between the reachable box and a target box, minimizing the loss with those weights should encourage overlap with obstacles and discourage overlap with the goal. That is the opposite of the stated intent. Section IV-C then adds a second inconsistency: the text says 'wobstacle overlap = 20, wgoal overlap = 15', while Eq. (11) reports w_overlap_goal = -15. I cannot tell whether the paper text or the code corresponds to what was actually run, but this must be resolved before the experimental claims are credible. The paper ships code, so a quick check would settle it.\n\nBeyond that, the experiments lack statistical rigor. There are no seeds or repeated runs, the invariance result depends on a hand-picked tinv=22 and the sensitivity is not examined, and the claim that the method enables fast runtime verification is not supported by any timing measurements. The volume-tightness result is to some degree by construction—the loss minimizes bound volume—but that is still a useful demonstration, not a flaw per se.\n\nMy overall read: the method is plausible, the verification story is honest because CROWN is sound, and the contribution is incremental but useful for the safety-critical control community. The sign issue is load-bearing if it is not just a typo, but it is easily checked. I would send this to review, with a strong request that the authors fix the weight signs, add seeds, report timing, and test the robustness to tinv.","headline":"A useful verification-in-the-loop training idea with a likely sign error in the reported loss weights; the core approach is worth refereeing, but the experiments need cleaning up.","tokens_in":10207,"tokens_out":2674,"would_cite":false,"duration_ms":25993,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Safety verification during training yields policies that pass checks","keywords":["verification-in-the-loop training","neural feedback loop","CROWN reachability analysis","reach-avoid specification","invariant set","bound tightness loss","unicycle control","quadrotor control"],"falsifier":"Run the unicycle experiments with $w_{\\mathrm{overlap\\,danger}}$ changed from $-15$ to a positive value and with $t_{\\mathrm{inv}}$ swept over nearby steps; if the CROWN reachable sets then intersect an obstacle or $\\bar{R}_{t_{\\mathrm{inv}}}$ stops being forward invariant, the reported verification success is an artifact of those settings. Independently, densely sample initial states in $X_0$ and simulate the closed loop; any true trajectory that hits an obstacle while the CROWN bounds avoid it would falsify the claimed specification satisfaction.","tokens_in":9203,"feed_emoji":"✅","tokens_out":5628,"duration_ms":49067,"temperature":0.7,"pith_summary":"This paper argues that safety verification should not wait until after training; instead, the verifier should participate in training so that the learned policy is verifiable with cheap relaxed bounds. The authors train neural network control policies by computing differentiable over-approximations of forward reachable sets with CROWN and adding loss terms for goal-reaching, obstacle overlap, bound tightness, and invariance. On unicycle and 6D quadrotor models, the trained policies reach goal regions while the CROWN reachable sets avoid obstacles, and in one experiment the reachable set at a fixed time is a certified forward invariant set. If the approach is right, safety-critical learning-based control can rely on lightweight, fast verification at runtime rather than expensive post-hoc analysis.","feed_headline":"Safety verification during training yields policies that pass checks","feed_subtitle":"Neural policies trained with CROWN reachability bounds avoid obstacles and certify invariant sets.","key_machinery":"The central mechanism is CROWN, a LiRPA-style backward linear-relaxation bound propagation that computes hyperrectangle outer bounds on the image of a computation graph. Applied to the closed-loop dynamics $x_{t+1}=f(x_t,\\pi_\\theta(x_t))$, it produces reachable set over-approximations $\\bar{R}_t$ that are differentiable with respect to the policy parameters $\\theta$. These bounds feed the loss terms $L_{\\mathrm{overlap}}$, $L_{\\mathrm{goal}}$, $L_{\\mathrm{vol}}$, and $L_{\\mathrm{inv}}$, so gradient descent directly optimizes verifiability of the resulting closed-loop system.","core_discovery":"The central claim is that including verification-derived terms in the training loss produces neural control policies whose closed-loop behavior, as bounded by the CROWN relaxation, provably satisfies reach-avoid and invariance specifications. The paper reports, for example, a 3-layer ReLU unicycle policy whose CROWN over-approximations $\\bar{R}_1,\\dots,\\bar{R}_{24}$ do not intersect the avoid sets and reach the goal region in 24 steps, and a separate run where $\\bar{R}_{22}$ is forward invariant. A key sub-claim is that the bound-volume term $L_{\\mathrm{vol}}$ lets a lightweight relaxed verifier produce tight bounds that normally require expensive branch-and-bound. The method requires no external reward signal; all loss terms are computed from the reachable set bounds themselves.","pith_inferences":["A testable extension is to vary the invariance start time $t_{\\mathrm{inv}}$ and the obstacle-overlap weight; the paper fixes $t_{\\mathrm{inv}}=22$ and sets $w_{\\mathrm{overlap\\,danger}}=-15$, so robustness of the reported invariant set and obstacle avoidance to those choices is open.","The same differentiable-CROWN loss could be applied to backward reachable sets or to Lyapunov-like certificate search, potentially eliminating the need for a separate certificate network.","If bound tightness transfers across initial sets, one could pre-train on a coarse initial set and refine on smaller subsets, making verified-in-the-loop training scale to higher-dimensional systems.","The absence of an external reward signal suggests the reachability-derived losses themselves could serve as a dense reward substitute in reinforcement learning settings."],"forward_implications":["Runtime monitoring can use the same cheap CROWN bounds the policy was trained against, so specifications that change during operation can be re-checked quickly.","The bound-volume term reduces the conservatism of relaxed verification, so guaranteeing a trained policy may not require branch-and-bound or SDP.","The invariance term can turn a trained policy into one with a certified forward invariant set around the goal, verified by a single subset check.","The approach trains policies without an external reward signal, using only reachable-set-derived losses.","The method scales to a 6D quadrotor model with obstacles within thousands of training epochs."],"supporting_citations":[{"why":"Supplies the CROWN/LiRPA bound propagation used to compute differentiable reachable set over-approximations at each training step.","marker":"[1]"},{"why":"Baseline verification-in-the-loop method with numerical gradient estimates; the unicycle experiment shows the proposed CROWN-based loss outperforms it.","marker":"[33]"},{"why":"Cited as the source of remedies (n-step recursive reachable sets, per-subset training) for cases where soft penalties fail to satisfy specifications.","marker":"[35]"},{"why":"Comparison approach that finds invariant sets for an already-trained policy via SDP; the paper contrasts its own training-to-invariance approach.","marker":"[41]"},{"why":"Line of work extending differentiable reachable-set optimization to certified reinforcement learning; this paper removes the external reward signal.","marker":"[34]"}],"fun_headline_variants":["Training with verification yields provably safe policies","Certifiable control policies from verification-aware training","Verification in the loss function learns policies that pass checks","Reachability-based training produces verifiable control policies"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The hand-tuned soft penalty weights—particularly the negative obstacle-overlap weight ($w_{\\mathrm{overlap\\,danger}}=-15$) and the fixed invariance start time $t_{\\mathrm{inv}}=22$—must produce specification satisfaction that survives reasonable perturbations of those choices.","fun_headline_variants_meta":{"raw":{"variants":["Training with verification yields provably safe policies","Certifiable control policies from verification-aware training","Verification in the loss function learns policies that pass checks","Reachability-based training produces verifiable control policies"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1298,"prompt_tokens":800,"completion_tokens":498,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":416,"completion_tokens_details":{"reasoning_tokens":438}},"tokens_in":416,"tokens_out":498,"duration_ms":5056,"temperature":1.0,"reasoning_tokens":438,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:52:58.789755+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the unicycle experiments with $w_{\\mathrm{overlap\\,danger}}$ changed from $-15$ to a positive value and with $t_{\\mathrm{inv}}$ swept over nearby steps; if the CROWN reachable sets then intersect an obstacle or $\\bar{R}_{t_{\\mathrm{inv}}}$ stops being forward invariant, the reported verification success is an artifact of those settings. Independently, densely sample initial states in $X_0$ and simulate the closed loop; any true trajectory that hits an obstacle while the CROWN bounds avoid it would falsify the claimed specification satisfaction.","supporting_citations":[{"cited_title":"Verification in the Loop: Correct-by-Construction Control Learning with Reach-avoid Guarantees","cited_arxiv_id":"2106.03245","evidence_quote":"Baseline verification-in-the-loop method with numerical gradient estimates; the unicycle experiment shows the proposed CROWN-based loss outperforms it."},{"cited_title":"Joint Differentiable Optimization and Verification for Certified Reinforcement Learning","cited_arxiv_id":"2201.12243","evidence_quote":"Line of work extending differentiable reachable-set optimization to certified reinforcement learning; this paper removes the external reward signal."}],"review_version":1}