{"id":"08073414-97ed-4953-8d7c-597aa445836d","arxiv_id":"2412.12597","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"ConformalDQN filters ventilator-setting actions by behavioral-policy confidence before Q-value selection, and the authors report higher FQE-estimated survival than baselines.","lead":"A team at Michigan State and NYU combined conformal prediction with a Double Deep Q-Network to recommend mechanical ventilator settings and flag low-confidence actions. The paper claims their system outperforms doctors and prior RL models on estimated 90-day survival, but the survival figures come from a model-based evaluation rather than measured outcomes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 90-day survival claim rests on an unvalidated FQE-to-survival mapping; Table 2 imputes survival rates from physician Q-value bins rather than measuring outcomes under ConformalDQN.","rationale":"The reader's weakest_assumption emphasized the marginal conformal guarantee failing under OOD exchangeability, which is a real and important limitation. I agree that Eq. 12 does not protect the deliberately shifted OOD set. However, even if the conformal mechanism worked exactly as claimed, the headline clinical benefit would still fail if the FQE-based survival estimates are biased. Table 2's survival column is constructed by a Q-to-survival mapping fit on physician actions and then applied to all policies, and the paper gives no evidence that this mapping is valid across policies. That makes the FQE estimator the most load-bearing component of the central claim. The paper also lacks confidence intervals and independent verification of the FQE implementation, and the reported numbers are internally inconsistent with the reward definition in Eq. 8. These are fixable through re-analysis and clearer reporting, so I keep the reader's CONDITIONAL verdict rather than moving to REJECT. If the suggested re-computation shows overlapping intervals or invalid calibration, the verdict should move to REJECT or UNVERDICTED for the survival claim.","tokens_in":11370,"tokens_out":6021,"duration_ms":57467,"concrete_test":"Recompute Table 2 under Eq. 8's sparse terminal reward: for each policy, report the FQE mean initial Q and the implied survival probability (Q+1)/2, with patient-level bootstrap 95% confidence intervals. Then, on a held-out split, fit the physician Q-bin-to-survival mapping and check whether it predicts actual 90-day mortality in that split; if the implied and mapped survival rates differ by more than Monte Carlo error, the table's survival column is not a valid outcome measure. Use an independent FQE implementation to rule out implementation bias.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim — ConformalDQN 'outperforming other methods by increasing the 90-day survival rate' — is supported only by Table 2, which is not a direct outcome evaluation. The survival percentages are obtained by binning the physician policy's FQE Q-values, computing observed 90-day mortality in each bin, and then mapping each policy's mean initial Q-value through that binning function. This assumes a single Q-to-survival calibration, fit on physician actions, transfers to CQL, DDQN, and ConformalDQN even though their action distributions and Q-scales differ. FQE is itself an approximate off-policy estimator with known bias in offline RL, and the paper reports no confidence intervals, no validation of the Q-calibration, and no independent estimator such as importance sampling. Moreover, with the terminal reward in Eq. 8 (+1 survive, -1 die), an FQE mean Q of 0.639 implies expected return 0.639, i.e. survival probability (0.639+1)/2 = 81.95%, not the reported 83.89%; the discrepancy suggests the mapping is doing real numerical work. Without an unbiased estimator or direct outcome validation, the claimed survival benefit over clinicians and prior RL methods is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ConformalDQN, a framework that augments Double DQN with an estimate of the behavioral policy's action probabilities P_ω(a|s), a conformal threshold τ computed on a calibration set, and an action-selection rule that restricts Q-value maximization to actions with probability at least 1−τ. The method is trained and evaluated on a MIMIC-IV cohort of mechanically ventilated patients, and the paper claims improved 90-day survival relative to physician practice, CQL, and DDQN, as well as robustness to out-of-distribution states. The evaluation uses Fitted Q-Evaluation (FQE), correlation analyses, action-distribution plots, and mean initial Q-values on in-distribution and deliberately constructed out-of-distribution subsets.","tokens_in":11548,"tokens_out":4431,"duration_ms":42483,"significance":"If the central claims were established, the framework would be a useful contribution to safe offline RL in clinical decision support: it provides a distribution-free calibration procedure for action filtering, releases code, and targets a clinically meaningful problem. However, the headline survival benefit is not supported by the evidence: Table 2 is an indirect FQE-based mapping, not a measurement of outcomes under the learned policies, and the conformal guarantee in Eq. (12) is a marginal coverage statement about behavioral actions, not a safety guarantee for the chosen Q-optimal action or for out-of-distribution states. The paper's contribution is best assessed as a modest action-filtering mechanism with a correctly stated but limited conformal guarantee; the clinical and OOD-safety claims require substantial additional work.","major_comments":[{"comment":"The reported 90-day survival rates are not measured outcomes. They are obtained by fitting FQE on the same MIMIC-IV data and the same sparse terminal reward, binning physician Q-values, and mapping each policy's mean initial Q-value through that binning function. FQE is an approximate off-policy estimator with known bias in offline RL, and the paper provides no confidence intervals, no importance-sampling estimator, and no direct validation of the Q-to-survival calibration. Furthermore, with the terminal reward in Eq. (8) (+1 survive, −1 die), a mean Q of 0.639 implies expected survival probability (0.639+1)/2 = 0.8195, not the reported 83.89%, so the physician-Q binning mapping is doing unexplained numerical work. The abstract's claim that ConformalDQN 'outperforms other methods by increasing the 90-day survival rate' is therefore unsupported by the evidence presented.","section":"Evaluation, Table 2"},{"comment":"Equation (12) states P(a* ∈ {a : P_ω(a|s) ≥ 1−τ}) ≥ 1−α for 'an optimal behavioral action', but the conformal guarantee is marginal over the calibration distribution and only covers inclusion of a behavioral action drawn exchangeably from that distribution. It does not cover inclusion of the argmax-Q action, nor does it guarantee that the selected action is safe or clinically appropriate, and the phrase 'optimal behavioral action' is never defined. In the OOD evaluation, the test distribution is deliberately shifted by selecting patients with top or bottom 1% initial features, so exchangeability between calibration and test fails and τ has no stated coverage guarantee. The claim that the filter avoids potentially harmful actions in unfamiliar states is not supported by Eq. (12).","section":"Conformal predictor calibration, Eq. (12)"},{"comment":"The fallback rule in Eq. (14), which selects arg max_a Q(s,a) when A_c is empty, removes the conformal filter precisely in unfamiliar states where no action meets the probability threshold. There is no coverage or safety guarantee for this fallback, and it can select actions with low behavioral probability and potentially high extrapolation error exactly when the model is least reliable. This undercuts the stated safety motivation and should either be removed, analyzed, or replaced with an explicit abstention mechanism.","section":"Uncertainty-Aware Action Selection, Eq. (14)"},{"comment":"The 'overestimation threshold' set at 1.0 is not a valid threshold for the objective actually trained. The maximum return is 1 only in the absence of intermediate rewards and with discount factor γ = 1, whereas the experiments use γ = 0.75 and the reward in Eq. (8) includes intermediate rewards weighted by λ. Comparing mean initial Q-values against this fixed line therefore conflates scale differences with overestimation and does not establish that standard DDQN overestimates or that CQL and ConformalDQN do not.","section":"Out-of-Distribution Evaluation, Figure 4"},{"comment":"The reward function depends on a 'modified APACHE II score' AP(s), but the paper never defines how this score is computed from the 44 state variables, which variables enter it, or how the modification differs from APACHE II. Without this definition, the reward, all learned Q-values, and Table 2 cannot be reproduced or audited, and the claim that the reward aligns with 90-day survival cannot be checked.","section":"RL Problem Formulation, Eq. (8)"}],"minor_comments":[{"comment":"The text says 'L_θ is the standard Double DQN loss describes as eq. 9', but the Double DQN loss is defined in Eq. (3), not Eq. (9); this cross-reference should be corrected.","section":"Loss function, Eq. (9)"},{"comment":"There are several typographical errors, including 'Correspondance', 'weight-asjusted', 'loctate', and 'V olume'; the paper should be carefully proofread.","section":"Methods and Appendix"},{"comment":"The composite loss in Eq. (9) contains an NLL term and an L2 regularization term, but the grid search lists learning rate, discount factor, layer count, and units only; the values of λ, the L2 regularization weight, and any weight on the NLL term are not reported, making the training configuration incompletely specified.","section":"Experimental Setups"},{"comment":"The FQE estimator is described only by a reference; the paper should report the FQE architecture, training details, target update procedure, and whether FQE was trained on the same train/calibration splits as the policies being evaluated.","section":"Evaluation"},{"comment":"The OOD dataset is described as consisting of patients with top or bottom 1% initial features, but it is not stated whether these patients are disjoint from the training, calibration, and test splits; if the OOD selection is performed on the full cohort before splitting, information leakage and overlapping evaluation sets are possible and should be clarified.","section":"Data split and OOD selection"},{"comment":"Table 2 reports mean initial Q-values with standard deviations for five runs, but the corresponding survival-rate columns are shown without any uncertainty intervals; given that the survival mapping is nonlinear, the standard deviation of the survival estimate should be reported or propagated.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The advertised clinical benefit is not established by the current evaluation, and the conformal guarantee is applied to the behavioral action rather than to the Q-selected action. If the authors can reframe the contribution as a behavioral-action filter with marginal coverage, add a direct or unbiased evaluation of outcomes, and repair the OOD claims, the paper could become a useful contribution. In its current form, I would not recommend acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest take: the conformal-DQN integration is real and the mechanics are sound, but the paper overclaims the survival benefit. The headline numbers are FQE estimates mapped through a physician-only calibration curve; that mapping is doing real work and is not validated. There is a legitimate kernel here: the composite loss tying the Q-network to a behavioral probability estimator, the post-training threshold filter, and the OOD stability check all make sense as a method contribution for trustworthy offline RL in critical care.\n\nWhat is actually new: combining split conformal calibration with a Double DQN for ventilator settings, and using the calibrated threshold to restrict the action set at deployment. The method is an incremental variant, not a breakthrough, but it is clearly described and the data pipeline follows DeepVent closely. The paper gives credit to prior work and the code is available.\n\nWhere it is soft. First, the 90-day survival claim. Table 2 is not a measured outcome. It takes FQE initial-state Q-values, bins the physician policy's Q-values, computes observed survival in those bins, then maps each policy's mean Q through that function. That assumes a single Q-to-survival calibration transfers across policies whose action distributions and Q-scales differ. No confidence intervals, no independent estimator. The stress-test note is right that the mapping is numerically inconsistent with the reward structure. Second, the conformal guarantee is only marginal over the calibration distribution; it does not extend to the deliberately shifted OOD set, so the sentence claiming protection against out-of-distribution states is too strong. Third, several hyperparameters (α, λ, L2 weight) are tuned on validation and not fully reported; reproducibility is partial.\n\nThe method itself is sensible, and the OOD figure showing stable Q-values below the overestimation threshold is a fair check. The paper needs a revision that softens the claims, reports conditional or OOD-robust conformal methods, and validates the survival estimator. I would not cite the survival improvement as a result, but I would cite the method as an example of conformal action filtering.\n\nWho should read it: people working on offline RL safety in healthcare and on conformal prediction for decision support. It deserves serious peer review; the flaws are fixable and the topic is important. I would accept it for review with major revision expectations.","headline":"Conformal integration is genuine, but the 90-day survival claim rests on an unvalidated FQE mapping, not measured outcomes.","tokens_in":12152,"tokens_out":2842,"would_cite":true,"duration_ms":23926,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that adding a conformal prediction layer to deep Q-learning yields safer, more conservative ventilator action selection and improves estimated 90-day survival over clinician and prior RL policies.","keywords":["conformal prediction","offline reinforcement learning","deep Q-learning","mechanical ventilation","uncertainty quantification","distribution shift","intensive care decision support","90-day survival"],"falsifier":"Using the paper's calibration set and threshold, compute how often the expert's actual action falls in the confident set on the deliberately unusual out-of-distribution patients; if the rate is well below the target confidence, the out-of-distribution safety mechanism is not doing what is claimed.","tokens_in":11086,"feed_emoji":"🫁","tokens_out":10869,"duration_ms":88474,"temperature":0.7,"pith_summary":"This paper argues that a distribution-free conformal prediction layer can solve a specific safety problem in offline reinforcement learning for intensive care: the tendency of a deep Q-agent to act on overestimated values in unfamiliar patient states. ConformalDQN learns action probabilities for the clinician behavior policy, calibrates a threshold on held-out data, and refuses to consider any action whose probability falls below that threshold unless no action qualifies. In the paper's evaluation on a large ICU dataset, this filtering raised the estimated 90-day survival rate to 83.9%, above the 81.65% of a conservative Q-learning baseline and the 74.9% of physician practice, while keeping recommended ventilator settings inside lung-protective ranges. The authors present the method as a step toward clinically usable decision support with an interpretable confidence measure for each recommendation.","feed_headline":"Conformal filter lifts estimated 90-day survival to 83.9 percent","feed_subtitle":"A distribution-free uncertainty layer keeps AI ventilator advice inside lung-protective ranges.","key_machinery":"The load-bearing object is the conformal confidence threshold $\\tau$ and the confident-action set it defines. Calibrated with $n$ held-out state-action pairs at confidence level $1-\\alpha$, the threshold guarantees $P(a^* \\in \\{a : P_\\omega(a|s) \\geq 1-\\tau\\}) \\geq 1-\\alpha$ under exchangeability, where $a^*$ is the expert action. At decision time the agent takes $\\arg\\max_{a \\in A_c} Q(s,a)$ when $A_c$ is nonempty and falls back to $\\arg\\max_a Q(s,a)$ otherwise; the composite training loss combines Double DQN loss, negative log-likelihood of expert actions, and L2 regularization on logits to keep the probability head well calibrated. The threshold thus acts as a tunable, post-training filter that blocks high-Q but low-confidence actions.","core_discovery":"The central claim is that the conformal threshold, applied to the action probabilities of the behavioral policy, is a sufficient safety mechanism for offline deep Q-learning in mechanical ventilation. ConformalDQN extends Double DQN with a probability head $P_\\omega(a|s)$ trained to imitate the clinician behavior policy, computes the threshold $\\tau$ as the $\\lceil(n+1)(1-\\alpha)\\rceil/n$ quantile of the nonconformity scores $1-P_\\omega(a_i|s_i)$ on a calibration set, and restricts action selection to the confident set $\\{a : P_\\omega(a|s) \\geq 1-\\tau\\}$. The authors report that this mechanism yields the highest mean initial Q-value (0.639), the strongest negative correlation between predicted Q-values and mortality ($-0.563$), and stable Q-values under an out-of-distribution test, and they interpret these results as evidence that the method improves expected 90-day survival and avoids unsafe actions.","pith_inferences":["The authors list state-conditioned conformal prediction as future work; if realized, it would replace the current population-level coverage guarantee with per-patient confidence sets, which is what a clinician would need at the bedside.","The filter-plus-maximization pattern does not depend on Double DQN specifically; any offline RL method that can estimate behavior-policy action probabilities could wear the same conformal safety layer, including continuous-action variants the authors suggest.","Because the reported survival rates are estimated by Fitted Q-Evaluation rather than observed outcomes, an independent off-policy evaluator or a prospective pilot is the natural next check on the 83.9% figure.","The survival comparison would be sharper if future work reports observed 90-day mortality on the test cohort directly, rather than mapping Q-values through binned physician outcomes."],"forward_implications":["If the reported Q-value estimates are right, ConformalDQN recommendations correspond to an estimated 90-day survival of 83.9%, a gain over both the CQL baseline (81.65%) and observed physician practice (74.9%).","Clinicians could adjust the confidence threshold after training to shift the agent between conservative and exploratory behavior without retraining the network.","The method's output includes an interpretable confidence set of acceptable actions, which is the kind of information a human-in-the-loop decision support system needs.","In the paper's OOD evaluation, ConformalDQN keeps mean initial Q-values below the overestimation threshold while standard DDQN exceeds it, supporting the claim that uncertainty filtering mitigates distribution shift.","The same architecture can be transferred to other treatment decisions such as drug dosing, where offline data and safety constraints dominate."],"supporting_citations":[{"why":"Supplies the conformal prediction framework and the coverage guarantee that the method's threshold relies on.","marker":"Shafer and Vovk 2008"},{"why":"Provides the split-conformal calibration procedure and notation used to compute the threshold $\\tau$.","marker":"Angelopoulos and Bates 2021"},{"why":"Defines DeepVent, the CQL baseline, and the data preprocessing and intermediate reward scheme ConformalDQN follows.","marker":"Kondrup et al. 2023"},{"why":"Introduces Conservative Q-Learning, the policy-constraint baseline ConformalDQN is compared against.","marker":"Kumar et al. 2020"},{"why":"Establishes the mechanical ventilation MDP formulation and the 44-feature state representation used here.","marker":"Peine et al. 2021"},{"why":"Provides the Double DQN architecture that ConformalDQN extends with the conformal predictor.","marker":"Van Hasselt, Guez, and Silver 2016"},{"why":"Supplies Fitted Q-Evaluation, the off-policy estimator used to obtain the reported 90-day survival values.","marker":"Le, Voloshin, and Yue 2019"},{"why":"Provides the ICU patient dataset used to build training, calibration, validation, and test cohorts.","marker":"Johnson et al. 2020"},{"why":"Provides the APACHE II score used in the intermediate reward function.","marker":"Knaus et al. 1991"}],"fun_headline_variants":["ConformalDQN lifts 90-day survival to 83.9% — safely","AI ventilation with conformal safety boosts survival to 83.9%","Conformal filter makes AI ventilation safer, lifts survival","Uncertainty-aware AI ventilation reduces risky actions, raises survival","Conformal deep Q-learning for safe mechanical ventilation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The safety claim rests on the assumption that a threshold tuned on typical patients also protects deliberately unusual patients, even though the statistical guarantee only covers patients drawn from the same distribution as the tuning data.","fun_headline_variants_meta":{"raw":{"variants":["ConformalDQN lifts 90-day survival to 83.9% — safely","AI ventilation with conformal safety boosts survival to 83.9%","Conformal filter makes AI ventilation safer, lifts survival","Uncertainty-aware AI ventilation reduces risky actions, raises survival","Conformal deep Q-learning for safe mechanical ventilation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000501,"raw_usage":{"total_tokens":2478,"prompt_tokens":998,"completion_tokens":1480,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":1392}},"tokens_in":614,"tokens_out":1480,"duration_ms":10420,"temperature":1.0,"reasoning_tokens":1392,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:55:44.396712+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Using the paper's calibration set and threshold, compute how often the expert's actual action falls in the confident set on the deliberately unusual out-of-distribution patients; if the rate is well below the target confidence, the out-of-distribution safety mechanism is not doing what is claimed.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the conformal prediction framework and the coverage guarantee that the method's threshold relies on."},{"cited_title":"D.; Precup, D.; and Basu, S","cited_arxiv_id":null,"evidence_quote":"Defines DeepVent, the CQL baseline, and the data preprocessing and intermediate reward scheme ConformalDQN follows."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Conservative Q-Learning, the policy-constraint baseline ConformalDQN is compared against."},{"cited_title":"B.; Schmeink, A.; Ascheid, G.; Thiemermann, C.; Schuppert, A.; Kindle, R.; et al","cited_arxiv_id":null,"evidence_quote":"Establishes the mechanical ventilation MDP formulation and the 44-feature state representation used here."},{"cited_title":"A.; Wagner, D","cited_arxiv_id":null,"evidence_quote":"Provides the APACHE II score used in the intermediate reward function."}],"review_version":1}