{"id":"c223eade-0247-4806-8c4b-b7b53afac6df","arxiv_id":"2505.13081","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The paper proposes Counterfactual Preference Optimization, a DPO-style method that uses concept-graph-generated counterfactual reasoning trajectories to improve multimodal LLM reinforcement fine-tuning on chest X-ray tasks.","lead":"This paper claims that chain-of-thought reasoning in multimodal large language models drifts during reinforcement fine-tuning, and proposes a counterfactual preference optimization method that uses generated counterfactual reasoning paths to reduce harmful drift. The authors report improved chest X-ray classification, report generation, and zero-shot generalization, and release a large counterfactual reasoning dataset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The CPO loss (Eq. 8) is standard DPO with generated t−; the causal effect ψ in Eq. 4 is never estimated or used, so the claimed disentanglement is not actually enforced.","rationale":"The reader's CONDITIONAL verdict is appropriate, and my pass does not move it. My reading sharpens the required condition: the paper must rule out the null hypothesis that CPO is standard DPO with arbitrary hard negatives. Because Eq. 8 is the DPO objective and ψ never appears, the causal formalism is not doing work in the optimization; therefore the empirical gains, however large, cannot by themselves support the disentanglement claim. A generic-negative ablation is the direct test: if it matches CPO, the concept graph and counterfactual generation are not serving the claimed causal function. I also credit the paper's contribution of the CCF dataset and the consistency of the DPO derivation, which is why I do not recommend rejection; the verdict should remain conditional on this test.","tokens_in":16013,"tokens_out":6171,"duration_ms":66035,"concrete_test":"Ablation: replace the Med-PaLM concept-graph counterfactual negatives in Eq. 8 with generic hard negatives obtained by sampling alternate reports from π_ref or by perturbing only non-diagnostic phrases of the ground-truth report, matched for disease label and length; keep the SFT/RFT pipeline, data size, and hyperparameters fixed. Run 3 seeds and report bootstrap CIs on MS-CXR-T accuracy and MIMIC-CXR generation metrics. If generic negatives match CPO within CI, the disentanglement claim is unsupported because Eq. 8 provides no causal mechanism beyond preference against generated negatives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires CPO to separate beneficial domain adaptation from detrimental concept drift. The formal support for the separation is the causal model of Sec. 2.2 and the effect ψ = E[Z_{T←t,D←d} − Z_{T←t′,D←d}] (Eq. 4). However, the objective actually optimized, Eq. 8, is exactly the DPO loss: L_CPO = −E log σ(β log πθ(t+|v,l)/πref(t+|v,l) − β log πθ(t−|v,l)/πref(t−|v,l)). It contains no do-operator, no adjustment for the backdoor path X←D→Z, and no estimate or use of ψ. The concept graph affects only how the counterfactual negative t− is produced (Appendix B), not how the model is optimized. Consequently, the decoupling is delegated entirely to the assumption that the generated t− trajectories are faithful manifestations of detrimental drift and that rejection of those trajectories removes only harmful signal. If t− are merely plausible alternate reports for the same image, CPO is generic DPO against model-generated hard negatives; the 'disentanglement' claim is then not established. This is a stronger version of the reader's concern: even if the counterfactuals are clinically valid, the loss has no mechanism that distinguishes beneficial adaptation from detrimental drift, so the distinction must be imported through the data-generation process and validated independently.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies reinforcement fine-tuning (RFT) of multimodal large language models (MLLMs) in non-stationary medical settings. It formalizes autoregressive chain-of-thought (CoT) generation as a concept-drift process, introduces a structural causal model {X, Z, T, D} in which a latent drift D confounds inputs and predictions through a backdoor path, and defines a counterfactual effect ψ (Eq. 4) that would isolate the front-door path X→T→Z. To operationalize this, the authors construct a concept graph of 12 disease entities and 53 attributes from MIMIC-CXR using Med-PaLM, generate 320,416 counterfactual reasoning trajectories (the CCF dataset), and train Qwen2.5-VL 7B with Counterfactual Preference Optimization (CPO, Eq. 8), which is a DPO-style loss over positive human trajectories t+ and counterfactual negatives t−. Experiments report gains over prior art on MS-CXR-T classification, MIMIC-CXR report generation, and zero-shot transfer to six chest X-ray benchmarks, plus an ablation isolating CoT and CPO.","tokens_in":16381,"tokens_out":4292,"duration_ms":42766,"significance":"If the disentanglement claim were established, the paper would offer a practical way to stabilize RFT for safety-critical domains, and the CCF dataset would be a reusable resource. The empirical results are large and consistent across several external benchmarks, which is encouraging and goes beyond circular evaluation. However, the paper's central scientific claim—that CPO separates beneficial domain adaptation from detrimental concept drift—is not presently supported: the optimized loss is standard DPO over generated negatives, the defined causal effect is never estimated, and comparisons to prior work are not controlled. The dataset construction is also not validated for clinical plausibility. With additional estimation or validation and controlled baselines, the contribution could become significant; in its current form it is an interesting but not yet demonstrated causal claim.","major_comments":[{"comment":"The defined causal effect ψ is never estimated or used. The loss L_CPO in Eq. (8) is exactly the standard DPO loss with t− produced by Med-PaLM; it contains no do-operator, no backdoor adjustment, and no term involving ψ. Thus the decoupling of beneficial and detrimental drift is not enforced by the optimization objective; it is delegated entirely to the data-generation step. The paper must either estimate and use ψ, or provide independent evidence that the generated t− remove harmful drift while preserving useful domain adaptation; without this, the central claim of the paper is unsupported.","section":"Sec. 2.2, Sec. 2.4, Eq. (4), Eq. (8)"},{"comment":"All reported results are from single training runs; there are no error bars, confidence intervals, or significance tests. The comparisons mix architectures and training data: Table 2 compares a 7B MLLM against specialist vision-language encoders with different backbones and corpora, Table 4 compares against CLIP-based zero-shot models, and Table 3 compares against report generators that do not use CoT. Without same-backbone and same-data baselines (e.g., Qwen2.5-VL SFT/RFT/DPO under identical settings), the large observed margins could be confounded by base-model strength rather than by CPO. The ablation in Table 5 is the most informative comparison but also lacks seed variance and a clear specification of what negatives are used in the DPO-only condition.","section":"Sec. 3, Tables 2-5, Appendix C"},{"comment":"The validity of the counterfactual trajectories is the load-bearing assumption of the method, yet it is not verified. There is no clinician evaluation, no inter-annotator agreement, no automated coherence or plausibility metric, and no analysis of how often the concept-graph perturbation corresponds to actual radiological contradictions. If these trajectories are merely plausible alternate reports, the method reduces to DPO with model-generated hard negatives, and the stated disentanglement is unsupported. The authors should validate the counterfactual generation process, for example by human expert ratings or by measuring whether the generated negatives systematically flip the diagnosis while preserving the original image evidence.","section":"Sec. 2.3, Appendix B"},{"comment":"The report generation metrics improve, but the relation to drift disentanglement remains correlational. The METEOR gain of 34.8% may reflect lexical differences between counterfactual and positive texts rather than reasoning stability; a direct measure of drift—such as token distribution shift during training, or diagnostic consistency of the CoT before and after RFT—is missing. The paper should quantify concept drift directly to substantiate the central phenomenon (Observation 1.1) and its mitigation, since the current metrics only measure output quality, not drift disentanglement.","section":"Sec. 3.2, Table 3"}],"minor_comments":[{"comment":"The abstract and Sec. 3 claim 'statistically significant improvements', but no statistical tests are reported anywhere in the manuscript.","section":"Abstract, Sec. 3"},{"comment":"The symbol ψ is defined but never used again in the paper; clarify its role in the derivation or remove it to avoid implying that the causal effect is estimated.","section":"Sec. 2.2, Eq. (4)"},{"comment":"There is a notation inconsistency: the state is defined as s_j, but the text later says 'a0 is usually the token <think>'; this should be s0 or the state-action notation should be unified.","section":"Sec. 2.4"},{"comment":"The venue for AlignTrans is listed as 'MICCAI’2'; this should be 'MICCAI’21' (or the full citation).","section":"Table 3"},{"comment":"The affiliation contains a typo: 'Faulty of Engineering' should be 'Faculty of Engineering'.","section":"Author affiliation"},{"comment":"The prompt text says 'chest DR examination'; this is likely intended as 'chest X-ray (DR)' or 'chest radiograph', but the expression is ambiguous.","section":"Appendix B"},{"comment":"Several references are incomplete: for example, references [56] and [57] lack venue information, and some reference strings are cut off.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper contains a notable number of self-citations and appears to be a direct follow-up to the authors' prior work on concept drift and causal contrastive learning; the novelty relative to those papers should be clarified. The CCF dataset could be a valuable community resource, but its construction and validation need more scrutiny. The fit for a top ML venue would depend on the authors addressing the major comments, especially the gap between the causal claim and the actual loss, and on obtaining statistically grounded, controlled comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing worth knowing about this paper: it ships a large, reusable counterfactual reasoning dataset for chest X-ray (CCF, 320k trajectories) and shows that DPO against Med-PaLM-generated counterfactual negatives improves a 7B medical MLLM substantially on classification and report generation. The empirical gains are real enough to take seriously. The framing of CoT as a non-stationary token stream is a reasonable lens, and the concept graph gives a principled way to generate negatives that are clinically plausible.\n\nWhere it gets soft is the causal story. The paper defines a structural causal model and an effect ψ in Eq. 4, then never estimates it. The actual objective, Eq. 8, is exactly the DPO loss. The concept graph only affects how the negative trajectory t− is generated, not the optimization. So the claimed decoupling of beneficial adaptation from detrimental drift is not enforced by the method; it is delegated entirely to the assumption that the generated counterfactuals are faithful manifestations of harmful drift. That assumption is worth testing, but the paper does not test it directly. A minimal check would be an ablation against generic hard negatives or randomly perturbed reports. Without that, the disentanglement claim is more rhetoric than result.\n\nOther soft spots: the experiments are single runs, no error bars or significance tests, and claim \"statistically significant\" in the contributions with no statistics behind it. The comparison tables mix architectures and pretraining regimes, though the same-backbone SFT vs. CPO ablation is the clean comparison and it does favor CPO. The self-citations are fine and on-topic; no issue there. The writing is overclaimed in places (\"pioneer\", \"walks the tightrope\"), but that is cosmetic.\n\nWho gets value from this? Anyone doing RL fine-tuning of medical VLMs, and anyone who wants a large counterfactual reasoning benchmark. The dataset alone justifies a serious look. The method is a legitimate DPO variant with model-generated negatives; it just is not the causal intervention the paper claims. That is fixable with honest language and a few ablations.\n\nRecommendation: send it to peer review, but with a clear request to either estimate the causal effect or drop the causal framing, add error bars, and compare against generic negative sampling. The core empirical contribution deserves referee time.","headline":"A useful counterfactual reasoning dataset and a DPO variant that looks strong empirically, but the causal disentanglement story is not actually enforced by the loss.","tokens_in":16838,"tokens_out":1507,"would_cite":true,"duration_ms":17899,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Counterfactual Preference Optimization keeps medical AI reasoning stable during reinforcement fine-tuning.","keywords":["concept drift","reinforcement fine-tuning","counterfactual preference optimization","chain-of-thought reasoning","multimodal large language models","chest X-ray diagnosis","causal inference","MIMIC-CXR"],"falsifier":"Run CPO against a control that uses equally fluent negative reports generated without the concept graph (for example, rewriting the same report with a different diagnosis by prompting the same LLM without the graph), keeping all training settings identical. If the graph-free control matches CPO's accuracy on MS-CXR-T and report metrics on MIMIC-CXR, the concept-graph counterfactual mechanism is not carrying the result; if the control falls short, the causal constraints are responsible.","tokens_in":15797,"feed_emoji":"🩻","tokens_out":10272,"duration_ms":94895,"temperature":0.7,"pith_summary":"During reinforcement fine-tuning (RFT), the chain-of-thought reasoning of a multimodal large language model (MLLM) can drift in ways that look harmless locally but corrupt the final diagnosis. This paper claims that this is a form of concept drift: the token-by-token distribution of the reasoning stream changes over the course of tuning, and the change can be either beneficial domain adaptation or detrimental bias. To separate the two, the authors propose Counterfactual Preference Optimization (CPO), which builds a hierarchical graph of chest-radiology concepts, uses a medical language model to generate plausible counterfactual reasoning trajectories from that graph, and optimizes a preference loss that pulls the model toward the true report and away from the counterfactual one. On chest X-ray benchmarks, CPO is reported to improve classification accuracy, report-generation metrics, and zero-shot generalization relative to supervised fine-tuning and existing baselines, and the paper releases the CXR-CounterFact (CCF) dataset of 320,416 counterfactual reasoning trajectories.","feed_headline":"Counterfactual negatives stabilize medical AI fine-tuning","feed_subtitle":"Training against plausible wrong diagnoses boosts chest X-ray accuracy and report quality.","key_machinery":"The central object is the counterfactual preference optimization loss $\\mathcal{L}_{\\mathrm{CPO}}(\\pi_\\theta;\\pi_{\\mathrm{ref}}) = -\\mathbb{E}_{(v,l,t^+,t^-)}[\\log\\sigma(\\beta\\log\\frac{\\pi_\\theta(t^+|v,l)}{\\pi_{\\mathrm{ref}}(t^+|v,l)} - \\beta\\log\\frac{\\pi_\\theta(t^-|v,l)}{\\pi_{\\mathrm{ref}}(t^-|v,l)})]$, where $t^+$ is the preferred radiologist report and $t^-$ is a counterfactual trajectory. The negatives are generated by a medical-domain large language model expert that consults a hierarchical concept graph of 12 pulmonary diseases, 53 attributes, and association, irrelevance, and exclusion relations, perturbing attributes while keeping the narrative radiologically plausible. A structural causal graph with input $X$, reasoning $T$, prediction $Z$, and latent drift $D$ supplies the interpretation: the counterfactual trajectories are interventions on the mediator $T$ that block the backdoor path $X \\leftarrow D \\to Z$. This object carries the argument because it turns 'detrimental drift' from a metaphor into a training signal.","core_discovery":"The paper's central claim is that the autoregressive chain-of-thought of an MLLM is a non-stationary distribution, so reinforcement fine-tuning should be understood as adaptation to concept drift rather than simple reward maximization. It models each cognitive state as the tokens generated so far together with a latent predicted-outcome distribution, and defines drift as a change in the joint distribution $P_i(t,z)$ across positions. To keep the useful part of adaptation while removing the harmful part, the paper builds a structural causal graph $X \\to T \\to Z$ with a latent drift node $D$ confounding $X$ and $Z$, and uses the counterfactual quantity $\\psi = \\mathbb{E}[Z_{T\\leftarrow t, D\\leftarrow d} - Z_{T\\leftarrow t', D\\leftarrow d}]$ to justify training against concept-graph-constrained counterfactual trajectories. The resulting CPO loss is a direct-preference-optimization-style objective whose negative examples are not disliked human outputs but causal interventions on the reasoning path. The paper claims that this disentangles beneficial distribution adaptation from detrimental concept drift and that its experiments demonstrate improved robustness, generalization, and coordination in RFT.","pith_inferences":["The same recipe—build a domain concept graph, generate counterfactual reasoning paths, and use them as negative preferences—should transfer to any safety-critical domain where a structured ontology of plausible alternatives exists, such as pathology reports, legal reasoning, or autonomous-driving scene descriptions; the paper demonstrates it only for chest radiology.","A direct way to test whether the concept graph is doing the causal work is to ablate it: generate the same number of negative trajectories from the same LLM without graph constraints. If performance does not drop, CPO may simply be a stronger form of preference optimization; if it drops, the graph's relational constraints are what separate beneficial adaptation from detrimental drift.","The formal equivalence between concept drift and CoT token streams suggests that drift detectors could monitor reasoning during fine-tuning directly, flagging the moment when $P_i(t,z)\\neq P_{i+1}(t,z)$ along a trajectory; the paper does not build such a detector, but its definition makes one straightforward to construct."],"forward_implications":["CPO makes reinforced fine-tuning more robust on the chest X-ray classification task: on MS-CXR-T it reports 81.8% top-1 accuracy against 73.8% for supervised fine-tuning alone, with the largest gains on consolidation and pneumonia, the two conditions the paper identifies as drift-prone.","The counterfactual preference signal also improves report generation: CPO reports BLEU-4 of 0.155, ROUGE-L of 0.321, and METEOR of 0.236 on MIMIC-CXR, all above the prior baselines listed in the paper, which the authors read as more coherent and complete clinical reasoning.","Models tuned with CPO generalize zero-shot: across six chest X-ray benchmarks, CPO's AUC is reported to be highest on every dataset, including Open-I, PadChest, ChestXray14, ChestXpert, and ChestXDet10.","The ablations indicate that chain-of-thought and counterfactual preference are complementary: adding only CoT to RFT yields +1.1%, adding only CPO yields +4.5%, and adding both yields +8.0% over SFT on the MS-CXR-T test split."],"supporting_citations":[{"why":"Supplies the direct preference optimization objective that CPO extends with counterfactual negatives.","marker":"[17]"},{"why":"Provides the MIMIC-CXR reports and images used to build the concept graph, the CCF dataset, and the training and test splits.","marker":"[11]"},{"why":"Identifies the medical-domain LLM that extracts the concept graph and generates the counterfactual reasoning trajectories.","marker":"[15, 16]"},{"why":"Supplies the concept drift formalism used to define the CoT token stream as a non-stationary distribution.","marker":"[6]"},{"why":"Provides the structural causal model and backdoor-path reasoning used to frame counterfactual intervention.","marker":"[8]"},{"why":"Supplies the MS-CXR-T benchmark whose test split carries the main classification and ablation results.","marker":"[25]"},{"why":"Supplies the zero-shot baseline that CPO reports improving on across the six chest X-ray benchmarks.","marker":"[44]"}],"fun_headline_variants":["Counterfactual paths stabilize medical AI fine-tuning","Causal counterfactuals cure drift in model tuning","Steady medical fine-tuning via counterfactual negatives","Disentangling drift with counterfactual preference optimization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the counterfactual trajectories produced by the medical language model under the concept graph are faithful examples of detrimental concept drift, so that training the model away from them removes harmful bias without also suppressing useful domain adaptation—if they are simply fluent alternative reports, CPO collapses into ordinary preference optimization and the claimed disentanglement is not demonstrated.","fun_headline_variants_meta":{"raw":{"variants":["Counterfactual paths stabilize medical AI fine-tuning","Causal counterfactuals cure drift in model tuning","Steady medical fine-tuning via counterfactual negatives","Disentangling drift with counterfactual preference optimization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000232,"raw_usage":{"total_tokens":1518,"prompt_tokens":1001,"completion_tokens":517,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":455}},"tokens_in":617,"tokens_out":517,"duration_ms":5645,"temperature":1.0,"reasoning_tokens":455,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:20:05.681829+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CPO against a control that uses equally fluent negative reports generated without the concept graph (for example, rewriting the same report with a different diagnosis by prompting the same LLM without the graph), keeping all training settings identical. If the graph-free control matches CPO's accuracy on MS-CXR-T and report metrics on MIMIC-CXR, the concept-graph counterfactual mechanism is not carrying the result; if the control falls short, the causal constraints are responsible.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MIMIC-CXR reports and images used to build the concept graph, the CCF dataset, and the training and test splits."},{"cited_title":"Causal diagrams for empirical research","cited_arxiv_id":null,"evidence_quote":"Provides the structural causal model and backdoor-path reasoning used to frame counterfactual intervention."},{"cited_title":"Hyland, F","cited_arxiv_id":null,"evidence_quote":"Supplies the MS-CXR-T benchmark whose test split carries the main classification and ablation results."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the zero-shot baseline that CPO reports improving on across the six chest X-ray benchmarks."}],"review_version":1}