{"id":"da8cc1f7-52f8-4d36-ba9d-acffc62763f6","arxiv_id":"2508.10455","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"RealAC generates counterfactual explanations by matching pairwise feature dependencies via mutual information and applying a user-defined immutability mask, but the reported performance gains are not uniformly supported by its own tables.","lead":"This paper proposes RealAC, a counterfactual explanation generator that uses a variational autoencoder to keep feature pairs statistically consistent with the original data while letting users freeze immutable attributes. It reports improvements over existing methods on realism metrics, but several experimental choices and metric definitions undercut the headline claim.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Synthetic 1 results rely on an undocumented LAMBDA SUM term for X6+X7=constant, so the domain-agnostic dependency-preservation claim is not supported by the reported experiments.","rationale":"The reader's weakest_assumption identifies the core risk: pairwise MI matching is not sufficient for per-instance dependency preservation. My reading of the manuscript supports this, and the appendix supplies direct evidence: Table 4 includes a LAMBDA SUM term specifically 'for X6 + X7 = k' in Synthetic 1, while the Methodology section defines only L_dep via MI differences. The Limitations section acknowledges L_dep fails for this exact constraint. Thus the experimental results on Synthetic 1, the dataset designed to test nonlinear dependencies, were produced with an auxiliary, undocumented loss that injects explicit domain knowledge. This undercuts both the 'domain-agnostic' selling point and the reproducibility of the headline numbers. The differentiability of the histogram-based MI estimator is a second, independently serious gap, but the LAMBDA SUM issue is more directly fatal to the central claim because it demonstrates the method's failure mode on a simple deterministic relation. I credit the authors for releasing code and for a thoughtful limitations section that names the insufficiency; however, the paper does not reconcile that limitation with the presence of a dataset-specific fix in the experiments. A single re-run with LAMBDA SUM removed would settle whether the reported results depend on it. If they do, the central claim fails and REJECT is appropriate; if they do not, the paper still needs to explain the differentiability mechanism and the discrepancy. I agree with the reader's verdict.","tokens_in":17139,"tokens_out":5292,"duration_ms":54460,"concrete_test":"Run the official code (github.com/Arefeen06088/RealAC) for Synthetic 1 with the same hyperparameters as Table 4 but with LAMBDA SUM set to 0 (or the corresponding term removed). If validity and DPS drop materially from the reported 0.996 and 0.719, the reported performance depends on domain-specific knowledge and the central claim fails. Also inspect the training code to confirm whether LAMBDA SUM is present in the loss.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that RealAC preserves inter-feature dependencies automatically, without explicit domain knowledge, by aligning pairwise mutual information via L_dep (Methodology, Feature Dependency). However, the appendix's hyperparameter table (Technical Appendix, Optimizer/V AE Description, Table 4) lists 'LAMBDA SUM = 2.0, for X6 + X7 = k' for Synthetic 1. This term is absent from the method section's L_total and from the description of L_dep; it enforces the deterministic constraint X6+X7=constant using explicit knowledge of the data-generating process. The Limitations section concedes that L_dep alone is insufficient for this constraint. Thus the reported Synthetic 1 DPS=0.719, CES=-6.558, and related claims were obtained with an unstated domain-specific loss, contradicting the abstract's 'without relying on explicit domain knowledge.' Additionally, L_dep as defined operates on batch-level co-occurrence histograms, so it matches the empirical joint distribution across the batch, not per-instance dependencies. This is precisely why the constant-sum constraint, which is per-instance, fails. The differentiability assertion is also unsupported: the indicator-function histogram estimator has zero gradient almost everywhere, unless some smoothing not described is used. The load-bearing issue is that the method as described cannot, in general, preserve deterministic per-instance dependencies in a domain-agnostic way, and the one dataset that tests nonlinear dependencies was run with a hidden auxiliary term.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"RealAC is a VAE-based counterfactual generation framework. It proposes to preserve inter-feature dependencies by matching pairwise mutual information between factual and counterfactual batches (L_dep), to enforce user-specified immutability via binary masks, and to optimize flip, proximity, and KL losses. The paper introduces a Dependency Preservation Score (DPS) and compares RealAC against several CF baselines and LLM-based SenseCF on three synthetic and two real tabular datasets, claiming consistent superiority in causal edge score, DPS, and IM1. The abstract claims that RealAC is domain-agnostic and automatically preserves complex dependencies without explicit domain knowledge.","tokens_in":17529,"tokens_out":4346,"duration_ms":43652,"significance":"The motivation is solid, and a domain-agnostic method for dependency-preserving counterfactuals would be valuable. The code release and ablation studies are positive features. However, the central claim is not supported by the manuscript as written. The differentiability of L_dep is asserted without a viable estimator, the reported Synthetic 1 results rely on an undocumented domain-specific LAMBDA SUM term that is absent from the method and is conceded in the Limitations to require explicit domain knowledge, and the headline outperformance claim is directly contradicted by Table 1. Because these issues concern the core methodology and its evaluation, the contribution cannot be accepted in its current form.","major_comments":[{"comment":"The paper defines the joint distribution estimator using indicator functions 1[x_n,i in b_p] and then states: 'Importantly, this regularizer is differentiable with respect to the perturbation.' Indicator-based histograms have zero gradient almost everywhere, so gradient-based optimization of L_dep is not defined as written. If soft binning, a straight-through estimator, or a kernel approximation is used, it must be specified. This is load-bearing because the entire realism mechanism depends on optimizing L_dep.","section":"Methodology, Feature Dependency"},{"comment":"Table 4 lists 'LAMBDA SUM = 2.0, for X6 + X7 = k' for Synthetic 1, but the L_total equation in Methodology contains no such term. The Limitations then state that L_dep alone is insufficient for X6+X7=constant and that capturing this constraint 'requires explicit use of domain knowledge or structural priors.' Therefore the reported Synthetic 1 results (DPS=0.719, CES=-6.558) were obtained with an unstated domain-specific loss, contradicting the abstract's claim of preserving dependencies 'without relying on explicit domain knowledge.' This is a fundamental mismatch between the described method and the reported experiments.","section":"Technical Appendix Table 4 and Limitations"},{"comment":"The abstract claims RealAC 'outperforms state-of-the-art baselines and Large Language Model-based counterfactual generation techniques in causal edge score, dependency preservation score, and IM1 realism metric.' Table 1 contradicts this for DPS: SenseCF(o3) achieves DPS=0.793 on Synthetic 1 and 0.768 on Synthetic 2, both higher than RealAC's 0.719 and 0.709. For CES (closer to zero is better), SenseCF(4o) on Synthetic 1 (-4.658) is closer to zero than RealAC (-6.558), and CEILS on Diabetes (0.159) is closer than RealAC (0.278). The claim of consistent outperformance is not supported by the paper's own tables.","section":"Abstract and Table 1"},{"comment":"The dependency preservation constraint is stated per instance: ρ(x_cf_i, x_cf_j) ≈ ρ(x_i, x_j). However, L_dep is computed from batch-level co-occurrence histograms and therefore matches average pairwise statistics over a batch, not per-instance dependencies. A batch-level match can hold while individual counterfactuals violate the dependency, which is exactly what happens with the X6+X7=k constraint. The paper should clarify that the method preserves distribution-level pairwise statistics, not per-instance causal or structural dependencies.","section":"Problem Formulation and Methodology"},{"comment":"DPS is introduced in this paper as a headline metric. It measures how well generated CFs follow known conditional relationships via regression residuals—precisely the kind of pairwise dependency structure L_dep is designed to enforce. Without independent validation of DPS against human judgment or an established realism benchmark, the evaluation is partly circular: the method is rewarded for matching the objective it was trained on. The paper should either validate DPS independently or de-emphasize it in favor of metrics with established validity.","section":"Validation Metrics, DPS definition"}],"minor_comments":[{"comment":"Typo: 'binzry target variable' should be 'binary target variable.'","section":"Technical Appendix, Dataset Description"},{"comment":"Inconsistent spacing in 'V AE' and 'C-CHV AE'; use 'VAE' consistently.","section":"Throughout"},{"comment":"The legend includes 'ExAct' but this method is not described in the baselines list or the results tables. Either add a description or remove it.","section":"Figure 4"},{"comment":"The sentence 'λ_fixed > λ_mse should theoretically suppress change in immutable features and allow controlled mutation in the rest' appears as an isolated fragment; it should be integrated into the surrounding discussion.","section":"Methodology, Proximity"},{"comment":"No standard deviations or confidence intervals are reported. Given the variability of CF generation and the small margins on some datasets, statistical significance of the claimed advantages is unclear.","section":"Tables 1 and 2"},{"comment":"Minor typo: 'income' is misspelled as 'inome' in the prompt template.","section":"Technical Appendix, Adult prompt"}],"recommendation":"reject","confidential_remarks":"The paper has a useful code release and a sensible motivation, but the central 'domain-agnostic' claim is not reproducible as described because of the LAMBDA SUM mismatch for Synthetic 1. In addition, the differentiability statement for L_dep is unsupported. The claimed outperformance is also contradicted by the paper's own Table 1. These are not local presentation issues; they concern the core method and its evaluation. I would encourage the authors to revise substantially, either by removing the domain-agnostic claim and documenting all constraints transparently, or by re-running experiments without the extra term and honestly reporting the resulting failure cases."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nRealAC has one genuinely new idea: aligning pairwise mutual information between factual and counterfactual batches as a dependency-preservation regularizer, plus a binary mask for immutable features. That is a reasonable contribution to the counterfactual-explanation literature, and the paper ships code and a clear ablation on lambda_dep. The masking mechanism is simple but effective, and the writing on actionability is fine.\n\nThe problems are in the evidence, and they are load-bearing. The method section defines L_total without any LAMBDA SUM term, but the appendix hyperparameter table lists LAMBDA SUM = 2.0 for X6+X7=k for Synthetic 1. The Limitations section then concedes that L_dep alone is insufficient for exactly that constraint. So the one dataset designed to test nonlinear per-instance dependencies was run with a hand-added, dataset-specific loss term that is absent from the stated method. That directly contradicts the abstract's claim of working \"without relying on explicit domain knowledge.\"\n\nSecond, the abstract says RealAC outperforms baselines in DPS, but Table 1 shows SenseCF(o3) gets higher DPS on both Synthetic 1 (0.793 vs 0.719) and Synthetic 2 (0.768 vs 0.709). The CES claim also varies by dataset. So the headline empirical superiority is not supported by the paper's own tables.\n\nThird, the regularizer is described as differentiable, but the estimator shown is a histogram of indicator functions, whose gradients are zero almost everywhere. Unless the implementation uses some smoothed binning (not described), that assertion does not hold.\n\nNone of these are mathematical errors in the core idea; they are reporting gaps and overclaims. But they matter because the central claim is domain-agnostic dependency preservation, and the strongest evidence for that comes from a configuration that includes explicit domain knowledge. The idea could still be right, and the pairwise-MI regularizer might work with proper soft-histogram estimation and honest ablations. As it stands, the paper does not support its claims.\n\nWho should read it: people working on counterfactual generation and evaluation will find the MI alignment idea worth thinking about, and the ablation is useful. But the paper should not be taken as evidence for the method's effectiveness until the hidden term is either removed and results re-run, or fully integrated and disclosed with the differentiability question resolved.\n\nI'd send this to peer review but with clear instructions to the authors to address the constraint-term discrepancy and the differentiability issue. The idea deserves a hearing; the current evidence does not.\n\nBest,","headline":"Interesting pairwise-MI regularizer, but the hidden constraint term on Synthetic 1 undercuts the domain-agnostic claim.","tokens_in":17957,"tokens_out":4023,"would_cite":false,"duration_ms":39077,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RealAC generates realistic counterfactual explanations by matching pairwise mutual information between factual and counterfactual features, without needing domain knowledge.","keywords":["counterfactual explanations","actionable recourse","inter-feature dependencies","mutual information","variational autoencoder","domain-agnostic","dependency preservation","tabular data"],"falsifier":"Run RealAC with only L_dep (no extra penalty) on a dataset that has a known deterministic functional constraint such as X6 + X7 = constant, and check whether the generated counterfactuals violate that constraint; or measure the causal edge score on counterfactuals where L_dep is perfectly matched to see whether per-instance causal edges are still broken.","tokens_in":17085,"feed_emoji":"🔍","tokens_out":9761,"duration_ms":80203,"temperature":0.7,"pith_summary":"The paper claims that realistic and actionable counterfactual explanations—what-if explanations that say which minimal feature changes would flip a model's prediction—can be generated without hand-coded causal knowledge by aligning the mutual information of every pair of features between the original data and the counterfactual. RealAC is a variational-autoencoder-based generator that optimizes this pairwise dependency loss alongside label-flip, proximity, and user-specified 'freeze' constraints. Across three synthetic and two real tabular datasets, RealAC outperforms existing counterfactual methods and LLM-based generators on causal edge score, the proposed dependency preservation score, and the IM1 realism metric. If correct, the result suggests that pairwise statistical alignment is a cheap, domain-agnostic route to structurally plausible what-if explanations.","feed_headline":"Matching pairwise mutual information yields realistic counterfactuals","feed_subtitle":"RealAC preserves inter-feature dependencies without domain knowledge and beats LLM-based counterfactual generators.","key_machinery":"The load-bearing object is the pairwise dependency loss L_dep: features are discretized into B bins, empirical joint distributions are estimated from co-occurrence histograms, and the loss is the average absolute difference in mutual information between each feature pair in the factual batch and the counterfactual batch. This mechanism claims to preserve complex, nonlinear inter-feature relations without any explicit domain knowledge. A separate binary actionability mask m merges immutable factual features with optimizer updates via x_cf = m ⊙ x_0 + (1-m) ⊙ x̂, keeping frozen attributes fixed during optimization.","core_discovery":"The paper's central claim is that preserving pairwise mutual information—not explicit structural equations or causal graphs—is sufficient to keep counterfactuals realistic. RealAC estimates the joint distribution of each feature pair by binning and co-occurrence histograms, then minimizes the absolute difference in mutual information between the factual and counterfactual pairs (L_dep). This regularizer is differentiable and plugs into a VAE whose total loss also enforces prediction flip, proximity, and a binary actionability mask that freezes immutable features. On Synthetic 1/2, Diabetes, Sangiovese, and Adult, RealAC reports the best causal edge score, dependency preservation score, and I","pith_inferences":["Because L_dep is a batch statistic, it may be insensitive to per-instance constraints; the paper's own X6+X7 example shows a deterministic relation that escapes it, so a robust deployment should pair the method with lightweight explicit rules for such pre-identified invariants.","The approach suggests a testable general principle: pairwise MI alignment may be enough when dependencies are pairwise and smooth, but higher-order or deterministic constraints require higher-order terms; this could be checked by systematically adding ternary MI terms.","LLM-based counterfactual generation (SenseCF) performed poorly on prediction flipping without classifier access; this points to a division of labor where LLMs propose edits and optimization-based methods validate or refine them.","The freeze-mask mechanism could be reused in other generative models (GANs, diffusion) for constrained generation, not just VAEs."],"forward_implications":["If RealAC's central claim holds, counterfactual generators no longer need causal graphs or hand-crafted constraints; statistical alignment of feature pairs can substitute.","The same framework should transfer to any tabular dataset with unknown or heterogeneous feature relations, including categorical features.","End-users get a practical control: features they mark immutable stay fixed, making the suggested recourse actions actually feasible.","The dependency preservation score gives researchers a simple quantitative target for evaluating structural realism.","RealAC's speed advantage over most baselines makes it usable in interactive settings where explanations must be generated on demand."],"supporting_citations":[{"why":"Supplies the causal edge score metric, the Diabetes and Sangiovese datasets, and the prior paradigm of preserving causal constraints that RealAC extends to a domain-agnostic setting.","marker":"Mahajan, Tan, and Sharma 2019"},{"why":"Supplies Synthetic 1 and Synthetic 2 datasets with nonlinear dependencies and a learned-relations baseline that RealAC positions itself as improving on.","marker":"Xiang and Lenskiy 2022"},{"why":"Provides the C-CHVAE baseline and the VAE/ELBO formulation on which RealAC's base optimizer is built.","marker":"Pawelczyk, Broelemann, and Kasneci 2019"},{"why":"Supplies the DiCE baseline and the diversity/proximity objectives that RealAC must match while adding dependency preservation.","marker":"Mothilal, Sharma, and Tan 2019"},{"why":"Supplies the CEILS baseline, a wrapper that injects causal relations, which RealAC aims to beat without requiring explicit structural priors.","marker":"Crupi et al. 2022"},{"why":"Supplies the MCCE baseline, a Monte Carlo generative method that RealAC compares against for realism.","marker":"Redelmeier et al. 2024"},{"why":"Supplies the NICE nearest-instance baseline, representing a non-generative alternative RealAC must outperform.","marker":"Brughmans and Martens 2021"},{"why":"Supplies the IM1 realism metric used as a primary success criterion.","marker":"Looveren and Klaise 2019"},{"why":"Supplies the SenseCF LLM-prompting baseline that RealAC outperforms in flipping predictions and realism.","marker":"Soumma et al. 2025"}],"fun_headline_variants":["Realistic counterfactuals without domain knowledge","Data-driven dependency preservation for counterfactuals","RealAC: Realistic and actionable counterfactuals","Mutual information keeps counterfactuals feasible","Domain-agnostic framework for realistic counterfactuals"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"RealAC's 'domain-agnostic' claim rests on the assumption that matching batch-level pairwise mutual information (L_dep) is enough to preserve per-instance causal and structural dependencies; the paper's own Synthetic 1 example (X6 + X7 = constant) shows this fails without an extra explicit penalty term.","fun_headline_variants_meta":{"raw":{"variants":["Realistic counterfactuals without domain knowledge","Data-driven dependency preservation for counterfactuals","RealAC: Realistic and actionable counterfactuals","Mutual information keeps counterfactuals feasible","Domain-agnostic framework for realistic counterfactuals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000217,"raw_usage":{"total_tokens":1283,"prompt_tokens":768,"completion_tokens":515,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":445}},"tokens_in":512,"tokens_out":515,"duration_ms":5487,"temperature":1.0,"reasoning_tokens":445,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T20:25:33.537816+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run RealAC with only L_dep (no extra penalty) on a dataset that has a known deterministic functional constraint such as X6 + X7 = constant, and check whether the generated counterfactuals violate that constraint; or measure the causal edge score on counterfactuals where L_dep is perfectly matched to see whether per-instance causal edges are still broken.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the C-CHVAE baseline and the VAE/ELBO formulation on which RealAC's base optimizer is built."},{"cited_title":"K.; Sharma, A.; and Tan, C","cited_arxiv_id":null,"evidence_quote":"Supplies the DiCE baseline and the diversity/proximity objectives that RealAC must match while adding dependency preservation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CEILS baseline, a wrapper that injects causal relations, which RealAC aims to beat without requiring explicit structural priors."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MCCE baseline, a Monte Carlo generative method that RealAC compares against for realism."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the NICE nearest-instance baseline, representing a non-generative alternative RealAC must outperform."},{"cited_title":"B.; Arefeen, A.; Carpenter, S","cited_arxiv_id":null,"evidence_quote":"Supplies the SenseCF LLM-prompting baseline that RealAC outperforms in flipping predictions and realism."}],"review_version":1}