{"id":"f565b9f9-644c-454c-8ddc-111f42977251","arxiv_id":"2501.05795","paper_version":4,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A Pareto-based multi-objective optimization method generates counterfactual explanations that aim to improve outcomes across several equally accurate machine learning models simultaneously.","lead":"The paper proposes generating counterfactual explanations by optimizing all outcomes across several equally accurate machine learning models at once, using Pareto multi-objective techniques. The authors report higher robustness and lower cost than single-model baselines, but the current manuscript has an undefined objective and missing results tables.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The optimization objective in §2.1 is mathematically undefined at yt=∞, leaving the proposed MOO method degenerate and the experimental claims irreproducible.","rationale":"I read the paper in good faith and identified the central claim: the proposed multi-objective optimization method (Method 3) is robust and practical for counterfactual explanations under model multiplicity. For this claim to hold, the optimization problem must be well-defined and the experiments must support the reported superiority. The most load-bearing defect is the undefined objective in §2.1 combined with the stated use of yt = ∞ in §3.1. Since loss(∞, f) is infinite for all finite predictions, the formal objective is degenerate and cannot yield the nontrivial Pareto fronts or varied CEs described. The paper's own explanation—that they used predicted values directly—indicates a hidden surrogate objective that is never formalized. This makes the method irreproducible and the empirical results uninterpretable. The reader's weakest assumption about representativeness of the top-m models is a valid secondary concern, but it presupposes that the optimization procedure itself is coherent. The reader's rationale does mention the undefined objective, so there is partial agreement, but the stated weakest assumption did not foreground it. The absence of the actual tables and figures further weakens the evidence, but the undefined objective alone is sufficient to block the central claim. Therefore, I do not change the reader's REJECT verdict, but I emphasize a different, more fundamental reason.","tokens_in":10775,"tokens_out":4188,"duration_ms":39573,"concrete_test":"Request the source code or an exact mathematical statement of the objective used in the NSGA-II runs. Then re-run Experiment 1 with two formalizations: (i) minimize (-f1(Xcf), ..., -fm(Xcf)) subject to d(Xb,Xcf) ≤ C and the stated constraints, and (ii) minimize (loss(T,f1),...,loss(T,fm)) with a large finite T (e.g., 10^6) and compare against the reported values. If neither formalization reproduces the reported rankings or the Pareto fronts in Figures 3–4, the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that Method 3, using multi-objective optimization over the top-m accurate models, produces robust and practical CEs. The formal basis for this is the problem in §2.1: X_cf* = argmin_{X} (loss(yt, f1(Xcf)), ..., loss(yt, fm(Xcf))) with constraints. In §3.1 the authors 'set yt = ∞' and use squared error / absolute error for continuous y. With yt = ∞, loss(yt, f(Xcf)) is infinite for every finite prediction, so every objective is identically infinite; the Pareto front is degenerate and the stated optimization cannot select any particular CE. The subsequent text says that 'since yt = ∞, this study simply used the predicted values of y, where higher values indicated better evaluations,' revealing that the implemented objective must have been something like maximizing f_j(Xcf) or minimizing -f_j(Xcf), but no such objective is formally defined. As written, the method cannot be reproduced, and the reported results—the higher FIR and val/dissim ratios attributed to Method 3—are not tied to the claimed optimization problem. This is more fundamental than the reader's representativeness concern: even if the top-m models were perfectly representative, the procedure that supposedly uses them is not well-defined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method for generating counterfactual explanations (CEs) that are robust under model multiplicity. It formulates the problem as a multi-objective optimization over m models with similar accuracy, defines the desired solution as a Pareto improvement, and proposes to solve it with NSGA-II. The experimental sections compare this method (Method 3) with single-model optimization (Method 1) and a stacking-based weighted objective (Method 2), using simulated data with a known true function and real survey data on academic achievement. The central claim is that Method 3 produces CEs that maintain higher validity/dissimilarity and validity/plausibility ratios, achieve higher True Improvement Ratio, and are therefore both robust and practical.","tokens_in":11083,"tokens_out":7397,"duration_ms":69573,"significance":"The conceptual direction is relevant: applying Pareto improvement to CE robustness under model multiplicity, and extending the setting to regression with constraints, addresses a real gap in the literature. The proposed True Improvement Ratio is also a reasonable simulation-only evaluation idea, since it checks the CE against an independently specified true function. However, the central method as written is not well defined because the objective is stated with an infinite target value, and the empirical evidence is incomplete because several key tables and figures are missing from the manuscript. If the formulation and experiments were corrected, the idea could be worth further study, but in its current form the paper does not support its claims.","major_comments":[{"comment":"The optimization problem in §2.1 is not well defined for the experimental setting. The text states that for continuous y the loss is squared or absolute error between yt and f_j(Xcf), and §3.1 then sets yt = ∞. With yt = ∞, loss(yt, f_j(Xcf)) is infinite for every finite prediction, so all m objectives are identical and constant over the feasible set; no NSGA-II run on these objectives can select one counterfactual over another. The later statement in §3.1 that “this study simply used the predicted values of y, where higher values indicated better evaluations” describes a different objective, such as minimizing -f_j(Xcf), which is never formally defined. As a result, Method 3 is not reproducible from the manuscript, and the reported FIR and balance ratios are not tied to the formalized problem. This is a load-bearing gap because it affects the definition of the proposed method and every experiment that uses it.","section":"§2.1 and §3.1"},{"comment":"Pareto optimality of X_cf with respect to the vector of losses does not imply that X_cf is a Pareto improvement over the base point X_b. A Pareto-optimal solution can have higher loss than f_j(X_b) in every objective when the base point is not in the feasible set, and in general the argmin of the objective vector need not dominate the base input. The paper's phrase “improves consistently in all models” requires a condition such as f_j(X_cf) > f_j(X_b) for all j, or loss(yt, f_j(Xcf)) < loss(yt, f_j(Xb)); no such constraint or post-hoc check is present in §2.1. The experiments should verify per-model improvement over the base, and the formulation should include this condition explicitly.","section":"§2.1 and §2.2"},{"comment":"The selection of the top-m models by test accuracy in Process 3 makes the robustness claim contingent on the representativeness of that particular set, but the paper provides no justification or bound. The True Improvement Ratio in §2.4 is evaluated against the true function only for the same selected models used to generate the CEs; therefore a high TIR shows improvement relative to those selected models and to the true function at those inputs, not robustness to arbitrary equally accurate models in the multiplicity set. A concrete fix would be to hold out several accurate models not used in generating the CEs and report FIR on them, or to provide a finite-sample or Lipschitz-style bound over the set of models within a given accuracy threshold. This is central because the paper's claim is robustness under model multiplicity, not only under the chosen top-m models.","section":"§2.3 and §2.4"},{"comment":"The experimental evidence as presented is incomplete. In the provided manuscript, Tables 3, 4, 5, 7, and 8 contain only captions and no data, and Figures 1–4 appear as captions only, so the reported comparisons of val, dissim, plaus, FIR, and the ratio values cannot be checked. In addition, Table 6 in §3.2 reproduces the same numerical values as Table 1 and includes Model 3 and the stacking model, while the text says the real-data analysis uses Model 1, Model 2, and Model 4; this suggests a copy-and-paste error and makes the real-data experiment unverifiable. These are not cosmetic issues because the central claim that Method 3 could be confirmed as robust rests on those tables.","section":"§3.1, §3.2, and Tables 3–8"}],"minor_comments":[{"comment":"In the definition of Val_j, the notation |yt - f_j(X*)| is used with yt = ∞, which is inconsistent with the sentence explaining that higher predicted values are better; please define Val as an explicit function of f_j(X*) when maximization is intended, and define the corresponding objective in §2.1.","section":"§2.4"},{"comment":"The notation min_{θ ∈ C} F(θ) uses C for a constraint set, but C was previously used for the distance bound in §2.1; the constraint set is also not defined in §2.2. Please use a distinct symbol and define the feasible set precisely.","section":"§2.2"},{"comment":"In the TIR formula, ys is not defined; if it denotes the true function value at the base point, this should be stated explicitly, and the comparison should be tied to f_j(X_b) for each model if the claim is per-model improvement.","section":"§2.4"},{"comment":"The sentence “we set the condition that the improvement is negative when the value is 1 and positive when the value is 0 in T1–T19” is unclear and does not correspond to any constraint form introduced in §2.1; please specify the constraint mathematically.","section":"§3.2"},{"comment":"No implementation details for NSGA-II are given (population size, generations, crossover and mutation rates, seed), and no code or data are provided; given that evolutionary computation is stochastic, these details are necessary for reproducibility.","section":"General"}],"recommendation":"reject","confidential_remarks":"The manuscript has internal inconsistencies and missing experimental artifacts; a future submission would need a complete reformulation of the objective and full data. The editor may also wish to verify that Table 6 is not accidentally copied from Table 1 and that all tables and figures are present before sending the paper to review."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The stress-test note is right, and it lands harder than the reader's representativeness concern. In Section 2.1 the problem is defined as minimizing loss(yt, f1(Xcf)) through loss(yt, fm(Xcf)), where loss is squared or absolute error for continuous y. Then Section 3.1 sets yt = ∞. With yt = ∞, the loss is infinite for every finite prediction, so all m objectives are identically infinite and the Pareto front is degenerate. The paper then says it 'simply used the predicted values of y, where higher values indicated better evaluations'—but no such maximization objective is ever formally defined. The experiments are therefore not tied to the optimization problem in the manuscript. That is a load-bearing flaw, not a minor wording issue.\n\nWhat is actually new: applying the Pareto-improvement lens to CE under model multiplicity, extending to regression and general constraints, and proposing the True Improvement Ratio as an evaluation index. These are reasonable contributions to the explainability subfield. Comparing against single-model CE and a stacked-model baseline is the right kind of experiment. The educational-interventions case study is a good demonstration of constraint flexibility in practice.\n\nWhere it falls apart: besides the undefined objective, virtually all empirical content is missing. Tables 3, 4, and 7 and Figures 1–4 are referenced but none appear in the text. I cannot check the claimed 'consistently higher FIR' or the robustness conclusion. No code or data are provided. And the logic that Pareto optimality implies improvement over the base point is not established—a Pareto-optimal solution in the candidate set need not dominate the base point. So the central claim that Method 3 is robust is unsupported both formally and empirically.\n\nNet: the idea has a kernel worth developing, but this manuscript is not close to publishable. The objective mis-specification alone makes the method irreproducible, and the missing tables and figures mean the experimental claims cannot be scrutinized. I would desk-reject with an invitation to resubmit after fixing the formulation, including the actual numbers, and releasing code.\n\nRecommendation: do not send this to external review in its current form. It would consume referee time without a testable method.","headline":"The idea of using Pareto improvement for counterfactual explanations under model multiplicity is reasonable, but the paper's core objective is undefined, the empirical evidence is missing from the text, and the stated method cannot be reproduced.","tokens_in":11529,"tokens_out":1785,"would_cite":false,"duration_ms":19133,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes generating counterfactual explanations as Pareto-improving solutions over a set of equally accurate models, and reports that the resulting explanations are more robust than single-model or stacked baselines.","keywords":["counterfactual explanations","algorithmic recourse","model multiplicity","Pareto improvement","multi-objective optimization","NSGA-II","explainable artificial intelligence","robustness"],"falsifier":"Take a dataset with a known true function, generate CEs with Method 3 using the top-m models, then compute the True Improvement Ratio on a different model with equal test accuracy that was not among the m; if the held-out TIR is no better than single-model or stacked baselines, the robustness claim over model multiplicity is contradicted.","tokens_in":10598,"feed_emoji":"🎯","tokens_out":12266,"duration_ms":102824,"temperature":0.7,"pith_summary":"The paper claims that a counterfactual explanation can be made robust to model multiplicity by requiring it to Pareto-improve a set of similarly accurate models rather than any single model. It proposes generating such explanations by multi-objective optimization over the vector of prediction losses, one loss per model, and evaluates them with a new True Improvement Ratio that checks how often the explanation improves on the true data-generating function. Experiments on simulated and real data lead the author to conclude that the method is both robust and practical: it achieves higher validity relative to dissimilarity and plausibility, works for regression and classification, allows constraints on the explanation, and produces a diverse set of candidate explanations. The motivation is that in automated machine-learning settings many models can have the same accuracy, and safe decisions in domains such as lending, marketing, and medical care should not depend on which of those models happens to be chosen.","feed_headline":"Pareto optimization keeps counterfactuals valid across models","feed_subtitle":"A single explanation improves all equally accurate models at once, aiding safe decisions in lending, medicine, and marketing.","key_machinery":"The machinery is multi-objective optimization over the vector of model losses, solved with NSGA-II (a fast elitist non-dominated sorting genetic algorithm) implemented through an evolutionary computation library. The vector objective is $F(X_{cf})=(\\mathrm{loss}(y_t,f_1(X_{cf})),\\ldots,\\mathrm{loss}(y_t,f_m(X_{cf})))$, constrained by $d(X_b,X_{cf})\\le C$ and user-supplied equality and inequality constraints; the Pareto set gives diverse candidate explanations. The new evaluation index, True Improvement Ratio (TIR), computes the fraction of generated CEs for which the true function's value improves over the base prediction, and the paper uses average validity-over-dissimilarity and validity-over-plausibility ratios to compare methods.","core_discovery":"The central claim, stated on the paper's own terms, is that Pareto improvement supplies the missing link between counterfactual explanation and model multiplicity. Given $m$ models $f_1,\\ldots,f_m$ with similar test accuracy, the paper defines a robust CE as a solution $X_{cf}^*$ minimizing the vector $(\\mathrm{loss}(y_t,f_1(X_{cf})),\\ldots,\\mathrm{loss}(y_t,f_m(X_{cf})))$ subject to a distance bound $d(X_b,X_{cf})\\le C$ and arbitrary constraints. Because the minimization is multi-objective, the output is a Pareto set rather than a single point, so the explanation improves at least one model's loss without worsening the others. The paper reports that using the top two or three models by accuracy and searching with NSGA-II gives consistently higher True Improvement Ratios than single-model baselines and stacked-model weighting, at lower dissimilarity and plausibility cost, and that the same procedure transfers to a real dataset with intervention-type features.","pith_inferences":["Editorial inference: if Pareto improvement over the top-m models transfers to other equally accurate models, the method could serve as a model-selection-free recourse procedure in automated machine-learning pipelines; the paper does not test held-out models, since the Section 2.4 True Improvement Ratio is computed only for the selected models and a known true function.","Editorial inference: the Pareto-front framing connects CE robustness to social choice; utilitarian selection (maximize average gain) and Rawlsian selection (maximize the worst-case gain) could be tested as preference-based extensions.","Editorial inference: because NSGA-II is stochastic, the Pareto set varies across runs; measuring the stability of the selected medoid CE across random seeds would quantify how much of the reported robustness comes from the method rather than a particular run."],"forward_implications":["A counterfactual that Pareto-improves a set of equally accurate models can be found for regression as well as classification, since the loss vector is defined for continuous targets.","User-specified constraints, such as sign restrictions on features or a budget on the amount of change, can be embedded directly into the optimization, so the robust CE respects real-world feasibility.","Generating a Pareto set instead of a single CE lets a decision maker pick the most acceptable explanation, or use the medoid or centroid as a safe default.","In simulated settings where the true function is known, the Pareto-based method yields a higher fraction of genuinely improving CEs (TIR) than single-model or stacked baselines.","The approach extends to automated machine-learning settings where many models have comparable accuracy, reducing the dependence of recourse on an arbitrary model choice."],"supporting_citations":[{"why":"Establishes the problem of counterfactual explanations under predictive multiplicity and the trade-off between cost and robustness that the proposed method targets.","marker":"[22]"},{"why":"Defines a CE that is consistent across all models using verification on white-box classifiers, the closest prior formulation that this paper extends to regression and constraints.","marker":"[17]"},{"why":"Proposes argumentative ensembling to extract universal CEs under model multiplicity, an alternative approach that the direct multi-objective method is compared against.","marker":"[13]"},{"why":"Provides the NSGA-II multi-objective genetic algorithm that the proposed Method 3 uses to generate Pareto-improving CEs.","marker":"[5]"},{"why":"Supplies the Python multi-objective optimization implementation used to run NSGA-II in the experiments.","marker":"[2]"},{"why":"Supplies the literature review and benchmarking definitions of proximity, plausibility, and constraints used in the evaluation framework.","marker":"[10]"},{"why":"Provides the review of counterfactual explanations and algorithmic recourse that supplies the Validity index definition and general background.","marker":"[27]"},{"why":"Describes the COBYLA direct-search optimizer used to generate the single-model and stacked baselines in Methods 1 and 2.","marker":"[24]"}],"fun_headline_variants":["Counterfactuals robust to model multiplicity via Pareto gains","One explanation that works for every equally accurate model","Multi-objective counterfactuals beat single-model baselines","Pareto improvement for robust counterfactuals under model multiplicity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the top-m models selected by test accuracy stand in for all equally accurate models, so a counterfactual that Pareto-improves them is assumed to also improve the true outcome and any other model outside the chosen set, yet the Section 2.4 True Improvement Ratio checks improvement only for those selected models and no held-out model validation is provided.","fun_headline_variants_meta":{"raw":{"variants":["Counterfactuals robust to model multiplicity via Pareto gains","One explanation that works for every equally accurate model","Multi-objective counterfactuals beat single-model baselines","Pareto improvement for robust counterfactuals under model multiplicity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0009,"raw_usage":{"total_tokens":3859,"prompt_tokens":911,"completion_tokens":2948,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":2883}},"tokens_in":527,"tokens_out":2948,"duration_ms":24675,"temperature":1.0,"reasoning_tokens":2883,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:06:29.633011+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a dataset with a known true function, generate CEs with Method 3 using the top-m models, then compute the True Improvement Ratio on a different model with equal test accuracy that was not among the m; if the held-out TIR is no better than single-model or stacked baselines, the robustness claim over model multiplicity is contradicted.","supporting_citations":[{"cited_title":"Pawelczyk, K","cited_arxiv_id":null,"evidence_quote":"Establishes the problem of counterfactual explanations under predictive multiplicity and the trade-off between cost and robustness that the proposed method targets."},{"cited_title":"Leofante, E","cited_arxiv_id":null,"evidence_quote":"Defines a CE that is consistent across all models using verification on white-box classifiers, the closest prior formulation that this paper extends to regression and constraints."},{"cited_title":"Jiang, A","cited_arxiv_id":null,"evidence_quote":"Proposes argumentative ensembling to extract universal CEs under model multiplicity, an alternative approach that the direct multi-objective method is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the NSGA-II multi-objective genetic algorithm that the proposed Method 3 uses to generate Pareto-improving CEs."},{"cited_title":"Blank and K","cited_arxiv_id":null,"evidence_quote":"Supplies the Python multi-objective optimization implementation used to run NSGA-II in the experiments."},{"cited_title":"Guidotti","cited_arxiv_id":null,"evidence_quote":"Supplies the literature review and benchmarking definitions of proximity, plausibility, and constraints used in the evaluation framework."},{"cited_title":"Verma, V","cited_arxiv_id":null,"evidence_quote":"Provides the review of counterfactual explanations and algorithmic recourse that supplies the Validity index definition and general background."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes the COBYLA direct-search optimizer used to generate the single-model and stacked baselines in Methods 1 and 2."}],"review_version":1}