{"id":"635496bc-a3fe-468b-94f0-ae3f76af7f1a","arxiv_id":"2506.08030","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MOSS formulates stable rule-set construction as a bi-objective integer program and uses a cutting-plane method to trace the accuracy-stability Pareto frontier, outperforming SIRUS, RuleFit, FIRE, and GLRM on average.","lead":"MOSS is a new optimization method for building short decision-rule lists that balances three goals at once: few rules, accurate predictions, and rules that stay the same when the data is perturbed. It traces the tradeoff between accuracy and stability and, on 30 regression datasets, its best variant ranks higher on average than SIRUS, RuleFit, FIRE, and GLRM when accuracy and stability are combined.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 3.3.1's claim that the rolling-window epsilon sequence E computes the entire Pareto frontier is false; achievable H1 values between rolling sums can hide nondominated points, so EPM returns only a partial frontier.","rationale":"The reader's designated weakest assumption is the H1 proxy for out-of-sample stability, but the most load-bearing concern is the Pareto-frontier completeness claim in §3.3.1. That claim is mathematically false as stated, and it directly supports the abstract and contribution statements about rapidly computing the Pareto frontier. The H1 proxy concern is real but empirically mitigated by the stability rankings in Tables 5 and 9, where MOSS-epsilon-H is second only to SIRUS in stability; it is not the single point most likely to invalidate the central claim. The Proposition 1 regularizer notation inconsistency flagged by the reader is also serious but may be a typographical or notational slip. The Pareto completeness flaw, by contrast, cannot be fixed by notation and is a standalone logical overstatement. This does not destroy the experimental contribution: MOSS-epsilon-H's combined rank could still hold even if the frontier is partial, and the paper's recommended settings do not depend on enumerating every nondominated point. The appropriate response is therefore to keep the reader's CONDITIONAL verdict: the paper needs a corrected and weakened frontier claim, a re-derived Proposition 1, and ideally code before the central claims are fully accepted.","tokens_in":33560,"tokens_out":11740,"duration_ms":127485,"concrete_test":"Run an exhaustive small-scale check using the paper's own pipeline: generate m=8 candidate rules and k=3, draw random Pi, M, and y, enumerate all C(8,3) subsets, compute (H1,H2) for each, and keep the nondominated set. Then run Algorithm 2 with EPM over the E defined in §3.3.1 and compare its output with the exhaustive nondominated set. Repeat over many random seeds, and also try the explicit construction Pi=(0.9,0.7,0.6,0.5), k=2, with ridge columns chosen so that H2({0.9,0.6})=2, H2({0.9,0.5})=1, and all other 2-subsets have H2>2. If any nondominated point is absent from the EPM output, the 'entire Pareto frontier' claim is refuted, and the paper must be revised to claim only a partial frontier or to solve over all achievable subset sums.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central algorithmic promise is to rapidly compute the Pareto frontier between stability (H1) and accuracy (H2). Section 3.3.1 asserts that solving Problem 9 for every epsilon in E, where E is the sequence of sums of k consecutive entries of the descending-sorted selection proportions Pi, yields the entire Pareto frontier with complete granularity. This is not correct. The Pareto frontier is defined over all subsets of size at most k, so its breakpoints occur at arbitrary subset sums of Pi, not only at the m-k+1 rolling-window sums. For example, with k=2 and Pi=(0.9,0.7,0.6,0.5), E={1.6,1.3,1.1}, but the subset {0.9,0.6} has H1=1.5. If H2({0.9,0.6})=2, H2({0.9,0.5})=1, and all subsets with H1>=1.5 other than {0.9,0.6} have H2>2, then {0.9,0.6} is Pareto-optimal: it has higher stability than {0.9,0.5} and lower loss than {0.9,0.7}. Yet solving at epsilon=1.6 makes it infeasible, while solving at epsilon=1.3 returns {0.9,0.5} because that point has lower H2. Thus the Pareto point is missed. Nothing in the structure of H2 prevents such orderings, so the 'entire Pareto frontier' claim in §3.3.1 is not merely unproven; it is false for general instances. The method still produces a useful set of Pareto-optimal solutions, but the paper overstates the completeness of the computed frontier.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MOSS, a multi-objective optimization framework for constructing sparse sets of decision rules that jointly account for stability, accuracy, and sparsity. Stability is measured in-sample as the weighted sum of selection proportions (H1), and accuracy as a ridge-regularized least-squares loss (H2). The framework is formulated as a bi-objective integer program (Problem 5), converted to an epsilon-constraint problem (Problem 9), and solved with an outer-approximation cutting-plane method (Algorithm 1) plus an efficient Pareto method (Algorithm 2) that reuses cutting planes across a sequence of epsilon values. Experiments on 30 OpenML regression datasets compare MOSS variants against SIRUS, FIRE, GLRM, and RuleFit, reporting that MOSS-epsilon-H has the best average combined accuracy-stability rank (3.13).","tokens_in":33837,"tokens_out":12618,"duration_ms":121604,"significance":"The problem addressed is relevant: stability is an underexplored objective in interpretable rule-set learning, and a tool that lets practitioners trade off accuracy and stability in sparse rule sets would be practically valuable. The paper has several strengths: the epsilon-constraint formulation is natural, the convexity argument for H2 is standard and correct, the outer-approximation framework is appropriate for the problem class, the EPM warm-starting idea is sensible, and the experimental comparison covers a broad set of datasets and competing methods. A public code repository is referenced in the appendix. However, the central algorithmic claim that the rolling-window epsilon sequence yields the complete Pareto frontier is false as stated, and the derivation of the main reformulation contains an algebraic error; these issues currently prevent the paper from delivering on its headline promise.","major_comments":[{"comment":"The claim that solving Problem 9 for every epsilon in the rolling-window sequence E computes the entire Pareto frontier 'with complete granularity' is false. A Pareto-optimal set can have an H1 value that is a subset sum of the selection proportions but not a rolling-window sum. For example, with k=2 and Pi=(0.9,0.7,0.6,0.5), E={1.6,1.3,1.1}, but the subset {0.9,0.6} has H1=1.5. If H2({0.9,0.6})=2, H2({0.9,0.5})=1, and every subset with H1>=1.5 other than {0.9,0.6} has H2>2, then {0.9,0.6} is Pareto-optimal; solving at epsilon=1.6 makes it infeasible, while solving at epsilon=1.3 returns {0.9,0.5} because that point has lower H2. Such orderings are compatible with the ridge-regularized loss, since H2 is monotone with respect to set inclusion but not with respect to H1. The completeness statement should therefore be removed or replaced with a precise characterization of when the rolling-window sequence suffices; otherwise the method should be described as computing an approximate or partial Pareto frontier. This does not invalidate the specific epsilon comparisons in the experiments, but it is a load-bearing overstatement of the algorithmic contribution.","section":"§3.1.2 and Appendix A.1, Eq. (8)"},{"comment":"The derivation of the binary reformulation contains an algebraic error. For the objective in Eq. (4), min_w 1/2||y-Mw||^2 + (gamma/2)||w||^2, the stationary point is w*=(M^T M + gamma I)^{-1} M^T y, not (I/gamma - M^T M)^{-1} M^T y as written in Appendix A.1. The resulting reduced loss is 1/2 y^T (I + gamma^{-1} sum_i z_i M_i M_i^T)^{-1} y, whereas Eq. (8) has gamma in place of 1/gamma. Since Problem 9 is built on Eq. (8), the reformulation as stated is not justified. The authors should correct the algebra or redefine the regularization parameter consistently; the same gamma versus 1/gamma inconsistency appears again in Eq. (11) of §3.5.","section":"Algorithm 1, lines 2-3"},{"comment":"The pseudocode for the core cutting-plane algorithm is not executable as written. It initializes nu0 to H2(z0) and then loops while H2(z_t) > nu; at t=0 this condition is false, so the algorithm returns the warm start without adding any cutting plane or solving the ILP. Presumably nu should be initialized to a valid lower bound (for example, -infinity or a bound inherited from a previous epsilon solve in EPM), and the indexing of nu and z should be clarified. This is a central algorithmic description and needs to be corrected for reproducibility.","section":"Algorithm 1, lines 2-3"}],"minor_comments":[{"comment":"The expression for empirical stability is missing a reciprocal factor: the displayed formula T(T-1)/2 times the sum over i != j gives a number much larger than an average when T>2; the intended definition should be (2/T(T-1)) times the sum over i<j of DSC(R_i,R_j).","section":"§2.4 and Appendix A.1"},{"comment":"There are small notational errors: in §2.4 the response variable is said to be in R^m but should be R^n, and in Appendix A.1 the matrix M_s is described as a sub-matrix with 'rows' selected when it should be columns.","section":"§4.2 and Appendix F"},{"comment":"The dataset count is inconsistent: the text says 30 OpenML regression datasets, then mentions '24 out of the 31 datasets', and Appendix F says 'One 24 out of 30'; the list in Appendix D.1 contains 30 datasets, so the count in the main text should be fixed to match.","section":"§5 and Appendix D.2"},{"comment":"The appendix states both that code is available at github.com/brianliu12437/MOSS and, in D.2, that the implementation 'will be open-sourced after the review period'; this should be clarified so readers know what is currently available.","section":"Throughout"},{"comment":"There are several typos and infelicities: 'sample algorithm' in §2.2, 'uncover insights' in §5, 'One 24 out of 30' in Appendix F, and '10 percent increase of 10 R2'. A careful proofread would improve readability.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The Pareto-completeness issue is the most serious concern: the paper's central algorithmic claim is false as stated, but the empirical comparison uses fixed epsilon values and could survive a reframing to approximate frontiers. The algebra error in Proposition 1 is local and fixable. I would encourage the editor to invite a revision rather than reject, provided the authors correct the reformulation and qualify the frontier-completeness claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid but uneven paper. The core idea—treat rule-set construction as a bi-objective integer program over selection proportions and ridge loss, then scalarize with an epsilon-constraint—is new and worth publishing. The cutting-plane reuse across epsilons (EPM) is a nice practical touch, and the 30-dataset comparison is a real empirical contribution. But the theoretical claim in Section 3.3.1 is overblown: solving for the rolling-window epsilon sequence E does not, in general, recover the entire Pareto frontier. The counterexample with k=2 and Pi=(0.9,0.7,0.6,0.5) is valid: the subset {0.9,0.6} has stability 1.5, which is not a rolling-window sum, and it can be Pareto-optimal yet missed by all solves over E. So the paper should stop saying 'entire Pareto frontier with complete granularity' and instead say 'a set of Pareto-optimal solutions spanning the stability range.' The method remains useful, but the advertised completeness is not correct.\n\nTwo other issues need attention. First, the proof of Proposition 1 in Appendix A.1 contains an algebraic error: the minimizer is written as (I/gamma - M^T M)^{-1} M^T y, which has the wrong sign and does not follow from the stated objective. The final formula (8) may be correct under the intended 1/(2γ) penalty, but as written the derivation is not reproducible. Second, the experimental narrative has small inconsistencies: the text says 30 datasets, then 24 out of 31, and Appendix F says 24 out of 30. The code is promised in one place and said to be open-sourced after review in another. These are minor but erode trust.\n\nWhat is genuinely good: the MOSS formulation unifies stability selection and accuracy in a way that SIRUS and RuleFit do not. EPM is a clever reuse of cutting planes across nested problems, and the timing experiments support the scalability claims. The average-rank results are reasonable, and the sensitivity analysis for gamma and k is a useful addition.\n\nRecommendation: send it to peer review. The formulation and empirical results deserve referee time, but the paper needs a major revision to correct the frontier-completeness claim, fix the proof, and clean up the dataset counts. A serious referee should be able to separate the useful contributions from the overclaim.","headline":"MOSS has a genuinely useful formulation and a strong empirical section, but the headline claim about computing the entire Pareto frontier is false as stated, and the proof of Proposition 1 needs correction.","tokens_in":34490,"tokens_out":5258,"would_cite":false,"duration_ms":51025,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["90C11","90C29","68T05","62J07"],"pacs":[],"model":"deepseek-v4-flash","headline":"MOSS jointly optimizes rule-set sparsity, accuracy, and stability in one integer program, and traces the full accuracy-stability Pareto frontier.","keywords":["multi-objective optimization","decision rules","stability","rule ensembles","cutting plane algorithm","Pareto frontier","interpretable machine learning","integer programming"],"falsifier":"Run a bootstrap perturbation study on a held-out set: for each training fold, refit the random forest many times, record the rules MOSS selects at each epsilon value, and measure the average pairwise Dice-Sorensen coefficient of the actual selected rule sets. If solutions with higher $H_1$ do not systematically show higher empirical DSC than solutions with lower $H_1$, or if MOSS's high-epsilon rule sets are not at least as stable as SIRUS's on most of the 30 datasets, the proxy claim is falsified.","tokens_in":1675,"feed_emoji":"⚖️","tokens_out":8861,"duration_ms":134730,"temperature":0.7,"pith_summary":"This paper argues that stability, sparsity, and accuracy for rule sets do not have to be separate modeling choices: all three can be traded off in a single bi-objective integer program. MOSS maximizes an in-sample stability score built from random-forest rule selection frequencies while minimizing regularized training loss, then sweeps a stability threshold to trace the Pareto frontier between accuracy and stability. The paper reports that a specialized cutting-plane method plus a cut-reuse scheme compute this frontier in seconds to minutes on problems where commercial solvers run for hours or fail outright. On 30 public regression datasets, the MOSS rule sets beat state-of-the-art rule-ensemble baselines on a combined accuracy-stability rank.","feed_headline":"Optimizer maps the accuracy-stability trade-off of rule sets","feed_subtitle":"MOSS computes a Pareto frontier for sparse rule sets and beats six baselines on combined accuracy-stability rank.","key_machinery":"The machinery has three parts. First, the in-sample stability objective $H_1(z)=\\sum_i \\Pi_i z_i$ is built from selection proportions $\\Pi_i$, the frequencies with which candidate rules appear across random-forest bootstrap fits. Second, the accuracy objective $H_2(z)=\\frac12 y^\\top (I_n + \\gamma \\sum_i z_i M_i M_i^\\top)^{-1} y$ is regularized in-sample loss re-expressed as a convex function of the binary selection vector $z$. Third, an outer-approximation cutting-plane algorithm solves each epsilon-constrained problem as a sequence of integer linear programs, and an Efficient Pareto Method reuses cutting planes across nested epsilon levels. Convexity of $H_2$ is what makes the cutting planes valid and gives finite convergence of the algorithm.","core_discovery":"The central discovery is that the accuracy-stability trade-off for sparse rule sets can be computed rather than only acknowledged. Concretely, the paper claims that solutions slightly below the stability-selection optimum can be substantially more accurate out-of-sample while retaining nearly the same empirical stability, and that these solutions can be found efficiently by solving a binary integer program for a descending sequence of stability thresholds. The empirical evidence is that MOSS at a high stability setting has the best average combined accuracy-stability rank, 3.13, against 3.51 for stability-only SIRUS and worse ranks for the accuracy-oriented baselines. On 24 of 30 datasets, the MOSS rule sets match SIRUS in empirical stability within one standard error while improving out-of-sample R-squared by about 10 percent on average.","pith_inferences":["If the selection-proportion proxy is trustworthy, the same Pareto machinery should transfer to other rule-generation schemes that produce selection frequencies, such as boosting stumps or randomized split ensembles, not just random forests.","The reported numbers suggest that accuracy-focused baselines sit on the steep low-stability end of the same frontier, so MOSS is effectively letting practitioners pick the knee of a curve that other methods force them to choose one point from.","A testable extension the paper leaves implicit is to replace $H_1$ with an ensemble-averaged holdout Dice-Sorensen estimate inside the objective; the appendix shows ranking insensitivity across stability metrics, but not whether the proxy itself can be improved."],"forward_implications":["Practitioners can choose where to sit on the accuracy-stability frontier instead of committing to a stability-only or accuracy-only rule-construction algorithm.","MOSS at a high stability setting appears to be a strong default: it nearly matches SIRUS on stability on most datasets while delivering a roughly 10 percent R-squared improvement on those datasets.","The stability-selection optimum is far from the whole story; solutions slightly suboptimal in in-sample stability can have much lower in-sample loss and better out-of-sample accuracy without sacrificing measured stability.","The same algorithm scales to candidate pools of thousands of rules beyond the reach of commercial solvers, with the cut-reuse scheme giving up to an order-of-magnitude speedup when computing the full Pareto frontier.","Accuracy-focused rule-set methods pay for their accuracy with dramatically lower stability; MOSS can take about a 2 percent R-squared cost relative to the most accurate baseline while increasing empirical stability by about 190 percent."],"supporting_citations":[{"why":"It supplies the SIRUS random-forest rule-generation scheme that produces the candidate rules and selection proportions, and it serves as the stability-only baseline.","marker":"[3]"},{"why":"It is the RuleFit baseline that selects rules by accuracy alone, illustrating the instability that MOSS aims to fix.","marker":"[14]"},{"why":"It is the FIRE baseline, an accuracy-oriented MCP-penalized rule extraction method, and it supplies the interpretability convention of rule sets with at most 20 rules.","marker":"[21]"},{"why":"It introduces stability selection, whose top-k selection reinterpretation becomes MOSS's in-sample stability objective $H_1$.","marker":"[24]"},{"why":"It is the GLRM baseline, a compact rule-model method built by column generation that MOSS is compared against.","marker":"[31]"},{"why":"It supplies the 30 public regression datasets used in the performance and timing experiments.","marker":"[30]"},{"why":"It establishes finite convergence of the outer-approximation cutting-plane method used by Algorithm 1.","marker":"[13]"}],"fun_headline_variants":["MOSS computes trade-offs in sparse rule sets","Pareto frontier for rule-set accuracy and stability","Optimize rule sets for both accuracy and stability","Cutting-plane algorithm finds Pareto-optimal rule sets"],"cache_read_input_tokens":36352,"weakest_assumption_plain":"The central bet is that the frequency with which a rule appears in random-forest bootstrap fits, measured on the same training data, faithfully predicts how often that rule would appear under fresh data perturbations; if this proxy diverges from the Dice-Sorensen empirical stability, the Pareto frontier is optimizing the wrong quantity.","fun_headline_variants_meta":{"raw":{"variants":["MOSS computes trade-offs in sparse rule sets","Pareto frontier for rule-set accuracy and stability","Optimize rule sets for both accuracy and stability","Cutting-plane algorithm finds Pareto-optimal rule sets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001095,"raw_usage":{"total_tokens":4503,"prompt_tokens":808,"completion_tokens":3695,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":424,"completion_tokens_details":{"reasoning_tokens":3634}},"tokens_in":424,"tokens_out":3695,"duration_ms":23965,"temperature":1.0,"reasoning_tokens":3634,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:45:36.707164+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a bootstrap perturbation study on a held-out set: for each training fold, refit the random forest many times, record the rules MOSS selects at each epsilon value, and measure the average pairwise Dice-Sorensen coefficient of the actual selected rule sets. If solutions with higher $H_1$ do not systematically show higher empirical DSC than solutions with lower $H_1$, or if MOSS's high-epsilon rule sets are not at least as stable as SIRUS's on most of the 30 datasets, the proxy claim is falsified.","supporting_citations":[{"cited_title":"Sirus: Stable and interpretable rule set for classification.Electronic Journal of Statistics, 15:427–505, 2021","cited_arxiv_id":null,"evidence_quote":"It supplies the SIRUS random-forest rule-generation scheme that produces the candidate rules and selection proportions, and it serves as the stability-only baseline."},{"cited_title":"Fire: An optimization approach for fast inter- pretable rule extraction","cited_arxiv_id":null,"evidence_quote":"It is the FIRE baseline, an accuracy-oriented MCP-penalized rule extraction method, and it supplies the interpretability convention of rule sets with at most 20 rules."},{"cited_title":"Generalized linear rule models","cited_arxiv_id":null,"evidence_quote":"It is the GLRM baseline, a compact rule-model method built by column generation that MOSS is compared against."},{"cited_title":"Algorithms for interpretable machine learning","cited_arxiv_id":null,"evidence_quote":"It supplies the 30 public regression datasets used in the performance and timing experiments."},{"cited_title":"Solving mixed integer nonlinear programs by outer approximation.Mathematical programming, 66:327–349, 1994","cited_arxiv_id":null,"evidence_quote":"It establishes finite convergence of the outer-approximation cutting-plane method used by Algorithm 1."}],"review_version":1}