{"id":"2bde8d5b-c7c6-4418-a8d8-7fb80b29fd88","arxiv_id":"2412.17158","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The authors introduce and demonstrate MOODE, an R package for finding experimental designs that optimize compound criteria combining estimation, lack-of-fit detection, and mean-squared-error robustness.","lead":"MOODE is a new R package that searches for experimental designs balancing several goals at once, such as precise estimation, detecting when the assumed model is wrong, and protecting estimates from model errors. The paper demonstrates the package on two case studies, including a comparison with the widely used Plackett-Burman design.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'nearly optimal' claim is not verified: Table 3's 100.54% LoF-DP efficiency proves the LoF-DP benchmark is a local optimum, so the reported efficiencies and the package's near-optimality demonstration are unreliable without exact or far more extensive search.","rationale":"My reading: the paper is a software introduction, so the central claim is that the package works and provides near-optimal compound designs. The methodology is mostly standard and the code is available, which is real evidence. However, the manuscript's own results undercut the optimality part: the 100.54% LoF-DP efficiency in Table 3 is not a harmless artifact; for a deterministic minimization criterion it signals that the 'optimal' benchmark design is not optimal. This directly substantiates the reader's concern about exchange heuristics. It also affects the interpretation of all efficiency tables, because the normalizing denominator may not be a true optimum. The Plackett-Burman appendix appears to include a spurious 'trt' column in X1, making the baseline comparison suspect. These issues do not by themselves show the package is unusable; the criteria and algorithms may be sound, and the paper does not need a mathematical guarantee of global optimality to be useful. But they do show that the evidence for 'nearly optimal' is incomplete and that the case-study code as printed is not reproducible. I therefore keep the conditional verdict, with the condition being independent verification of optimality (or explicit reporting of optimization gaps) and corrected, runnable code.","tokens_in":17287,"tokens_out":8039,"duration_ms":75499,"concrete_test":"Recompute the LoF-DP column of Table 3 with the LoF-DP benchmark obtained from an exact global search for a small reduced version of the case study (e.g., k=2, n=12 on the 3-level grid), or from Nstarts=10,000 rather than the default. If the new LoF-DP optimum has a lower criterion value than the row-5 design, the reported 100.54% efficiency will fall below 100% and the original benchmarks are confirmed to be local optima; the paper should then report margins to the exact optimum. Additionally, rerun the Section 4.2 analysis with X1 restricted to intercept plus main-effect columns only (omitting 'trt'), and verify whether the PB efficiencies and Figure 1 change.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that MOODE returns 'nearly optimal' designs rests on exchange heuristics whose output is never benchmarked against a global optimum. The paper itself provides internal evidence that the heuristics stop at poor local optima. In Table 3, the first compound design has 100.54% efficiency under the LoF-DP criterion. Since efficiency is normalized by the LoF-DP-optimal design found in row 5 (kappa=(0,1,0)), a value above 100% is impossible if that benchmark design is truly optimal. The text attributes the anomaly to 'the empirical nature of the design optimisation,' but the correct reading is that the algorithm failed to find the true LoF-DP optimum. All efficiencies in Table 3 are therefore relative to possibly suboptimal benchmarks, and the claimed trade-off demonstration is unquantified. The same issue is admitted in Section 4.2, where for k=7 the algorithm found a better LP design while searching under a compound criterion than when directly optimizing LP, 'indicating the difficulty sometimes found in finding pure error optimal designs.' Because no exact verifier (enumerative, MILP, or branch-and-bound) is applied to any output, the manuscript provides no evidence that the package's designs are near-optimal for the compound criteria it is designed for. A second, code-level issue compounds this: the appendix Plackett-Burman code builds X1 with a 'trt' column before the intercept and factors, so the PB baselines in Figure 1 are evaluated under a misspecified model matrix; if this changes the PB efficiencies, the comparison's conclusions are not reproducible.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MOODE, an R package for constructing experimental designs under compound optimality criteria that combine inference-based criteria (DS, L, (DP)S, LP), lack-of-fit criteria (LoF-DP, LoF-LP), and mean-squared-error robustness criteria (MSE(DS), MSE(L)). The package implements point-exchange and coordinate-exchange algorithms and is demonstrated on two case studies: a 3-factor, 36-run response surface experiment and a comparison with Plackett-Burman designs for k=3 to 9 factors in 12 runs. The paper's central claim is that MOODE provides a practical, open-source tool for finding nearly optimal designs that balance estimation, lack-of-fit detection, and bias protection.","tokens_in":17587,"tokens_out":9351,"duration_ms":80523,"significance":"If the package performs as claimed, it would fill a practical gap by making multi-objective design methods from Egorova and Gilmour (2022) and Gilmour and Trinca (2012) accessible to experimenters. The open-source implementation and the two case studies are useful demonstrations of the intended functionality. However, the paper currently does not provide adequate evidence for the 'nearly optimal' claim: the only internal evidence consists of exchange-algorithm outputs, and one reported efficiency exceeds 100%, which indicates that the benchmark designs are themselves local optima. The code in the appendix also contains several errors that prevent the Plackett-Burman comparison from being reproducible. These issues are load-bearing because the paper's demonstrations and package credibility rest on the reliability of the reported efficiencies and the correctness of the implemented criteria.","major_comments":[{"comment":"The near-optimality claim is not supported. In Table 3, the first compound design reports a LoF-DP efficiency of 100.54%, which is impossible if the row-5 design (kappa=(0,1,0)) is truly LoF-DP-optimal. The text attributes this to 'the empirical nature of the design optimisation,' but the correct implication is that the row-5 benchmark is a local optimum. The same issue is admitted in Section 4.2, where for k=7 the algorithm found a better LP design under a compound criterion than when directly optimizing LP. Because no exact verifier (enumeration, MILP, or branch-and-bound) is applied to any reported design, the paper provides no evidence that the package's outputs are near-optimal for the compound criteria, and the efficiency trade-offs in Table 3 and Figure 1 are not reliable benchmarks.","section":"Section 4.1, Table 3"},{"comment":"The code for the Plackett-Burman comparison is incorrect and non-reproducible. In the loop, the variable `pb` is first assigned a list via `pb <- list()`, and then the same name is called as a function in `undesign(pb(nruns = 12, nfactors = j))`, which would error in R because a list is not callable. Additionally, `mutate(trt = 1:12, intercept = rep(1, 12), .before = A)` inserts observation-number and intercept columns before the factor columns, so `X1` contains these extraneous predictors; `model.matrix(~ (.)^2, X1)` then builds interactions involving `trt` and `intercept`, and `[, -(1:(j + 1))]` removes the wrong columns. As a result, the Plackett-Burman baseline designs in Figure 1 and Table 5 are evaluated under a misspecified model, and the printed code cannot be run as shown.","section":"Section 4.2 and Appendix A.2"},{"comment":"The text and code disagree on the Monte Carlo sample size and the criterion used for the MSE(DS) benchmark. The text states that 'we use B = 1000 Monte Carlo samples' for the MSE(DS) component, but the code sets `control = list(Biter = 50)` and `criterion.choice = \"MSE.P\"`, which is the point-prior criterion (10), not the Monte Carlo criterion (9). The benchmark 'MSE(DS)-optimal design' in row 6 of Table 3 is therefore optimal for a different objective function than the one used to compute the efficiency column, so the 100% value in that column does not establish optimality under the reported criterion. Please reconcile the text, code, and evaluation.","section":"Section 4.1 and Appendix A.1"},{"comment":"Equation (8) appears to assume that the columns of X1 sum to zero. From the definitions of M and A1, the term A1' M A1 equals X2' (I - J/n) X1 M^{-1} X1' (I - J/n) X2, not X2' X1 M^{-1} X1' X2 as written, unless (I - J/n)X1 = X1. The paper does not state a centered-coding assumption, and the example designs (e.g., Table 4) are not all balanced. This simplification underpins the MSE(DS) criterion, so the formula should be corrected or the centering assumption made explicit.","section":"Section 2.3, Eq. (8)"}],"minor_comments":[{"comment":"The loop is written as `for(i in 1:nrow(kappa1))`, but the matrix is defined as `kappa` in the preceding code block; as printed, the code will fail because `kappa1` is undefined.","section":"Section 4.1, code chunk"},{"comment":"The sentence 'It is possible these design could be improved further do using more random starts of the algorithm' contains grammar errors; it should be 'It is possible these designs could be improved further by using more random starts of the algorithm.'","section":"Section 4.1, text"},{"comment":"In the LoF-DP and LoF-LP rows, the symbol L is used both as a matrix (in the trace expressions) and as part of the criterion names, which is confusing. In Section 2.2 the matrix is called R; please align the notation.","section":"Table 1"},{"comment":"The notation for the (DP)S criterion alternates between phi_D and phi_DS; please use one symbol consistently.","section":"Section 2.1"},{"comment":"The claim that MOODE provides 'the first accessible, open source, implementation of a multi-objective approach to the problem' is strong. Please soften it to 'to our knowledge' or substantiate it with a systematic comparison of existing software capabilities.","section":"Section 5, Discussion"}],"recommendation":"major_revision","confidential_remarks":"The heavy self-citation is not circularity because the criteria are evaluated against external benchmarks such as Plackett-Burman designs and individual-criteria designs; however, the paper's demonstrations would be considerably strengthened by an external benchmark or an exact-verification study for at least one small case. The code issues in the appendix are serious enough that the authors should be asked to provide a clean, runnable script (e.g., as a supplementary file) to ensure reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The MOODE package is a genuine contribution: it is the first open-source implementation of the Egorova-Gilmour compound criteria, and it fills a real gap for practitioners who want to balance estimation, lack-of-fit, and MSE robustness in response-surface designs. The formulas in Section 2 are standard and correctly presented, and the two case studies show the package working in realistic settings. That value should be credited.\n\nThe soft spots are concentrated in the demonstrations. First, the Plackett-Burman code in Appendix A.2 has a clear bug: when building X1 for the PB designs, it adds a `trt` column (1:12) and an `intercept` before the factors. The model.matrix call then includes `trt` as a predictor, so the PB baselines in Figure 1 are evaluated under a misspecified model. That changes the efficiencies and makes the PB comparison non-reproducible. This is not a typo; it needs to be fixed and the figure regenerated.\n\nSecond, Table 3 shows a compound design with 100.54% LoF-DP efficiency. That is impossible if the LoF-DP benchmark in row 5 is truly optimal. The text blames 'the empirical nature of the design optimisation,' but the correct reading is that the exchange algorithm failed to find the true LoF-DP optimum. So the efficiencies in Table 3 are relative to a suboptimal benchmark, and the paper's 'nearly optimal' claim is unverified. The paper itself admits the same problem in Section 4.2 for k=7. There is no exact verifier anywhere, so we don't know how close any of these designs are to optimal. This is a moderate but real issue.\n\nThird, there are text-code contradictions (B=1000 in text vs Biter=50 in code; kappa1 vs kappa) and no commit hash or version number. These are minor but should be cleaned up. The 'first accessible open source implementation' claim is not backed by a systematic survey, but I wouldn't hang the paper on that.\n\nThe methodology itself is solid, and the package is useful. The flaws are in the validation, not in the criteria. A serious referee would want the PB bug fixed, the benchmarks re-run against a global optimum or at least a much larger set of starts, and the code/text aligned. I'd send this to peer review rather than desk reject — it's a real contribution that needs revision, not rejection. Bring it to the reading group if you want a concrete example of how exchange heuristics can quietly fail.","headline":"A useful software package with a real bug in the Plackett-Burman comparison and an unverified near-optimality claim; the package itself is worth refereeing.","tokens_in":18179,"tokens_out":3227,"would_cite":true,"duration_ms":28242,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62K05","62K15"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper presents MOODE, an open-source R package that builds experimental designs by directly optimizing a user-weighted compound criterion covering precise estimation, lack-of-fit detection, and protection against model bias, with case…","keywords":["optimal design of experiments","compound criteria","response surface design","lack of fit","model misspecification","point exchange algorithm","coordinate exchange algorithm","R package"],"falsifier":"Solve a small instance exactly. For example, with $k=2$ two-level factors, $n=8$ runs, a main-effects primary model, and all two-factor interactions as potential terms, enumerate every multiset of eight runs (or use a global optimizer) and compute the compound criterion (12). If MOODE's best design over its default 10 starts has a criterion value more than, say, 1% worse than the global optimum, the near-optimality claim fails for that instance; repeating across several seeds and instances would show whether the failure is systematic.","tokens_in":17067,"feed_emoji":"🎯","tokens_out":7079,"duration_ms":56778,"temperature":0.7,"pith_summary":"This paper presents MOODE, an R package for constructing experimental designs when several goals compete. It combines criteria for precise estimation of a primary model, detection of lack of fit from that model, and protection against bias from unmodeled terms into one weighted compound criterion. The package searches for exact designs that are near-optimal under this compound criterion, using point or coordinate exchange algorithms. Two case studies show the trade-offs: designs optimized for a single objective score poorly on the others, while compound designs keep high efficiencies on all three. The paper's claim is that this makes a multi-objective approach to experimental design accessible and usable in practice.","feed_headline":"MOODE finds experimental designs that balance three competing goals","feed_subtitle":"Open-source R package combines inference, lack-of-fit, and bias criteria into one weighted design search.","key_machinery":"The machinery is the compound optimality criterion built as a product of weighted efficiencies. Estimation and inference criteria ($DS$, $L$, $(DP)S$, $LP$) are based on the centred information matrix $X_1^\\top(I_n - J_n/n)X_1$ and incorporate $F$-quantiles that depend on pure-error degrees of freedom $d = n - t$, which encourages replication. Lack-of-fit criteria use the posterior variance of potential terms, $\\Sigma_2 = \\sigma^2(R + \\tau^{-2}I_q)^{-1}$, with $R$ the Schur complement for $X_2$ given $X_1$, and target designs with $X_1$ and $X_2$ near-orthogonal. MSE-robustness criteria minimize the determinant or trace of the MSE matrix of $\\hat{\\beta}_1$, with the expected log-determinant evaluated either by Monte Carlo or a point prior. The search is carried out by a modified Fedorov point-exchange algorithm (default for $k \\le 4$ factors) or a coordinate-exchange algorithm (default for $k \\ge 5$), with multiple random starts.","core_discovery":"The paper's claim is that MOODE provides the first accessible, open-source implementation of a multi-objective approach to optimal experimental design. Rather than optimizing a single alphabetic criterion, the package minimizes a weighted product of individual criteria: one component for precise estimation and inference under a primary polynomial model, one for detecting lack of fit in the direction of potential higher-order terms, and one for mean-squared-error robustness against bias from those potential terms. The compound criterion is $\\phi_{\\rm det}(D) = \\phi_{(DP)S}^{\\kappa_{DP}}(D)\\,\\phi_{\\rm LoF-DP}^{\\kappa_{\\rm LoF-DP}}(D)\\,\\phi_{\\rm MSE(DS)}^{\\kappa_{\\rm MSE(D)}}(D)$, with a trace-based analogue, and the weights are chosen by the user. The paper demonstrates that designs found under such compound criteria achieve high efficiencies on all three individual criteria simultaneously, whereas designs optimized for any single criterion have poor performance on at least one other, and that in 12-run two-level cases the compound designs maintain pure-error replication that Plackett-Burman designs lack.","pith_inferences":["The real burden on users moves to choosing the weights $\\kappa$ and the prior scale $\\tau^2$; the paper gives defaults but no guidance for eliciting them from experimental priorities, and sensitivity to those choices could be explored systematically.","The Monte Carlo variant of $\\rm MSE(DS)$ with the default $B=50$ samples may be noisy, so a user comparing two designs could mistake Monte Carlo error for a real efficiency difference; increasing $B$ or using the point-prior variant removes this.","Because the exchange algorithms are heuristic, the 'nearly optimal' claim is only as good as the number of random starts; a practical extension would be a built-in diagnostic comparing the best design's criterion value with the distribution over starts.","The compound approach could be adapted to nonlinear or Bayesian design settings, where multi-objective trade-offs are currently handled by separate criteria such as T-optimality or Bayesian D-optimality."],"forward_implications":["Experimenters can obtain, in one R function call, an exact design that explicitly trades off precise estimation, lack-of-fit detection, and protection against model bias; the weights $\\kappa$ make the trade-off user-controlled.","Designs produced under compound criteria will routinely include replicated treatments, because the $F$-quantile components reward pure-error degrees of freedom, which supports model-robust inference.","The package should outperform existing single-criterion tools for response-surface problems with model uncertainty, since those tools focus only on precision-based criteria.","In screening settings with few runs, compound designs can beat the classic Plackett-Burman design on efficiency for the assumed model while gaining the ability to estimate pure error.","Because the code is modular, the same compound-criterion machinery can be extended to blocked designs, hybrid nonlinear models, or other optimization algorithms."],"supporting_citations":[{"why":"Proposes the compound criteria class that the package implements and supplies the methodology and first case study.","marker":"Egorova and Gilmour (2022)"},{"why":"Introduces the inference-focused criteria $(DP)S$ and $LP$ with $F$-quantiles, motivating replication for pure error.","marker":"Gilmour and Trinca (2012)"},{"why":"First formulated response-surface design as a balance between variance and bias, giving the encompassing-model idea used here.","marker":"Box and Draper (1959a)"},{"why":"Provides the Bayesian prior specification for potential terms, $\\beta_2 \\sim N(0, \\sigma^2 \\tau^2 I_q)$, used in the sensitivity and MSE criteria.","marker":"DuMouchel and Jones (1994)"},{"why":"Supplies the modified Fedorov point-exchange algorithm used for $k \\le 4$ factors.","marker":"Cook and Nachtsheim (1980)"},{"why":"Supplies the coordinate-exchange algorithm used for $k \\ge 5$ factors.","marker":"Meyer and Nachtsheim (1995)"},{"why":"Defines the 12-run design family used as the comparison benchmark in the second case study.","marker":"Plackett and Burman (1946)"},{"why":"Provides the generalized D- and A-criteria included in the package and related model-sensitivity criteria.","marker":"Goos et al. (2005)"},{"why":"Characterizes the MSE matrix combining variance and bias that the robustness criteria minimize.","marker":"Montepiedra and Fedorov (1997)"}],"fun_headline_variants":["R package MOODE balances three experimental goals in one design","Multi-objective design search in R: MOODE combines three criteria","MOODE: find designs that optimize inference, lack-of-fit, and bias","Open-source R tool for designs that satisfy multiple objectives","MOODE optimizes experimental designs for several competing aims"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the point-exchange and coordinate-exchange heuristics, when started from the default number of random designs, actually find designs that are nearly optimal for the compound criterion; the paper states that convergence to the global optimum is not guaranteed, so if the algorithms routinely stop at poor local optima, the package's central promise fails.","fun_headline_variants_meta":{"raw":{"variants":["R package MOODE balances three experimental goals in one design","Multi-objective design search in R: MOODE combines three criteria","MOODE: find designs that optimize inference, lack-of-fit, and bias","Open-source R tool for designs that satisfy multiple objectives","MOODE optimizes experimental designs for several competing aims"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000183,"raw_usage":{"total_tokens":1274,"prompt_tokens":864,"completion_tokens":410,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":480,"completion_tokens_details":{"reasoning_tokens":325}},"tokens_in":480,"tokens_out":410,"duration_ms":4125,"temperature":1.0,"reasoning_tokens":325,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:44:26.509630+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Solve a small instance exactly. For example, with $k=2$ two-level factors, $n=8$ runs, a main-effects primary model, and all two-factor interactions as potential terms, enumerate every multiset of eight runs (or use a global optimizer) and compute the compound criterion (12). If MOODE's best design over its default 10 starts has a criterion value more than, say, 1% worse than the global optimum, the near-optimality claim fails for that instance; repeating across several seeds and instances would show whether the failure is systematic.","supporting_citations":[{"cited_title":"Optimal response surface designs in the presence of model contamination","cited_arxiv_id":"2208.05366","evidence_quote":"Proposes the compound criteria class that the package implements and supplies the methodology and first case study."},{"cited_title":"Optimum design of experiments for statistical inference (with discussion)","cited_arxiv_id":null,"evidence_quote":"Introduces the inference-focused criteria $(DP)S$ and $LP$ with $F$-quantiles, motivating replication for pure error."},{"cited_title":"A simple B ayesian modification of D -optimal designs to reduce dependence on an assumed model","cited_arxiv_id":null,"evidence_quote":"Provides the Bayesian prior specification for potential terms, $\\beta_2 \\sim N(0, \\sigma^2 \\tau^2 I_q)$, used in the sensitivity and MSE criteria."},{"cited_title":"A Comparison of Algorithms for Constructing Exact D -optimal Designs","cited_arxiv_id":null,"evidence_quote":"Supplies the modified Fedorov point-exchange algorithm used for $k \\le 4$ factors."},{"cited_title":"The coordinate-exchange algorithm for constructing exact optimal experimental designs","cited_arxiv_id":null,"evidence_quote":"Supplies the coordinate-exchange algorithm used for $k \\ge 5$ factors."},{"cited_title":"The design of optimum multifactorial experiments","cited_arxiv_id":null,"evidence_quote":"Defines the 12-run design family used as the comparison benchmark in the second case study."},{"cited_title":"Model-robust and model-sensitive designs","cited_arxiv_id":null,"evidence_quote":"Provides the generalized D- and A-criteria included in the package and related model-sensitivity criteria."},{"cited_title":"Minimum bias designs with constraints","cited_arxiv_id":null,"evidence_quote":"Characterizes the MSE matrix combining variance and bias that the robustness criteria minimize."}],"review_version":1}