{"id":"04932956-7c4c-42cc-a8c3-300c2c01b40d","arxiv_id":"2507.12862","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"The paper applies entropy and KL-divergence weighting to the variance of ethical attributes in simulated command, illustrating with a two-attribute autonomous-weapon example.","lead":"This paper proposes replacing human-in-the-loop ethical weighting in military simulations with automatic, information-theoretic weight calculations based on how much each ethical attribute varies across scenarios. It is a small methodological contribution, but one of its three proposed methods is mathematically defective, which weakens the paper's central demonstration.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ICW weights are inverted relative to the paper's own premise: attributes with similar variance across alternatives receive the highest weight, not the lowest; IGD is degenerate.","rationale":"Reading the paper in good faith, its goal is to make automated T&E weighting of ethical attributes possible by deriving weights from the variance of utilities across situations. For the central claim to hold, the weights must at least be consistent with the paper's own motivating premise in Section III-A: 'An attribute should not be weighted high if it has similar variance on all courses of actions.' The ICW formula is not. Because p(m,r) is a distribution over alternatives, equal variances give uniform p and maximal entropy, and ICW(r)=E(r)/sum_j E(j) therefore awards the highest weight to the attribute with the most similar variances across alternatives. The toy example confirms the inversion: Force Protection's normalized variances (0.74/0.26) are closer to uniform than Proportionality's (0.2/0.8), and ICW gives Force Protection the higher weight (0.54 vs 0.46). This is an internal inconsistency, not an external validation gap. The IGD formula has an independent algebraic degeneracy (it equals sum_j E(j) - (R-1), a constant, so its weights are always uniform), but the ICW direction error is more fundamental because it defeats the paper's primary data-driven weighting scheme. A reader relying on the paper's prose would expect the opposite ordering. I therefore agree with the REJECT verdict, though for a different reason than the reader's weakest_assumption about metric validity. My concern is on the correctness of the proposed formulas themselves.","tokens_in":10084,"tokens_out":12791,"duration_ms":135024,"concrete_test":"Compute ICW (Section III-C) on a two-attribute, two-alternative synthetic case where Attribute A has equal variances across alternatives (p=(0.5,0.5)) and Attribute B has unequal variances (p=(0.9,0.1)). If the formula assigns A the higher weight while Section III-A states such an attribute 'should not be weighted high,' the paper's weight direction is internally contradicted; no simulation or external data is needed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-C defines p(m,r) as the variance of alternative m normalized over alternatives for attribute r, and ICW(r) as the normalized Shannon entropy of this distribution. A uniform p (equal variances across alternatives) maximizes E(r); therefore ICW awards the largest weight to the attribute whose variances are most similar across alternatives. This directly contradicts Section III-A's premise: 'An attribute should not be weighted high if it has similar variance on all courses of actions.' The toy example confirms the inversion: Force Protection's normalized variances (0.74/0.26) are more balanced than Proportionality's (0.2/0.8), and Table III gives Force Protection the higher ICW weight (0.54 vs 0.46). Moreover, IGD(r)=E(r)-sum_{j != r}(1-E(j)) simplifies to sum_j E(j) - (R-1), a constant independent of r, so IGDW is always uniform regardless of data. The central claim that variance-derived weights reflect discriminatory power is thus implemented by a primary formula that reverses the paper's stated principle and a secondary formula that is vacuous.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes to automate the weighting of ethical attributes in simulation-based test and evaluation. It defines a mean matrix Γ and a variance matrix Λ over alternatives and situations, then derives three information-theoretic weighting schemes from Λ: ICW (Shannon entropy of normalized variances), IGH (Kullback-Leibler divergence from a human-supplied prior), and IGD (a difference between an attribute's entropy and a function of the remaining entropies). A two-attribute, two-alternative toy scenario is used to illustrate the weights and the resulting scenario rankings. The authors conclude that the results demonstrate the importance of incorporating human judgment into the decision-making process.","tokens_in":10281,"tokens_out":9201,"duration_ms":107505,"significance":"If the proposed methods were valid, they would address a relevant problem in multi-criteria decision analysis and simulation-based testing: how to aggregate ethical attributes without requiring human input at each simulation step. The paper is transparent about its key assumptions, including that ethical metrics exist and that utilities encode subjective judgment, and it provides a fully worked numerical example. However, the central methodological claims are not established. The primary entropy-based weight formula rewards the opposite of the paper's stated principle; the IGD formula is degenerate and cannot differentiate attributes; and the IGH-based conclusion about human judgment follows by construction rather than from empirical evidence. The paper does not provide machine-checked proofs, reproducibility artifacts, or a comparison to existing entropy-weighting methods, so the positive contribution is limited to a clearly stated problem formulation and a reproducible arithmetic illustration.","major_comments":[{"comment":"The ICW formula is inverted with respect to the paper's own premise. Section III-C defines p(m,r) as the variance of alternative m for attribute r normalized over alternatives, and E(r) as the Shannon entropy of p. Since entropy is maximized by a uniform distribution, an attribute whose normalized variances are closest to being equal across alternatives receives the highest ICW. This is precisely the situation that Section III-A says should receive low weight: 'An attribute should not be weighted high if it has similar variance on all courses of actions.' The toy example confirms the inversion: Force Protection has normalized variances (0.74, 0.26) and Proportionality has (0.2, 0.8); after normalization, Force Protection is closer to the uniform distribution, and Table III assigns it the higher ICW weight (0.54 vs 0.46). Thus, the claim that ICW reflects discriminatory power is not supported by the formula as written.","section":"Section III-A, Section III-C, Table III"},{"comment":"The IGD method is mathematically degenerate. Since IGD(r) = E(r) - sum_{j != r} (1 - E(j)) = sum_j E(j) - (R - 1), the value is independent of r for any number of attributes R. Consequently, IGDW(r) = 1/R for every attribute whenever the denominator is nonzero, so the method can never differentiate attributes. The appendix confirms this: Table A5 reports identical IGD values of 0.0767 for both attributes, and Table III reports IGDW weights of 0.50 and 0.50. The IGD method as defined is vacuous and should either be removed or redefined with a formula that actually depends on r.","section":"Section III-C, Appendix A5, Table III"},{"comment":"The paper's conclusion that 'the results reveal the importance for incorporating human judgement into the decision making process' is circular. IGHW(r) is, by definition, the normalized Kullback-Leibler divergence between the simulation-driven distribution p(.,r) and the human prior q(.,r). An attribute that disagrees with the prior is therefore guaranteed to receive a high weight, and an attribute that agrees with the prior receives a low weight; if p = q, the attribute receives zero weight and is eliminated from the aggregation. The ranking difference between IGHW and the data-only methods in Table III is a direct consequence of this definition, not an empirical finding. To support the conclusion, the authors would need an independent criterion, such as ground-truth ethical outcomes or decision-maker validation, showing that divergence from the prior corresponds to attribute importance.","section":"Section III-C, Section IV-B"}],"minor_comments":[{"comment":"Section III-A states that utilities are normalized to U : (r,s) -> [0,1], but Table I reports means of 15, 8, 4, and 10 and variances up to 7.0, which cannot arise from a [0,1]-valued utility. Please rescale the example or explicitly relax the normalization assumption.","section":"Section III-A, Table I"},{"comment":"The logarithm base is inconsistent: Section III-C uses log base 2 in the ICW definition, while Appendix A3 uses natural logarithms; the IGH definition in Section III-C is written with an unspecified log, and Appendix A4 uses base-10 logarithms. Because the weights are normalized, the final weights are invariant to the base, but the reported intermediate entropy and IGH values do not match the formulas as written.","section":"Section III-C, Appendix A3, Appendix A4"},{"comment":"The appendix describes IGD as 'comparing the entropy of each attribute to the average informativeness of the others,' but the formula in Section III-C is E(r) - sum_{j != r} (1 - E(j)), which is neither an average nor a difference with the average entropy of the other attributes. The description and the formula should be aligned.","section":"Appendix A5"},{"comment":"The paper does not discuss the degenerate case in which all variances of an attribute are zero; in that case p(m,r) is undefined because the denominator of the normalization is zero. A brief note on handling zero total variance would make the method more robust.","section":"Section III-C"}],"recommendation":"reject","confidential_remarks":"The manuscript is very short for a journal paper and reads like an extended abstract. The core formulas contain load-bearing errors: ICW reverses the stated variance-discrimination principle, IGD is degenerate, and the IGH conclusion is definitional. These are not local presentation issues, and I would not encourage a resubmission without a fundamental redesign of the proposed methods."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this paper is not sound as submitted. Its main proposed method, ICW, assigns the highest weight to the attribute whose variances are most similar across alternatives, which is exactly what Section III-A says should receive the lowest weight. The toy example confirms it: Force Protection's normalized variances (0.74/0.26) are more balanced than Proportionality's (0.2/0.8), yet Force Protection gets the higher ICW weight (0.54 vs 0.46). The stated premise is that an attribute with similar variance across courses of action should be weighted less; Shannon entropy does the opposite. That is not a subtle disagreement, it is a direct internal contradiction in the central contribution.\n\nWhat the paper does well: the writing is clear, the assumptions are stated honestly, and the toy example is fully worked out in the appendix, which makes the errors easy to catch. The conceptual move of separating the variance matrix Lambda (weight derivation) from the mean matrix Gamma (decision scoring) is a reasonable framing, even if it is a relabeling of the input to standard entropy weighting. The IGH method, KL divergence against a human prior, is arithmetically correct and could be a legitimate way to inject human judgment into automated T&E.\n\nWhere it falls apart: IGD, the supposedly data-only method, reduces to a constant independent of the attribute, so IGDW is always uniform regardless of data. Table A5 shows both IGD values equal at 0.0767, but the paper still presents this as a working method. The IGH conclusion that \"human judgment matters\" is built into the definition: if you measure divergence from a human prior, you will find divergence from that prior. That is circular as a demonstration, though not as a method. There are also minor technical sloppiness issues: logarithm bases differ across the main text and appendix, and IGH is undefined when the prior has zero mass where the data has positive mass, which is not discussed. The use case has no validation, error bars, or comparison against expert-judgment weights, so there is no evidence the resulting weights are useful in practice.\n\nThe metric-design assumption is flagged honestly as an assumption, not hidden, and for that reason I would not call the paper deceptive. But the load-bearing formulas do not support the claims, and one of them is vacuous. A referee could salvage the IGH idea and possibly fix ICW by using entropy as a measure of non-discriminatory power (i.e., inverting it), but that would be a different paper.","headline":"The paper's central ICW formula reverses its own stated principle, and IGD is mathematically degenerate; this is not sound as submitted, though the framing of separating variance information from mean information is a useful seed.","tokens_in":10806,"tokens_out":2673,"would_cite":false,"duration_ms":31683,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that ethical-attribute weights in simulated command decisions can be derived automatically from each attribute's variance across situations, moving human judgment outside the simulation loop.","keywords":["ethical AI","multi-criteria decision making","entropy weighting","variance-based weights","simulation-based test and evaluation","autonomous weapon systems","information gain","Kullback-Leibler divergence"],"falsifier":"Collect a corpus of simulated command situations where a known ethically decisive attribute is constant across all situations for every course of action; if the proposed weights give that attribute zero weight, an externally validated judgment would show the weighting contradicts what human commanders consider decisive.","tokens_in":9854,"feed_emoji":"⚖️","tokens_out":5817,"duration_ms":60265,"temperature":0.7,"pith_summary":"This paper tries to solve the weighting problem inside simulation-based test and evaluation of AI command systems when the attributes being weighted are ethical. The core claim is that an attribute's second central moment—how much its utility varies across situations for the same course of action—carries the information needed to set its weight, so weights are computed from the variance matrix Lambda while decisions are scored with the mean matrix Gamma. Three low-cost information-theoretic formulas are offered: an entropy weight (ICW), a Kullback-Leibler weight against a human prior (IGH), and an entropy-difference weight (IGD). If this is right, simulated environments can explore huge scenario spaces and come back to the human commander with a few ranked options, without a human weighting every in-simulation decision. The paper's working premise, stated openly, is that the ethical measurement problem is solved and the utilities already embed the commander's subjective importance.","feed_headline":"Variance, not means, sets ethical weights in simulated command","feed_subtitle":"Three formulas use each attribute's variance across situations to weight ethics, keeping humans out of the simulation loop.","key_machinery":"The load-bearing objects are two matrices built from normalised utilities U(m,r,s): Gamma holds first central moments (mean utility of each alternative for each attribute) and Lambda holds second central moments (variance of that utility across situations). From Lambda the paper forms p(m,r), the relative variance of alternative m for attribute r, and then three weight families: ICW uses Shannon entropy of p, IGH uses Kullback-Leibler divergence between p and a human-supplied prior q, and IGD uses the difference between an attribute's entropy and the average entropy of the others. These formulas convert 'how much an attribute's utility wobbles across situations' into a normalised weight, and they are what let a simulation score alternatives without a human in the per-decision loop.","core_discovery":"The central discovery the paper argues for is that the variability of an ethical attribute across situations is itself an information source about how much that attribute should count, and that this information should be separated from the mean performance used to make decisions. Weights are therefore to be derived from Lambda, the matrix of per-alternative, per-attribute variances of normalised utilities, while Gamma, the matrix of mean utilities, is reserved for computing expected utility and ranking courses of action. Three concrete aggregation formulas are proposed: ICW normalises the entropy of each attribute's relative variance distribution; IGH computes the Kullback-Leibler divergence between the simulation-driven variance distribution and a subjective prior supplied by the human; and IGD computes the difference between an attribute's entropy and the average entropy of the other attributes. In the synthetic autonomous-weapon use case, the three methods yield different weights and can reverse which scenario is preferred, which the paper presents as evidence that the choice of information source and prior matters.","pith_inferences":["Beyond the paper, the same variance-as-weight principle could be applied to any intangible attribute in simulation-based evaluation—risk, trust, or mission impact—not only ethical attributes, wherever a defensible utility mapping exists.","Beyond the paper, the method suggests a testable calibration loop: collect human-assigned weights for a corpus of simulated command decisions and compare them with ICW/IGD weights to see whether variance tracks what commanders actually treat as informative.","Beyond the paper, the three formulas could be embedded inside an autonomous system as an internal deliberation step, so an agent re-derives its ethical trade-off weights on the fly; the paper mentions this possibility but does not develop it.","Beyond the paper, the IGH variant offers a way to monitor value drift: large and persistent divergence between the human prior and simulation-driven probabilities would flag that the metric space or the prior no longer matches the situations being generated."],"forward_implications":["Simulation-based test and evaluation can rank courses of action by ethically weighted expected utility without a human setting weights for each scenario, leaving human judgment to metric design before simulation and option choice after it.","An ethical attribute whose utility barely varies across situations is automatically downweighted, because it carries little discriminatory information; an attribute that fluctuates strongly in complex situations is upweighted.","The IGH variant lets a commander encode a subjective prior on attribute importance before simulation, and the resulting weights are pulled toward that prior only when simulation data diverge from it.","Because ICW, IGH, and IGD can reverse the preferred scenario in the paper's use case, the weight-derivation rule itself is part of the ethical assessment and must be chosen deliberately.","Using Lambda for weights and Gamma for decisions resolves the known limitation that classic entropy weights punish variability in alternatives rather than rewarding discriminatory power across situations."],"supporting_citations":[{"why":"Supplies the caveat that entropy weights reflect variability rather than subjective importance, motivating the paper's separation of Lambda and Gamma.","marker":"[22]"},{"why":"Provides the Kullback-Leibler/information-gain basis adopted for the IGH weighting scheme.","marker":"[25]"},{"why":"Grounds the multi-attribute decision analysis context and the TOPSIS-style ranking that the proposed weights feed into.","marker":"[4]"},{"why":"Reviews explicit attribute-weighting methods in multi-attribute decision making and frames the gap the paper targets.","marker":"[17]"},{"why":"Supplies a recent method for determining attribute weights from heterogeneous discriminating power, a direct comparison point for variance-derived weights.","marker":"[19]"},{"why":"Connects attribute weighting to discriminating power in belief distributions, the same intuition the paper applies to variance.","marker":"[20]"}],"fun_headline_variants":["Variance, not means, informs ethical weights in sims","Three variance-based formulas weigh ethics in simulations","Ethical weight from variance keeps humans out of sim loop","Split variance from mean to set ethical weights in sims","Info-theoretic ethics: variance sets weights, humans choose"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes the ethical measurement problem is solved: acceptable, granular ethical metrics exist and the utility values U(r,s) already carry the commander's subjective importance, so weighting failures are attributed to the weights alone.","fun_headline_variants_meta":{"raw":{"variants":["Variance, not means, informs ethical weights in sims","Three variance-based formulas weigh ethics in simulations","Ethical weight from variance keeps humans out of sim loop","Split variance from mean to set ethical weights in sims","Info-theoretic ethics: variance sets weights, humans choose"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000766,"raw_usage":{"total_tokens":3427,"prompt_tokens":1007,"completion_tokens":2420,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":623,"completion_tokens_details":{"reasoning_tokens":2341}},"tokens_in":623,"tokens_out":2420,"duration_ms":26540,"temperature":1.0,"reasoning_tokens":2341,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:37:09.623431+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a corpus of simulated command situations where a known ethically decisive attribute is constant across all situations for every course of action; if the proposed weights give that attribute zero weight, an externally validated judgment would show the weighting contradicts what human commanders consider decisive.","supporting_citations":[{"cited_title":"Effects of the entropy weight on topsis,","cited_arxiv_id":null,"evidence_quote":"Supplies the caveat that entropy weights reflect variability rather than subjective importance, motivating the paper's separation of Lambda and Gamma."},{"cited_title":"C 4.5: Programs for machine learning,","cited_arxiv_id":null,"evidence_quote":"Provides the Kullback-Leibler/information-gain basis adopted for the IGH weighting scheme."},{"cited_title":"Tzeng and J.-J","cited_arxiv_id":null,"evidence_quote":"Grounds the multi-attribute decision analysis context and the TOPSIS-style ranking that the proposed weights feed into."},{"cited_title":"Explicit methods for attribute weighting in multi-attribute decision-making: a review study,","cited_arxiv_id":null,"evidence_quote":"Reviews explicit attribute-weighting methods in multi-attribute decision making and frames the gap the paper targets."},{"cited_title":"Determining attribute weights based on het- erogeneous discriminating power and solution reliability in evidential reasoning approach,","cited_arxiv_id":null,"evidence_quote":"Supplies a recent method for determining attribute weights from heterogeneous discriminating power, a direct comparison point for variance-derived weights."},{"cited_title":"Determining attribute weights for multiple attribute decision analysis with discriminating power in belief distributions,","cited_arxiv_id":null,"evidence_quote":"Connects attribute weighting to discriminating power in belief distributions, the same intuition the paper applies to variance."}],"review_version":1}