{"id":"22e8a0ca-8671-4acb-9a86-93392083ec1a","arxiv_id":"2505.01445","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SHAP attributions identify the machine setting that was altered from its ideal value in injection moulding experiments, whereas ICE attributions consistently rank packing time first.","lead":"This paper compares two tools that explain what a machine-learning model thinks caused a product defect in injection moulding. The authors find that one tool (SHAP) points to the setting that was actually changed, while the other (ICE) almost always blames packing time.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Section 4.2 ground truth is not a valid oracle: exact Shapley values need not rank the changed feature first, and the paper's own cooling-time exception leaves the claim unquantified.","rationale":"Good-faith assessment: the experimental setup and model quality are solid; MAPE below 0.05% is strong, and the controlled single-feature-change design is a reasonable way to test explainers. The soft spot is the ground truth: it is asserted, not derived. Exact Shapley values are not guaranteed to put the changed feature first, and the paper's own cooling-time exception shows the pattern is not universal. This is not merely a missing robustness check; it is a missing definition of 'correct cause' that is independent of the explainer being tested. A revision that quantifies per-setting accuracy and compares against exact Shapley values would settle it. I therefore mark the central claim unverdictable as currently stated rather than fully rejecting it, since the models and data may support a weaker version of the claim.","tokens_in":12850,"tokens_out":11113,"duration_ms":130624,"concrete_test":"Take the trained RF and MLP models from Section 4 and, for every test sample in which exactly one of the six settings differs from its mid value, compute exact Shapley values (or a high-precision estimate with 10^5 permutations per sample) in addition to the paper's permutation SHAP. Count, per setting and overall across the 10 random splits, the proportion of samples in which the changed setting has the largest absolute Shapley value. Report the same count for the planarity outcome. If exact Shapley does not rank the changed setting first in all tested cases (in particular, cooling time on RF), the Section 4.2 oracle is not reflected by SHAP and the central claim must be downgraded to a probabilistic observation.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The paper's central claim reduces to the assertion in Section 4.2 that when one machine setting is changed from its mid value, that setting 'should be the first major cause' and that any accurate black-box model will reflect this. This assertion is not a consequence of model accuracy. Shapley attribution distributes the prediction difference f(x)-E[f(X)] among all features and splits interaction terms among the interacting features, so a changed feature can have a small or zero Shapley value even when it is the only feature that moved. A simple example is y = x1*x2 with x1 changed from 0 to 1 while x2 stays at 0: the Shapley value of x1 is 0. Thus 'the changed feature should rank first' is a normative claim, not an oracle. The paper itself reports an exception ('apart from the cooling time in RF', Section 4.1) without quantifying it. The reported comparison is therefore not a measured correctness rate but a visual alignment with an asserted label. Additionally, the Section 3.3 independence claim is unsupported: zero Pearson correlation among factors in the face-centered CCD does not imply statistical independence (e.g., for any pair, P(+1,+1)=16/77 but P(+1)^2=(33/77)^2). This further weakens the permutation-SHAP approximation that the claim relies on.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses root cause analysis of product quality in injection moulding by comparing two model-agnostic explainability methods, permutation-based SHAP and ICE, on random forest and multilayer perceptron models. Using a face-centered central composite design dataset of 77 machine-setting combinations repeated 20 times (1540 cycles), the authors train both models to high accuracy (MAPE < 0.05%) and use Friedman's H-statistic to argue that machine settings interact. They then rank the impact of six machine settings on product weight for each test sample and compare SHAP and ICE rankings. In controlled experiments where a single machine setting is changed from its mid value, the paper claims that SHAP ranks the changed setting first, whereas ICE always ranks packing time first. The same comparison is repeated for a second quality characteristic, product planarity.","tokens_in":13105,"tokens_out":4818,"duration_ms":50797,"significance":"If the central claim were established, the paper would offer a practically useful, model-agnostic recipe for root cause identification in injection moulding, a domain where small weight deviations matter and operators must decide which machine setting to adjust. The study's strengths include a real experimental dataset with repeated runs, careful model selection with confidence intervals for errors (Tables 5 and 6), use of an interaction diagnostic (H-statistic), comparison across two distinct model families, and an additional validation on a second quality characteristic (planarity). These elements make the paper useful as an empirical comparison of XAI methods in manufacturing. However, the central evaluation that SHAP 'correctly' identifies the cause is not supported by a quantitative measure: the correctness judgment rests on visual inspection of Figures 6, 7, and 9, and the paper's own exception ('apart from the cooling time in RF') is left unquantified.","major_comments":[{"comment":"The ground-truth assumption that a single changed machine setting 'should be the first major cause' is not a logical consequence of model accuracy. Shapley values distribute prediction differences and split interaction terms among features, so a changed feature can receive a small or even zero attribution when it is the only feature that moves; for example, with y = x1*x2 and x1 changed from 0 to 1 while x2 remains 0, the Shapley value of x1 is 0. Thus the 'correct cause' verdict is a normative claim, not an oracle. The paper itself reports an exception (cooling time under RF) but does not quantify it. To make the central claim defensible, the authors should report the proportion of test samples in which SHAP's top-ranked feature equals the experimentally changed setting (and the corresponding proportion for ICE), with confidence intervals across the 10 random splits, and explicitly discuss the observed exceptions.","section":"Section 4.2, Equations (7)-(8)"},{"comment":"The statement that the machine settings 'are independently tuned with zero correlations among themselves, hence, the feature independence holds' is not valid: zero Pearson correlation does not imply statistical independence. For the described face-centered CCD with 77 unique combinations, the joint distribution is not a product of marginals; for instance, for any two-level factor, P(+1,+1)=16/77 while P(+1)^2=(33/77)^2, so the independence assumption is violated. Since the permutation-based SHAP approximation in Equation (7) relies on sampling from a background distribution assuming independence, the authors should either use a dependence-aware SHAP variant or empirically assess the sensitivity of the SHAP rankings to this assumption. Without this, the claim that SHAP correctly captures interactions is not fully supported.","section":"Section 3.3"}],"minor_comments":[{"comment":"The sentence 'The rankings by ICE ... are used as the benchmark for SHAP based rankings' is misleading, since the paper later argues that ICE produces incorrect rankings. ICE should be described as a competing method rather than a benchmark.","section":"Section 3.4"},{"comment":"The H-statistic is computed only for the random forest model, but the SHAP-versus-ICE comparison is performed for both RF and MLP; computing the H-statistic for MLP as well would strengthen the claim that both models capture interactions and that the conclusions are model-agnostic.","section":"Section 4.1, Figure 4"},{"comment":"In Table 1, 'min mould height: 200 nm' appears to be a typo for '200 mm'; please correct it.","section":"Section 2, Table 1"},{"comment":"The phrase 'the better feature attribution translates to the correct cause identification' is stated as a general conclusion, but the evidence is limited to one dataset and one quality characteristic (weight, and secondarily planarity); the conclusion should be scoped accordingly to avoid overclaiming.","section":"Section 1 and Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The core concern is not the quality of the experiments, which are reasonably controlled, but the validity of the 'correct cause' oracle. The authors need to either justify why a changed feature must have the largest Shapley value in their specific physical process (e.g., by showing near-additivity or by reporting effect sizes), or substantially weaken the claim to a descriptive one. I believe the paper can be revised to a publishable state by quantifying the agreement rates and addressing the independence issue, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper is worth a look if you care about whether XAI methods actually point to the right root cause in manufacturing. The new thing is the direct comparison: on a designed CCD dataset from an injection moulding machine, they change one machine setting at a time from its mid value and ask whether SHAP or ICE ranks that changed setting as the top cause. SHAP does, for most settings; ICE always points to packing time. That is a concrete, practical result, and the second validation on a planarity measurement strengthens it.\n\nWhat the paper does well: the models are accurate (MAPE under 0.05%), errors come with confidence intervals, the H-statistic checks show real interactions among the settings, and both RF and MLP give similar explanations. The idea of using a controlled single-feature-change experiment as a benchmark is a good one.\n\nThe soft spots are real but not fatal. The biggest is Section 4.2. The claim that \"when a single machine setting changes from its mid value, it is obvious that that particular machine setting should be the first major cause\" is asserted, not argued. The stress-test example is exactly on point: a feature can move and yet have a small Shapley value if its effect is mostly through interactions. The paper's own data shows one exception (cooling time in RF), and that exception is left unquantified. I would like to see an agreement rate: for how many test samples does SHAP rank the changed feature first? Eye-balling Figures 6 and 7 is not enough.\n\nThe independence claim in Section 3.3 is also too strong. Zero Pearson correlation among the CCD factors does not imply independence, and permutation-based SHAP relies on that assumption to generate masked samples. It might be a fine approximation here, but \"zero correlations among themselves, hence feature independence holds\" is not a sound argument. They should either justify it empirically or soften it.\n\nThe paper also does not provide code or data, which makes the visual-only comparison harder to check. And the generalization to \"other production setups\" is speculative; the second experiment helps, but it is still one machine and one material.\n\nAll that said, the empirical pattern is consistent, and the central takeaway—ICE can mislead because it ignores interactions, while SHAP handles them better—is plausible and useful for practitioners. It deserves a serious referee, but the referee should ask for a quantified correctness metric, a proper treatment of the independence assumption, and code/data.","headline":"A useful applied comparison of SHAP and ICE for injection moulding root cause analysis, but the central 'correct cause' claim rests on an asserted ground truth and needs quantification.","tokens_in":13626,"tokens_out":2576,"would_cite":false,"duration_ms":26721,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62P30","62M45","91A80"],"pacs":["89.20.Bb","07.05.Mh","02.50.Le"],"model":"deepseek-v4-flash","headline":"Permutation-based SHAP correctly identifies the machine setting responsible for a product's weight deviation in injection moulding, whereas ICE misattributes the cause because it ignores interactions between settings.","keywords":["injection moulding","root cause analysis","explainable AI","SHAP","individual conditional expectation","feature interactions","quality control","feature attribution"],"falsifier":"Run the controlled protocol on a synthetic or real dataset where one setting is changed but a known interaction makes a different setting the dominant predictor of weight; if ICE then identifies the true dominant cause more often than SHAP, or SHAP's ranking matches the changed setting no better than chance, the central claim fails. A cheaper check is to collect many more trials around the cooling-time case, where SHAP sometimes fails on random forest, and test whether those failures are systematic rather than noise.","tokens_in":12655,"feed_emoji":"⚙️","tokens_out":8328,"duration_ms":71952,"temperature":0.7,"pith_summary":"The paper claims that when an injection-moulded part deviates from its target weight, the correct way to find which machine setting caused the deviation is permutation-based SHAP, not individual conditional expectation (ICE) plots. It first demonstrates, using a designed experiment with six operator-controlled settings, that those settings genuinely interact in determining product weight. In controlled trials where a single setting was moved from its middle value, SHAP named that changed setting as the top cause in nearly all cases, while ICE named packing time first in every trial, even when packing time had not changed. The conclusion is that choosing an explainability method that accounts for interactions is not a formality but determines whether the diagnosis is actionable.","feed_headline":"SHAP traces moulding weight errors to the true cause; ICE doesn't","feed_subtitle":"SHAP gives the correct root cause because it accounts for interactions; ICE ignores them and always blames packing time.","key_machinery":"The central machinery is the contrast between two local, model-agnostic explanation methods applied to the same fitted black-box models. Permutation-based SHAP assigns each machine setting a Shapley value, the average marginal contribution of that setting to the prediction over all random orderings of the settings, which is the mechanism that lets interaction effects enter each attribution. ICE, by contrast, varies one setting while holding the others fixed and records the standard deviation of the resulting prediction curve, so each setting is judged in isolation. The paper's third piece of machinery is the controlled single-change protocol: because the ideal product weight is defined at the mid value of every setting, changing exactly one setting defines which setting must be the first major cause, giving the comparison a ground truth. The H-statistic then supplies the evidence that interactions are large enough to make the choice between the two explanation methods consequential.","core_discovery":"On the paper's own terms, the discovery is that a better feature attribution translates to the correct cause identification: an accurate random forest or multilayer perceptron trained on six machine settings, when explained by permutation-based SHAP, attributes a product weight deviation to the machine setting that was actually changed from its mid value, while the same model explained by ICE attributes the deviation to packing time in every tested cycle. The paper uses the H-statistic and two-way partial dependence plots to show that interactions among the settings are real and substantial, with melt temperature and packing time interacting strongly. It interprets SHAP's correct attributions as a consequence of SHAP distributing credit over all feature orderings, so interaction effects are folded into each setting's score, whereas ICE varies one setting at a time and therefore misses the interaction component. The pattern repeats when the quality characteristic is changed from weight to planarity, supporting the claim that the finding is not specific to one product property.","pith_inferences":["If interactions are the reason SHAP outperforms ICE, the two methods should give nearly identical rankings in processes where the H-statistic is near zero; measuring that convergence would isolate the mechanism.","The same mid-value single-change protocol could benchmark additional explanation methods such as LIME or integrated gradients, without needing external ground truth, as long as the single-change assumption holds.","The paper's own cooling-time exception for random forest suggests SHAP is not infallible; a practical safety check would flag settings whose SHAP ranking flips across retrained models or random splits."],"forward_implications":["An operator facing a drifting product weight can trust SHAP's top-ranked setting as the first thing to adjust, rather than defaulting to packing time as ICE suggests.","Because SHAP on random forest costs only milliseconds per cycle, the explanation step can run in real time alongside the roughly ten-second production cycle.","The same attribution logic should apply to other quality characteristics, since the planarity experiment reproduces the pattern for a different property of the same product.","In processes monitored on multiple quality characteristics at once, correctly identifying the root cause before adjusting settings becomes more important, since changing the wrong setting to fix one characteristic can damage another."],"supporting_citations":[{"why":"Defines the permutation-based SHAP estimator and the Shapley-value sampling that the paper's explanation method uses.","marker":"[28]"},{"why":"Introduces the H-statistic that the paper uses to measure interactions among machine settings.","marker":"[23]"},{"why":"Earlier injection-moulding explanation study using rule-based explanations with PDP and ICE, the baseline the paper argues against.","marker":"[17]"},{"why":"Prior work applying Tree SHAP and ICE to plastic injection moulding, the approach whose disagreement this paper investigates.","marker":"[18]"},{"why":"Defines partial dependence and interaction analysis, and is the reference for the interaction plots used to confirm H-statistic results.","marker":"[27]"},{"why":"Demonstrates SHAP interpretation of transient process data in injection moulding, supporting the choice of SHAP.","marker":"[19]"},{"why":"Supplies the neural-network-based quality prediction pipeline that motivates the choice of MLP in injection moulding.","marker":"[4]"},{"why":"Shows how combinations of simulated and measured data are used to learn injection-moulding quality characteristics.","marker":"[8]"},{"why":"Discusses machine-specific behaviour of injection-moulding machines, supporting the claim that findings should hold across setups.","marker":"[30]"}],"fun_headline_variants":["SHAP finds true moulding culprit; ICE blames packing time","Interaction-aware AI nails true root cause in moulding","SHAP corrects moulding root cause; ICE always blames packing","Why SHAP finds the real moulding fault while ICE doesn't","Interaction-blind ICE blames packing; SHAP sees true cause"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison's ground truth is the assumption that when exactly one machine setting is changed from its mid value, that setting is necessarily the first major cause of the resulting weight deviation, and that an accurate black-box model fitted to the data will reflect that ordering.","fun_headline_variants_meta":{"raw":{"variants":["SHAP finds true moulding culprit; ICE blames packing time","Interaction-aware AI nails true root cause in moulding","SHAP corrects moulding root cause; ICE always blames packing","Why SHAP finds the real moulding fault while ICE doesn't","Interaction-blind ICE blames packing; SHAP sees true cause"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000468,"raw_usage":{"total_tokens":2336,"prompt_tokens":956,"completion_tokens":1380,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":1292}},"tokens_in":572,"tokens_out":1380,"duration_ms":9671,"temperature":1.0,"reasoning_tokens":1292,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:15:52.817357+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the controlled protocol on a synthetic or real dataset where one setting is changed but a known interaction makes a different setting the dominant predictor of weight; if ICE then identifies the true dominant cause more often than SHAP, or SHAP's ranking matches the changed setting no better than chance, the central claim fails. A cheaper check is to collect many more trials around the cooling-time case, where SHAP sometimes fails on random forest, and test whether those failures are systematic rather than noise.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the permutation-based SHAP estimator and the Shapley-value sampling that the paper's explanation method uses."},{"cited_title":"Obregon, J","cited_arxiv_id":null,"evidence_quote":"Earlier injection-moulding explanation study using rule-based explanations with PDP and ICE, the baseline the paper argues against."},{"cited_title":"Molnar, Interpretable Machine Learning: A Guide for Making Black Box Models Explainable, 2nd Edition, 2022","cited_arxiv_id":null,"evidence_quote":"Defines partial dependence and interaction analysis, and is the reference for the interaction plots used to confirm H-statistic results."},{"cited_title":"Lockner, C","cited_arxiv_id":null,"evidence_quote":"Supplies the neural-network-based quality prediction pipeline that motivates the choice of MLP in injection moulding."},{"cited_title":"Finkeldey, J","cited_arxiv_id":null,"evidence_quote":"Shows how combinations of simulated and measured data are used to learn injection-moulding quality characteristics."},{"cited_title":"Knoll, H.-P","cited_arxiv_id":null,"evidence_quote":"Discusses machine-specific behaviour of injection-moulding machines, supporting the claim that findings should hold across setups."}],"review_version":1}