{"id":"b7a06a73-0538-457a-bec5-89eb7705db27","arxiv_id":"2507.05976","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A tf-idf-style post-processing of rule-based explanations produces factor relevance scores that users rate as more interpretable and useful than raw rule output in a small study.","lead":"This paper applies information retrieval metrics to the rules of an explainable AI model to rank which risk factors most influence a lymphedema prediction. A user study suggests that presenting these factor rankings as radar charts or lists feels more interpretable and useful to readers than showing raw rules.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The attribution claim rests on an unvalidated frequency-as-contribution proxy; the user study compares presentation formats, not score correctness.","rationale":"The paper's stated contribution is attribution: it claims to compute the relevance and impact of risk factors using tf-idf on rule occurrence counts. The single most load-bearing premise is the assumption that occurrence frequency in activated rules corresponds to predictive contribution. This premise is not implied by the rule semantics: a rule is a conjunction of conditions, and multiple activated rules may share conditions for reasons unrelated to effect size, such as redundancy in the covering algorithm or repeated thresholds on the same variable. The paper explicitly labels this as an assumption and provides no validation against model outputs, ablations, or alternative attribution methods. The local examples in Section 4.2 are built from randomly assigned rule sets, so they provide no evidence that the scores track real predictions. The user study in Section 4.3 asks participants to rate interpretability and usefulness of three visualizations; it demonstrates that users prefer the simplified presentations, but it does not test whether the factor rankings are correct. Thus, the strongest claim about computing 'impact' is unsupported as it stands. I also verified the Eq. 7 inconsistency: since tf-idf is always non-negative, the logistic complement is bounded above by 0.5, contradicting the text that scores can approach 1. This is a concrete mathematical error, but it is secondary to the conceptual attribution problem because the relative ordering is still monotonic; the central issue remains the unvalidated frequency proxy. The reader's weakest assumption identifies the same concern, and my assessment does not shift the verdict: the contribution is promising but should not be accepted as an attribution method without empirical validation of the proxy, so CONDITIONAL remains appropriate. Given the possibility of a successful validation, the verdict is unchanged from the reader's CONDITIONAL. If validation fails, the method would at best be a visualization aid, not an attribution method.","tokens_in":9995,"tokens_out":7009,"duration_ms":82269,"concrete_test":"Construct a synthetic rule-based model with known ground-truth contributions: let factor A appear in 80% of the activated rules but have zero effect on the predicted risk, and let factor B appear in only one rule but change the risk from low to high. Compute the proposed relevance scores (Eqs. 1-7) for both factors. If B is not ranked above A, the frequency proxy in Section 3 is invalid. Repeat across several synthetic rule sets to confirm the result is not a single pathological case.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the proposed method 'computes the relevance of each attribute to the prediction' depends on the assumption stated in Section 3 (after Eq. 5): the more an attribute appears in the preconditions of activated rules, the greater its contribution. This is never derived from the model's semantics or tested against model behavior. For a rule set extracted from an MLP ensemble, frequent appearance can reflect overlapping rules or broad threshold conditions rather than causal influence; a rare but decisive factor could be ranked below a frequent but nearly inert one. The local demonstration in Section 4.2 compounds the problem: the 10 profiles are built by randomly assigning 7 rules each, so the frequency counts are not tied to any actual model input. The user study in Section 4.3 only measures subjective preference among visualization formats; it cannot validate the accuracy of the scores. A secondary but concrete internal issue is that Eq. 7 maps all non-negative tf-idf values to (0, 0.5], so the stated range [0, 1] and 'values close to 1' are impossible; this is fixable but signals that the metric's behavior has not been carefully checked.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an information-retrieval-based method to enhance the interpretability of rule-based explanations produced by an ensemble MLP model for lymphedema risk prediction. The method aggregates input attributes into semantic factors, applies tf-idf statistics to the set of activated rules for a given prediction, inverts the resulting scores, and visualizes factor relevance via radar charts or ranked lists. A user study with 30 participants compared the raw rule output against the two proposed visualizations and found statistically significant improvements in perceived interpretability and usefulness.","tokens_in":10241,"tokens_out":4078,"duration_ms":44866,"significance":"If the relevance scores are accepted as meaningful measures of factor contribution, the approach offers a lightweight, flexible, and model-agnostic way to contextualize rule-based XAI output for non-expert users, and the user study provides preliminary evidence of improved perceived interpretability. The use of IR metrics as a post-hoc explanation layer is a novel idea with potential applicability beyond this clinical use case. However, the paper's central claim that the method 'computes the relevance of each attribute to the prediction' is not supported by the evidence: the frequency-based proxy is unvalidated, and the user study only measures subjective preference. The internal inconsistency in Eq. (7) further undermines confidence in the reported scores.","major_comments":[{"comment":"For non-negative tf-idf values, the transformation tf_idf*_i = 1 - 1/(1 + exp(-tf_idf_i)) produces values in (0, 0.5], not in [0, 1]. The statement that 'Values close to 1 are assigned to factors i that are common' is therefore mathematically impossible. This affects the interpretation of Figures 2-4 and the logistic inversion description. Please correct the formula (e.g., use 2/(1+exp(-x)) - 1 or a suitable scaling) or revise the text and figures to reflect the actual range.","section":"Section 3.2, Eq. (7)"},{"comment":"The underlying assumption that the frequency of an attribute in the preconditions of activated rules is proportional to its contribution to the prediction is stated but never validated. The user study in Section 4.3 measures only subjective interpretability and usefulness; it does not test whether the computed scores correspond to the model's actual behavior or to clinically established risk factors. Consequently, the abstract's claim that the method 'computes the relevance of each attribute to the prediction' is unsupported. Please either add a validation study (e.g., model ablations, comparison with SHAP or LIME, or expert assessment of score correctness) or temper the claims to describe the scores as frequency-based indicators of prominence rather than validated contributions.","section":"Section 3, paragraph after Eq. (5)"},{"comment":"The local explanation demonstration uses 10 profiles with randomly assigned sets of 7 rules each, not actual patient inputs processed through the model. This shows the flexibility of the visualization but does not validate that the local scores reflect how the model would explain a real patient's prediction. Please clarify this limitation explicitly and consider including at least one case with a real input and the rules actually activated by the model for that input.","section":"Section 4.2"}],"minor_comments":[{"comment":"The text uses 'call' while the equation uses c_all for total coverage; please unify the notation throughout the paper.","section":"Section 3.2, Eq. (2)"},{"comment":"The reported degrees of freedom F(2, 56) are inconsistent with the stated sample size of 30 participants, as a repeated-measures ANOVA with k=3 conditions and n=30 would have df_error = 29*2 = 58. Please verify the statistical analysis and correct any typographical errors.","section":"Section 4.3"},{"comment":"The 'list of factors' visualization presents impact as a percentage, but the paper does not explain how these percentages are computed from the tf_idf* scores. Please specify the normalization used.","section":"Section 4.3, list of factors"},{"comment":"The description of the rule extraction algorithm from reference [11] is brief; a few more details (e.g., the number of attributes, typical rule length, and how fidelity is thresholded) would improve reproducibility and help readers interpret the subsequent results.","section":"Section 2"},{"comment":"The phrase 'infers causal correlations' overstates what a frequency-based analysis can establish. Suggest replacing 'causal correlations' with 'associations' or 'prominence' to avoid implying causal inference.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the journal's scope as an application of interpretable AI in healthcare. The main weakness is the gap between the claimed ability to compute factor relevance and the evidence provided: the core assumption is unvalidated, and the user study only demonstrates subjective preference for the presentation format. The internal inconsistency in Eq. (7) is a concrete technical error that must be fixed. I believe these issues are addressable within a revision, hence my recommendation of major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Briefly, this paper applies tf-idf (and an inversion) to the rules activated by a rule-based XAI model, aggregating attributes into clinical factors, to produce a ranked list of factor relevance for lymphedema risk. The specific post-hoc use of IR metrics on explanation outputs is new relative to their cited work, and the presentation is clear. The user study (N=30) shows that both AI experts and non-experts rate the summarized radar/list outputs significantly higher on interpretability and usefulness than the raw rule output. That result is credible, and the repeated-measures ANOVA looks properly executed. The authors also correctly identify why SHAP and LIME aren't directly usable under their privacy/data-access constraints.\n\nThe soft spots are real but not fatal. First, the attribution claim rests on an untested assumption: frequency of a factor in activated rules equals contribution to the prediction. The paper states this explicitly but never validates it against model ablations, alternative attribution methods, or clinical outcomes. The local demo makes it worse by randomly assigning seven rules per profile, so the frequency counts aren't tied to actual model behavior. The user study only measures preference for presentation format—it cannot validate score accuracy. So the paper proves that summaries are liked; it does not prove the scores are correct. Second, Eq. 7 as written maps non-negative tf-idf to the interval (0, 0.5], yet the text and figures claim scores close to 1 for common factors. That's an elementary error—likely a missing normalization—but it signals the metric's behavior wasn't checked. Third, no code or data is shipped yet; the repository is promised only upon acceptance.\n\nFor whom? Practitioners building user-facing XAI in healthcare who need a lightweight, retraining-free way to present rule-based explanations, and who treat the relevance scores as a heuristic, not ground truth. Referees should want a revision that either validates the frequency assumption or softens the claim, and fixes Eq. 7. This deserves peer review; it's a modest but honest contribution with a real user study. I'd send it out with a request for validation of the scores, not desk-reject it.","headline":"A lightweight, clearly-presented post-hoc method for ranking clinical factors from rule-based XAI, whose headline attribution claim rests on an unvalidated frequency heuristic and whose Eq. 7 has a fixable but real scaling error.","tokens_in":10732,"tokens_out":3173,"would_cite":false,"duration_ms":32427,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A tf-idf-based post-processing of rule-based AI output gives clinicians and patients factor-level rankings of what drove a lymphedema risk prediction, and a user study reports that the ranked visualizations are rated more interpretable…","keywords":["explainable AI","tf-idf","rule-based explanations","attribution method","interpretability","lymphedema risk prediction","user study","information retrieval"],"falsifier":"Construct or locate a patient case where a known decisive factor appears in only one activated rule while a neutral factor appears in many; if the proposed score ranks the neutral factor higher and removing the decisive factor from the rules changes the predicted risk, the frequency-proxy assumption is falsified.","tokens_in":9820,"feed_emoji":"📊","tokens_out":5331,"duration_ms":57917,"temperature":0.7,"pith_summary":"The paper proposes a post-hoc attribution method that makes an already rule-based AI model's predictions easier for non-AI users to grasp. It borrows the tf-idf metric from information retrieval and applies it to the rules activated for a prediction, treating each rule as a document and each clinically grouped risk factor as a token. The claim is that a factor's frequency in the activated rule conditions proxies its contribution to the risk prediction, so reversing tf-idf yields a per-patient factor ranking. A user study with 30 participants reports that radar-chart and factor-list presentations are rated significantly more interpretable and useful than the raw rules, by both AI experts and non-experts. If this holds, the approach offers a lightweight, retraining-free route to more readable explainable-AI output in clinical risk communication.","feed_headline":"Ranking risk factors with tf-idf beats raw AI rules in study","feed_subtitle":"Doctors and patients rated radar and list views of lymphedema risk as more interpretable and useful than raw rule output.","key_machinery":"The central object is an adapted tf-idf score computed over the rules activated by a prediction. Attributes are first clustered into contributing factors, so each factor may appear zero or multiple times in a rule's preconditions. For each activated rule the method computes a term frequency weighted by the rule's coverage, and across the rule set an inverse document frequency that is high when a factor appears in few rules; the per-rule scores are averaged to give each factor a relevance value. Because the raw average is low for factors that appear everywhere, the paper applies a logistic transform and inverts the score, so factors appearing in many activated rules end up with relevance close to 1. That inversion is what converts a standard information-retrieval metric into an attribution score for rule-based predictions.","core_discovery":"The paper argues that raw rule output from an explainable prediction model is not enough for clinicians and patients, and that the model can be treated as a black box and post-processed without retraining. It groups the model's attributes into semantically coherent factors, treats each activated rule as a document and each factor as a token, and computes a coverage-weighted term frequency and an inverse rule frequency. The standard tf-idf meaning is reversed: factors that recur across many rules receive high relevance, because they are the ones the paper takes to shape the prediction. Applied globally over all 301 rules it ranks familiar lymphedema risk factors such as BMI, involved nodes, and radiotherapy technique at the top, and applied locally it yields per-patient radar and list visualizations. In the user study, repeated-measures ANOVA found significant effects of visualization type on both interpretability and usefulness, with the processed visualizations scoring significantly higher than raw rules and no difference between AI-experts and non-experts in their judgments.","pith_inferences":["An extension left implicit is that the same frequency-as-relevance score could be applied to any model that emits multiple activated rules per prediction, as long as rule conditions can be mapped to semantically grouped factors.","A stronger validation would compare the factor rankings against known ground-truth causal roles in controlled rule sets, or against perturbation-based attributions; the paper itself does not test the frequency proxy in that way.","The clustering flexibility suggests a testable follow-up: measuring whether coarse versus fine factor groupings change clinicians' judgments or trust using decision-based or calibration metrics rather than self-report alone."],"forward_implications":["Global explanations over all rules yield a ranked list of risk factors, helping clinicians check the model against known lymphedema literature.","Local explanations per patient show which factors drove each prediction, with rankings and visualizations that differ across patient profiles.","The attribute-to-factor clustering can be changed to coarser or finer granularity without retraining the model, allowing explanations to be tailored to different user expertise levels.","In the reported user study, radar and factor-list presentations beat raw rule output on interpretability and usefulness, with no significant difference between AI-expert and non-expert ratings."],"supporting_citations":[{"why":"supplies the underlying rule-based prediction model and the rule-extraction mechanism whose output the proposed method interprets.","marker":"[11]"},{"why":"supplies the power analysis used to fix the required sample size of 28 participants, with 30 actually recruited for the user study.","marker":"[16]"},{"why":"provides the multi-perspective quality-of-XAI framing that motivates the user-centered interpretability and usefulness evaluation.","marker":"[10]"},{"why":"represents a canonical attribution method that the paper argues is not feasible here because it requires retraining on subsets of input attributes.","marker":"[19]"},{"why":"represents a canonical attribution method that the paper argues does not directly apply because the same input can occur multiple times across rules.","marker":"[20]"},{"why":"supplies clinical evidence that high BMI is a critical lymphedema risk factor, used to check the coherence of the computed global ranking.","marker":"[14]"},{"why":"supplies clinical evidence on lymphedema risk factors used to corroborate the relevance of nodes involved and radiotherapy technique in the global explanation.","marker":"[15]"}],"fun_headline_variants":["TF-IDF ranks lymphedema risk factors better than raw rules","Clinicians find tf-idf rules more interpretable than raw AI output","Post-processing rule output with tf-idf aids lymphedema decisions","Treat AI rules as documents to rank lymphedema risk factors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a factor's frequency in the rule preconditions activated for a prediction measures its actual contribution to that prediction; the paper states this assumption but does not test it against clinical outcomes, model ablations, or alternative attribution methods.","fun_headline_variants_meta":{"raw":{"variants":["TF-IDF ranks lymphedema risk factors better than raw rules","Clinicians find tf-idf rules more interpretable than raw AI output","Post-processing rule output with tf-idf aids lymphedema decisions","Treat AI rules as documents to rank lymphedema risk factors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001567,"raw_usage":{"total_tokens":6220,"prompt_tokens":873,"completion_tokens":5347,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":5269}},"tokens_in":489,"tokens_out":5347,"duration_ms":37844,"temperature":1.0,"reasoning_tokens":5269,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:13:39.981479+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct or locate a patient case where a known decisive factor appears in only one activated rule while a neutral factor appears in many; if the proposed score ranks the neutral factor higher and removing the decisive factor from the rules changes the predicted risk, the frequency-proxy assumption is falsified.","supporting_citations":[{"cited_title":"Fidex: An algorithm for the explainability of ensembles and svms","cited_arxiv_id":null,"evidence_quote":"supplies the underlying rule-based prediction model and the rule-extraction mechanism whose output the proposed method interprets."},{"cited_title":"Statistical power analyses using g*power 3.1: Tests for correlation and regression analyses","cited_arxiv_id":null,"evidence_quote":"supplies the power analysis used to fix the required sample size of 28 participants, with 30 actually recruited for the user study."},{"cited_title":"Moore, Marinka Zitnik, and John H","cited_arxiv_id":null,"evidence_quote":"provides the multi-perspective quality-of-XAI framing that motivates the user-centered interpretability and usefulness evaluation."},{"cited_title":"Lundberg and Su-In Lee","cited_arxiv_id":null,"evidence_quote":"represents a canonical attribution method that the paper argues is not feasible here because it requires retraining on subsets of input attributes."},{"cited_title":"Why Should I Trust You?","cited_arxiv_id":null,"evidence_quote":"represents a canonical attribution method that the paper argues does not directly apply because the same input can occur multiple times across rules."},{"cited_title":"Incidence and risk factors of lymphedema after breast cancer treatment: 10 years of follow-up","cited_arxiv_id":null,"evidence_quote":"supplies clinical evidence that high BMI is a critical lymphedema risk factor, used to check the coherence of the computed global ranking."},{"cited_title":"Incidence and risk factors of breast cancer lymphoedema","cited_arxiv_id":null,"evidence_quote":"supplies clinical evidence on lymphedema risk factors used to corroborate the relevance of nodes involved and radiotherapy technique in the global explanation."}],"review_version":1}