{"id":"e3a65869-60da-485a-b2fc-198f993d78fc","arxiv_id":"2412.16204","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"On synthetic AND/OR/XOR tasks, saliency methods frequently rank irrelevant baseline inputs above logically relevant ones, and retrained models can recover class information from masked inputs, indicating that score order can encode extra information.","lead":"This paper creates simple logic datasets (AND, OR, XOR) and tests 12 explanation methods to see whether saliency scores point to the inputs a model actually used. It finds that the ordering of saliency scores can silently carry classification information, so a low score is not proof that an input was ignored.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The DCA evidence for the headline claim is confounded by input-dependent mask patterns; without a random-mask control, 'encoding into the ordering' is not distinguished from leakage through the mask itself.","rationale":"The paper's central claim is that saliency methods encode classification-relevant information into the ordering of saliency scores. The DCA metric is the direct quantitative support for this encoding claim: it shows that after masking based on saliency rankings, retrained models can still assign different classes to inputs that share the same logical relevant set, implying the discarded (low-scoring/baseline) positions carry class information. However, the masking procedure itself creates an input-dependent pattern, because which positions are masked depends on the saliency scores, which depend on the input. A retrained model can therefore use the shape or location of the mask as a side channel, exactly the phenomenon the paper acknowledges in Assumption D and in the discussion of [32]. Without a control that randomizes or fixes the mask pattern while holding sparsity fixed, DCA > 0 does not distinguish 'saliency ordering encodes information' from 'the mask pattern leaks information.' This is not a minor technicality: it directly determines whether the headline conclusion is warranted. The NIB results are less affected by this confound, but they depend on the separate assumption that a 100% accurate model must follow the logical minimal sets R_min; that assumption is also stated rather than verified. The paper is otherwise careful: it uses fully enumerated datasets, balances classes, reports many conditions, and compares multiple saliency methods. The missing control is inexpensive and would settle the interpretation. For these reasons, the appropriate verdict remains conditional: the claim is plausible but should not be accepted as established until the mask-leakage confound is ruled out.","tokens_in":13508,"tokens_out":3280,"duration_ms":32113,"concrete_test":"Run the Full-DCA protocol again with a control masking condition: for each test sample, replace the saliency-based mask with a random mask of identical sparsity, sampling the same number of masked positions uniformly at random. If random-mask DCA is also significantly greater than zero, then DCA measures mask-pattern leakage rather than encoding by the saliency ordering. Additionally, include an oracle condition where masks are generated from the true R_min sets; if oracle DCA is not zero, the metric itself is biased by the imputation or retraining setup.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest evidence for the headline claim is the DCA metric (Section 3.8, metric 4), which reports that after LeRF masking and retraining, samples with identical relevant inputs can be assigned different classes, and concludes that 'decision relevant information is in the Baseline inputs.' This inference is valid only if the mask pattern itself carries no class information. But the mask is a deterministic function of the saliency ranking, which depends on the full input; therefore two samples that agree on the logically relevant inputs can differ in which positions are masked, and a retrained model can exploit those positional/shape cues (Assumption D, cf. [32]). The paper acknowledges this leakage but does not bound it: no control using random masks, fixed masks, or oracle masks is reported. Consequently DCA > 0 could be caused by mask-pattern side channels rather than by the saliency score ordering encoding classification information. The headline claim therefore rests on an uneliminated confound.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a controlled benchmark, ANDOR, for evaluating saliency methods on small propositional-logic classification tasks (AND, OR, XOR top levels over stacks of gates plus a non-informative baseline block). The authors train CNNs and transformers on fully enumerated, class-balanced versions of these datasets, keep only models that reach 100% accuracy on a split test set, and then evaluate twelve saliency methods under four LeRF masking thresholds using five metrics, including Needed Information below Baseline (NIB) and Full/Minimal Double Class Assignments (DCA). The central empirical claim is that all tested saliency methods often place logically minimal relevant inputs below the baseline block (NIB > 0) and that, after masking and retraining, logically equivalent relevant-input configurations can be assigned different classes (DCA > 0), from which the authors conclude that saliency methods encode classification-relevant information into the ordering of saliency scores rather than merely reporting relevance. The paper includes a detailed assumption list, a large-scale experiment (33,600 trained models), and a public code repository.","tokens_in":13609,"tokens_out":6269,"duration_ms":56977,"significance":"The benchmark design is a genuine strength: fully enumerated logical datasets, balanced training sets, a 100%-accuracy filter, twelve saliency methods, two architectures, and multiple thresholds constitute a carefully constructed testbed, and the reported code and experiment counts make the study reproducible. If the conclusions were fully supported, the paper would provide a useful cautionary result for the interpretability community. However, the headline claim depends on two load-bearing premises that are not independently verified: first, that a 100%-accurate model on a fully enumerated deterministic dataset must reason according to one of the logically defined minimal sets R_min (Section 3.4), and second, that the DCA metric measures encoding in saliency-score ordering rather than information leakage through the input-dependent mask pattern itself. The authors explicitly label the relevant items as assumptions and cite mask-leakage work, which is honest, but the empirical evidence does not yet eliminate the mask side-channel explanation.","major_comments":[{"comment":"The Full-DCA evidence for the headline claim is confounded by input-dependent mask patterns. Since LeRF masks are computed from saliency scores, which depend on the full input, two test samples with identical logically relevant inputs can receive different masked patterns, and a retrained model can use those patterns as class-discriminative cues. The paper acknowledges that information can leak into masks (Section 2 and Section 5, citing [32]), but it reports no control such as random masks, fixed masks, or oracle masks matched on retained-input fraction. Consequently, DCA > 0 could be produced by mask-pattern side channels rather than by the saliency score ordering encoding classification information. A control condition that decorrelates mask patterns from class while preserving masking statistics is necessary before the 'encoding into the ordering' interpretation is supported.","section":"Section 3.8, metric (4) (Full-DCA); Section 5"},{"comment":"The NIB metric relies on the premise that a model reaching 100% accuracy on a fully enumerated ANDOR dataset has 'understood the task completely' and that its local reasoning must correspond to one of the logical R_min sets. On a deterministic, fully enumerated dataset, many non-logical decision rules (for example, a linear readout that uses all gate inputs, or shortcuts based on input statistics) also achieve 100% accuracy. Thus NIB > 0 may measure a mismatch between the logical ground truth and the model's actual decision rule rather than a defect of the saliency method. The assumptions are stated transparently, but the abstract's causal reading ('saliency methods can encode...') requires stronger evidence about the model's decision rule. I recommend adding an independent probe of the trained models' decision rule, e.g., exhaustive single-input and pair-input ablations to derive per-model R_min, and reporting NIB against those model-specific sets.","section":"Section 3.4 and Assumption E"},{"comment":"The definition of Full-DCA as 'the count where the relevant inputs {{d1, . . . , dl−NrB} ⊆ d | d ∈ D} map towards different classes in the retrained model' is underspecified: it does not state which samples are grouped, what the set-builder notation denotes, or how a 'mapping towards different classes' is counted. Since Full-DCA is one of the two central quantitative supports for the paper's main conclusion, this metric should be defined formally (e.g., partition the test set by the values of the logically relevant inputs, and count groups in which the retrained model's predictions are not constant), with the exact formula used in the code stated in the text.","section":"Section 3.8, metric (4) (Full-DCA) definition"}],"minor_comments":[{"comment":"The union notation in constraints (1) and (2), e.g. 'S dj∈d f D(d,j)', is not typeset correctly and should be written with a proper union symbol and with the domain of f_D stated unambiguously.","section":"Section 3.1"},{"comment":"The dataset sizes in the caption of Figure 1 ('28 = 256', '48 = 65.536', '212 = 4.096') use a comma as a decimal separator and lack superscripts; they should read 2^8 = 256, 4^8 = 65,536, and 2^12 = 4,096.","section":"Section 3.3 and Figure 1 caption"},{"comment":"The logical accuracy metric says 'combinations of undefined inputs result in undefined,' but the text never states how 'undefined' is counted in the reported accuracy; please specify the treatment of undefined outputs in the numerator and denominator of the accuracy calculation.","section":"Section 3.8, metric (2)"},{"comment":"The description of the masking thresholds should be unified: the text mentions 'one of four thresholds,' then lists a Baseline-based threshold and factors t1.0, t0.8, t0.5; please state explicitly how the Baseline threshold is combined with the factor thresholds and whether all four are used in every experiment.","section":"Section 3.7"},{"comment":"The sentence 'Shah et al. [36] called some similar phenomena, information leakage, but we argue that this is some form of internal model encoding' asserts the conclusion rather than arguing it; given the mask-side-channel concern, this overstates what the experiments can currently distinguish.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"This manuscript is a solid empirical contribution to saliency evaluation with an unusually careful benchmark design, but its central interpretive claim currently outruns the controls it reports. The DCA confound and the R_min attribution premise are both load-bearing; I would advise the editor that the revision should either add the missing control experiments or substantially soften the headline claim to a statement about mismatches between saliency-based masks and logical ground truth rather than about encoding into score orderings. The paper is within scope for cs.LG and is likely to be of interest to the XAI community after these revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper earns a serious read. It contributes ANDOR, a synthetic logical dataset with a non-informative baseline block, plus metrics (NIB, DCA) for testing saliency methods. The empirical pattern is clear across CNN and Transformer models and twelve saliency methods: relevant inputs frequently get lower scores than baseline inputs, and masks derived from saliency rankings allow a retrained model to recover class information. I expect the descriptive findings to hold up. The contested part is the headline inference that saliency methods encode classification info into the ordering; that needs an extra control before I'd call it proven.\n\nWhat is genuinely new: the ANDOR framework with fully enumerated inputs, the explicit irrelevant-input baseline, the distinction between minimal and maximal information coverage, and the application to a broader method set. This goes beyond the earlier logical-dataset work by Tritscher et al. and Yalcin et al. The 100% accuracy filter is a sensible way to keep the ground truth clear, and the code being public is a plus. The metrics are anchored to external truth tables, so the circularity burden is low.\n\nThe soft spots are where the reader points. Assumption E, baseline inputs should never outrank logically relevant ones, is a normative claim. A model can solve ANDOR with a non-logical shortcut; if it does, NIB measures a mismatch between the logical ground truth and the model's actual decision rule, not a flaw in the saliency map. The paper treats 100% accuracy as equivalent to logical understanding but never verifies the model's actual reasoning. That is stated as an assumption, but it is load-bearing.\n\nThe DCA evidence is more relevant to the encoding claim, and the stress-test note lands: the mask is a deterministic function of the saliency ranking, which depends on the whole input. Samples can have identical relevant inputs and different masks, so a retrained model can exploit mask shape or position. The paper acknowledges mask leakage but does not bound it; no random-mask or fixed-mask control is reported. DCA > 0 is therefore consistent with leakage through the mask pattern, not just through the ordering.\n\nMinor issues: no significance tests, manual hyperparameter selection, and the reproducibility statement points to a GitHub link without a commit hash. All fixable.\n\nBottom line: this is a solid benchmark paper with an overreaching interpretation. The benchmark and the raw results deserve peer review; the 'encoding' conclusion should be softened or supported with a random-mask control. I would cite it for the benchmark and bring it to the reading group.","headline":"A solid controlled benchmark (ANDOR) with clear descriptive findings, but the 'encoding into the ordering' claim needs a random-mask control before it is fully convincing.","tokens_in":14203,"tokens_out":2852,"would_cite":true,"duration_ms":26416,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Saliency methods can encode classification-relevant information into the ordering of their scores, beyond the intended relevance signal.","keywords":["saliency maps","feature attribution","explainable AI","logical datasets","model reasoning ground truth","information encoding","ANDOR dataset","masking evaluation"],"falsifier":"Train a model on the same ANDOR data to 100% accuracy using a deliberately non-logical decision rule, such as a linear threshold over a non-minimal weighted combination of inputs, then apply a provably faithful saliency method like exact input gradients of that rule. If NIB greater than zero or DCA greater than zero still appears, then the metrics are detecting the mismatch between the assumed logical ground truth and the model's actual rule, not an encoding carried by the saliency method; if NIB and DCA are zero, the paper's interpretation is supported.","tokens_in":13235,"feed_emoji":"🧠","tokens_out":7135,"duration_ms":59319,"temperature":0.7,"pith_summary":"The paper aims to test whether saliency maps, the heatmaps that claim to show which inputs drive a model's decision, actually put the logically necessary inputs on top. It builds ANDOR, a family of toy datasets built from AND, OR, and XOR gates plus an explicitly irrelevant baseline block, and enumerates every possible input so a 100%-accurate model can be assumed to have learned the task completely. The paper introduces metrics that ask whether the minimal set of inputs needed to decide each sample is scored above the baseline, and whether models retrained on masked inputs can still recover the class. On all twelve tested saliency methods, the answer is no: relevant inputs frequently fall below the baseline, and masks carry enough class-discriminative information that retrained models beat the logical accuracy of what should remain. The central claim is that saliency methods do not merely rank relevance; they encode classification information into the ordering of the scores.","feed_headline":"Saliency scores hide class info in their ordering","feed_subtitle":"All 12 tested methods put minimal inputs below an irrelevant baseline and leak class info into masks.","key_machinery":"The machinery is the ANDOR dataset framework: four blocks of propositional gates, AND, OR, XOR, plus a baseline block of explicitly irrelevant inputs, layered under a top-level gate and fully enumerated so that the set of all possible model reasoning paths is known. From this, the paper defines the minimal information coverage R_min, the smallest sets of inputs that logically determine the output, and a non-informative baseline, the baseline block that should never outscore a relevant input. On top sit two metrics: NIB, the percentage of samples where a minimal relevant input scores below the highest baseline input, and DCA, which checks whether masked samples with identical relevant inputs map to different classes in a retrained model. The baseline and the minimal-coverage comparison are the load-bearing objects: they turn 'does a saliency map explain the model?' into a concrete ordering test.","core_discovery":"On the paper's terms, the central discovery is that saliency methods can encode classification-relevant information into the ordering of their scores, beyond the intended relevance signal. Across 144 experiment settings with CNN and transformer models on the ANDOR datasets, every one of the twelve methods showed a positive NIB, meaning at least one logically minimal relevant input scored below the highest baseline input, and retrained models consistently achieved higher accuracy than the logical accuracy of the masked data. The DCA metrics show that identical relevant inputs can map to different classes depending on masked baseline inputs, with the effect growing on the most complex setting, the quaternary-input XOR-top-level, and with higher masking thresholds. The paper interprets this not as an input-sensitivity failure along the lines of prior sanity checks, but as a systematic encoding of information into the score order.","pith_inferences":["An extension the paper leaves implicit: the same test run with a provably faithful attribution method, for example exact gradients of a hand-written decision rule, would separate 'the saliency method encodes' from 'the model encodes'.","The encoding result suggests a reinterpretation of saliency maps as lossy codes, where class information may live in rank statistics rather than pointwise scores; a decoding-based evaluation could test this directly.","Because the paper only treats binary outputs, an obvious testable extension is a multi-class or continuous-output ANDOR variant to see whether the encoding effect is an artifact of binary classification.","The paper leaves the cause of encoding open; one concrete experiment is to ablate internal aggregation steps of a method, for example layer-wise combination in LRP or attention rollout, and measure whether DCA drops when less information is aggregated."],"forward_implications":["If saliency scores encode class information in their ordering, then evaluating a saliency method by whether high-scored regions are plausible is not enough: the same score map can appear misleading while its full ordering carries a usable signal.","Retraining on masked inputs cannot by itself certify a saliency method, because the mask can leak the very information the explanation claims to remove.","The ANDOR framework provides a cheap, exhaustive trust test for new saliency methods: a method that keeps NIB at zero and DCA at zero on these nine datasets has demonstrated the basic property of keeping minimal relevant inputs above an irrelevant baseline.","The encoding effect intensifies with task complexity inside the controlled setting, suggesting that failures seen in simple logic can appear in more realistic tasks that contain redundant, complementary, or exclusive information."],"supporting_citations":[{"why":"Supplies the retraining-on-masked-inputs protocol that underlies the DCA and logical-accuracy comparisons.","marker":"[16]"},{"why":"Establishes that masks can contain classification-relevant information, the phenomenon the paper analyses through exhaustive redundancies.","marker":"[32]"},{"why":"Prior logical-dataset evaluation with a 100%-accurate model that the paper extends by separating minimal from maximal information coverage.","marker":"[44]"},{"why":"Prior attempt to verify input necessity in logical formulas using attribution scores, which motivates the model-reasoning ground-truth approach.","marker":"[47]"},{"why":"Shows that saliency methods can fail input-sensitivity sanity checks, motivating the controlled ground-truth test.","marker":"[2]"},{"why":"Supplies the transformer LRP implementations among the twelve saliency methods tested.","marker":"[7]"}],"fun_headline_variants":["Saliency order leaks class info","Saliency scores encode class info, not just relevance","Ordering of saliency scores carries hidden class data","Saliency maps secretly encode class info in score order","Twelve saliency methods leak class info via ordering"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a model reaching 100% accuracy on the fully enumerated ANDOR dataset has understood the task completely, so its local decisions must correspond to one of the logical minimal sets R_min; if the network uses a non-logical decision rule, or the masking procedure itself introduces statistical cues, then NIB and DCA measure the mismatch between the ground-truth definition and the model's actual behaviour rather than a defect in the saliency methods.","fun_headline_variants_meta":{"raw":{"variants":["Saliency order leaks class info","Saliency scores encode class info, not just relevance","Ordering of saliency scores carries hidden class data","Saliency maps secretly encode class info in score order","Twelve saliency methods leak class info via ordering"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00018,"raw_usage":{"total_tokens":1294,"prompt_tokens":928,"completion_tokens":366,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":294}},"tokens_in":544,"tokens_out":366,"duration_ms":3539,"temperature":1.0,"reasoning_tokens":294,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:50:21.789083+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a model on the same ANDOR data to 100% accuracy using a deliberately non-logical decision rule, such as a linear threshold over a non-minimal weighted combination of inputs, then apply a provably faithful saliency method like exact input gradients of that rule. If NIB greater than zero or DCA greater than zero still appears, then the metrics are detecting the mismatch between the assumed logical ground truth and the model's actual rule, not an encoding carried by the saliency method; if NIB and DCA are zero, the paper's interpretation is supported.","supporting_citations":[{"cited_title":"A benchmark for interpretability methods in deep neural networks","cited_arxiv_id":null,"evidence_quote":"Supplies the retraining-on-masked-inputs protocol that underlies the DCA and logical-accuracy comparisons."},{"cited_title":"Evaluation of post-hoc xai ap- proaches through synthetic tabular data","cited_arxiv_id":null,"evidence_quote":"Prior logical-dataset evaluation with a 100%-accurate model that the paper extends by separating minimal from maximal information coverage."},{"cited_title":"Sanity checks for saliency maps","cited_arxiv_id":null,"evidence_quote":"Shows that saliency methods can fail input-sensitivity sanity checks, motivating the controlled ground-truth test."},{"cited_title":"Transformer inter- pretability beyond attention visualization","cited_arxiv_id":null,"evidence_quote":"Supplies the transformer LRP implementations among the twelve saliency methods tested."}],"review_version":1}