{"id":"e3373d1d-873b-40ff-8672-702c2b116e9a","arxiv_id":"2506.23975","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Higher-relevance concept groups yield shorter contrastive explanations, and explanation length is stable under 180-degree rotation but not under 10-degree rotation or Gaussian noise.","lead":"This paper evaluates a concept-based contrastive explanation method for image classification, measuring how explanation length changes with concept relevance and under image rotation and noise. It finds higher-relevance concepts give shorter explanations and that explanations are stable under 180-degree rotation but not small perturbations, though the main trend is built into the relevance definition.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"R1's monotonic length trend is likely an artifact of cumulative-relevance binning; the paper never shows it survives fixed-size concept bins.","rationale":"R1 is the main empirical pillar of the abstract's claim. The four relevance ranges are defined by cumulative relevance mass, not by fixed concept counts; under any skewed relevance distribution, this guarantees that high-relevance bins contain fewer concepts than low-relevance bins. Because explanation length counts unique concepts from those bins, the monotonic increase in length is determined by the binning scheme. The paper does not report bin sizes, so the ANOVA cannot be interpreted as evidence for H1. The reader identified foil selection as the weakest assumption, but that concern is orthogonal: the cumulative-quantile artifact would persist under any foil. I therefore recommend keeping the conditional verdict but making acceptance contingent on demonstrating that H1 survives fixed-size bins; if it does not, the central claim should be rejected. The Gaussian-noise misreading in the Discussion is also a correctness issue, but it does not change H2's rejection and is less load-bearing.","tokens_in":10554,"tokens_out":9292,"duration_ms":103890,"concrete_test":"Re-run the R1 analysis with fixed-size relevance bins: for each image, take the top 25, top 50, top 100, and top 200 concepts by relevance rank instead of cumulative-relevance quantiles, and compute the same explanation lengths. If the monotonic increase in median length disappears or reverses, H1 is an artifact of cumulative-relevance binning. As a complementary diagnostic, report the median number of concepts in each cumulative-relevance bin; if these counts closely track the reported medians (17/40/78/100), the artifact is confirmed.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The load-bearing support for the central claim that higher concept relevance leads to shorter explanations is the R1 analysis in Section 4. That analysis defines the four relevance ranges as cumulative-relevance quantiles: 'very strong' is the top 25% of total relevance, 'strong' the next 25%, and so on. In any realistically skewed relevance distribution, the top 25% of total relevance contains few concepts, while the bottom 25% contains many. Explanation length is the number of unique concepts in the explanation (Algorithm 1, lines 16-18), so the very-strong range can only produce short explanations and the very-low range can only produce long ones. The reported medians of about 17, 40, 78, and 100 therefore largely mirror the concept count per range. The ANOVA F-value of 354.41 only shows that the four constructed groups have different mean lengths; it cannot distinguish a causal effect of relevance strength from the mechanical effect of bin size. The paper never reports the number of concepts per relevance range, so the reader cannot tell whether H1 is an empirical finding or a definitional consequence. The foil-selection issue is secondary: even a random foil leaves target-side bin sizes small in the high-relevance range, so the R1 trend would persist as an artifact.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a concept-based contrastive explanation method for image classification that pairs a target instance with the most cosine-similar instance from a contrast class, extracts concept relevance scores via Concept Relevance Propagation (CRP), and reports as an explanation the concepts unique to one of the two compared images. The method is evaluated along two axes: (R1) whether explanation length varies across relevance ranges defined as cumulative-relevance quantiles, and (R2) whether explanation length remains stable under Gaussian noise, 10-degree rotation, and 180-degree rotation. The authors report that higher-relevance concepts yield shorter explanations (H1 supported) and that robustness depends strongly on the augmentation type (H2 partially supported). The central positive claim is, however, undermined by the way relevance ranges are defined, and the discussion of the Gaussian noise result directly contradicts the tabulated and plotted results.","tokens_in":10765,"tokens_out":4158,"duration_ms":40904,"significance":"If the R1 finding were valid, the paper would make a useful empirical contribution to concept-based explainability, showing that relevance strength is a lever on explanation conciseness. The implementation is a concrete instantiation of earlier conceptual work (Finzel et al.), and the choice of a semantically close contrast pair is a reasonable design. However, the main statistical claim is potentially a definitional artifact of cumulative-relevance binning rather than an empirical property of the explanation method. The noise-robustness finding is also misdescribed in the Discussion, and the foil-selection strategy is not stress-tested. The paper's significance therefore depends on a re-analysis that separates bin-size effects from true relevance effects.","major_comments":[{"comment":"The primary result H1 is likely an artifact of the binning scheme. Relevance ranges are defined as cumulative-relevance quantiles: 'very strong' is the top 25% of total relevance, 'strong' is the next 25%, and so on. In any skewed relevance distribution, the top 25% of total relevance is concentrated in few concepts while the bottom 25% spans many concepts. Because explanation length is the number of unique concepts produced by Algorithm 1 (lines 16-18), the reported medians of approximately 17, 40, 78, and 100 closely mirror the concept counts per bin rather than any contrastive property of the explanation. The ANOVA F-value of 354.41 only shows that the four constructed groups differ in mean length; it cannot distinguish a relevance effect from a mechanical bin-size effect. The manuscript never reports the number of concepts in each relevance range or the shape of the relevance distribution. To support H1, the authors should either use fixed-size concept bins (e.g., the top 10 concepts vs. the next 10 concepts) or normalize explanation length by the number of concepts in each bin, and show that the monotonic trend survives.","section":"Section 4"},{"comment":"The Discussion misstates the Gaussian noise result. Section 4 states that 'across all relevance levels, explanation lengths are consistently and significantly lower for noisy images compared to the original ones,' and Figure 6 shows downward shifts in the distributions. Section 5, however, claims that 'Explanation lengths increased significantly across all relevance ranges when noise was added' and interprets this as the model losing focus on a small set of discriminative concepts. Table 1 confirms the Section 4 description with positive t-values (e.g., t=25.7762 for very strong relevance), indicating shorter explanations under noise. This internal contradiction directly affects the interpretation of H2 and the robustness conclusions; the Discussion must be corrected to describe the actual direction of the effect.","section":"Section 5"},{"comment":"The explanation length and all downstream results depend on the selection of exactly one contrastive image, namely the embedding-nearest neighbor from the opposite class. The algorithm does not vary this selection, compare against random foils, or test sensitivity to the similarity criterion. Because uniqueness is defined against this single foil, the sets of unique concepts and hence all measured lengths in R1 and R2 could change markedly if a different foil image were used. The authors should at least report a sensitivity analysis (e.g., lengths obtained with the second- and third-nearest neighbors, or with a random foil distribution) to establish that the reported trends are not specific to one arbitrarily chosen pairing.","section":"Algorithm 1"}],"minor_comments":[{"comment":"The p-values are reported as '0.0000'; this should read '<0.0001' or similar to avoid implying an exact zero probability.","section":"Table 1"},{"comment":"There is a spacing typo in the phrase 'whereP is the observed fact' twice; 'whereP' should be 'where P'.","section":"Section 2"},{"comment":"The statement 'The maximum possible explanation length is 512' is not derived or justified; the maximum depends on the number of extractable concepts in the chosen CRP layer and how uniqueness is counted, so it should be either formally derived or stated as an upper bound with explanation.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The paper's central H1 finding is, on the present evidence, a likely artifact of cumulative-relevance binning. This is a load-bearing issue that cannot be resolved by wording changes alone; the authors must re-analyze with fixed-size bins or otherwise control for bin size. The internal contradiction about Gaussian noise in Section 5 is also serious but easily corrected. I see no sign of misconduct, only an analysis that needs stronger controls and a discussion that needs careful rewriting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the headline result—higher concept relevance leads to shorter explanations—is probably not an independent empirical finding; it's built into how the relevance ranges are defined. The robustness comparison is more informative, but the Discussion misreads the Gaussian noise outcome. Both issues are fixable, but they change what the paper actually shows.\n\nWhat's genuinely new here: an end-to-end implementation of the CoReX-style idea using VGG16, CRP, cosine similarity on embeddings, and set-difference to produce contrastive explanations, plus a systematic comparison of explanation length across relevance ranges and under three augmentations. The statistical tests are simple and appear correctly applied. The writing is clear, and the limitations section is honest about the lack of human evaluation and the narrow augmentation set.\n\nNow the soft spots. The R1 analysis defines relevance ranges as cumulative-relevance quantiles: very strong is the top 25% of total relevance, strong the next 25%, and so on. In any realistically skewed relevance distribution, the top 25% of total relevance contains few concepts, while the bottom 25% contains many. Since explanation length is just the count of unique concepts in each range, the monotonic increase in medians (17, 40, 78, 100) largely mirrors the number of concepts available per bin. The paper never reports how many concepts fall into each range, so the reader cannot separate a causal effect of relevance from a definitional consequence. The ANOVA F-value of 354 only says the constructed groups have different means; it cannot distinguish the mechanical bin-size effect. The stress-test concern is valid, and it is not rescued by the foil-selection issue: even a random foil would leave the high-relevance bin small and the low-relevance bin large.\n\nSecond, the Discussion states that Gaussian noise increased explanation lengths across all ranges, but Table 1 and Figure 6 show the opposite: paired t-tests are strongly positive (e.g., 25.78 for very strong), meaning original lengths are higher, and the boxplots show downward shifts. That is a direct contradiction in the reported results and needs a correction.\n\nMinor but worth noting: the foil is always the single nearest neighbor from the contrast class, and the paper never varies foil selection, so the entire length distribution rests on one pairing choice. No baselines, no code or data release, and the preference for shorter explanations is assumed from prior work rather than tested here—though the authors do acknowledge this.\n\nWho this is for: people working on concept-based or contrastive XAI who want a cautionary example of how explanation metrics can inherit artifacts from evaluation design. It is not a strong empirical contribution in its current form, but it is a legitimate object for peer review. I would send it to referees with explicit instructions to examine the R1 binning artifact and the noise misstatement. With those addressed, it could be a reasonable workshop- or short-conference-level paper.","headline":"The main claim about relevance and explanation length is likely baked into the binning scheme; the robustness comparison is useful but contains a factual misreading of the noise results.","tokens_in":11298,"tokens_out":2497,"would_cite":false,"duration_ms":28603,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Contrastive explanations for image classifiers get shorter as concept relevance rises, and their length is stable under a 180-degree rotation but not under a 10-degree rotation or Gaussian noise.","keywords":["contrastive explanation","concept-based explanation","explanation complexity","explanation robustness","instance similarity","concept relevance","image classification","Concept Relevance Propagation"],"falsifier":"Re-run the explanation algorithm on the same test images while varying the foil: use the second-most-similar image, a random image from the contrast class, or an average over several near neighbours, and recompute explanation lengths within the four relevance ranges. If the monotone increase from very strong to very low relevance disappears, reverses, or becomes non-significant under alternative foil selection, then the paper's central length result is an artifact of the single nearest-neighbor pairing rather than a general property of concept relevance.","tokens_in":10317,"feed_emoji":"🤖","tokens_out":9667,"duration_ms":96730,"temperature":0.7,"pith_summary":"This paper proposes and evaluates a method for contrastive explanations of image classifiers that answers 'why this class rather than that one?' using human-understandable concepts. The central claim is that the concepts a model ranks as highly relevant produce shorter, less complex explanations, and that explanation length is mostly stable under a 180-degree rotation but not under a 10-degree rotation or Gaussian noise. The experiments, run on a fine-tuned VGG16 classifier distinguishing teapots from vases, show median explanation length rising from about 17 concepts in the top relevance range to about 100 in the bottom range, with an ANOVA F-value of 354.41. The robustness tests show the top relevance ranges surviving a 180-degree flip ($p > 0.05$) while Gaussian noise lengthens explanations significantly in every relevance range. The work matters because short, stable explanations are a prerequisite for trusting a model's stated reasons in real-world image classification.","feed_headline":"High-relevance concepts yield shorter contrastive explanations","feed_subtitle":"A teapot-vase classifier keeps explanation length stable under a 180° flip, but 10° rotation and noise break it.","key_machinery":"The central mechanism is the instance-similarity foil selection: for each explained image, Algorithm 1 chooses as the contrast the single opposite-class image whose embedding has the maximum cosine similarity, then differences the two images' concept sets. The explanation is the set of unique concepts—those appearing in only one of the two images—rendered as a natural-language sentence of the form 'classified as teapot instead of vase because it contains ... and does not contain ...'. Concept relevance comes from Concept Relevance Propagation, which isolates the neurons associated with a concept through a masked backward pass; the paper then sorts concepts by relevance and cuts them into four quartile ranges based on cumulative relevance. Explanation length, the count of concepts in the unique set, is the paper's operational measure of complexity, and it is the quantity compared across relevance ranges and across original versus augmented images.","core_discovery":"On its own terms, the paper discovers a monotone relation between concept relevance and explanation complexity: explanations assembled from very strong concepts are short and focused, while explanations from very low relevance concepts are long and diffuse. For each target image, the method picks the contrast-class image whose embedding is most cosine-similar, extracts per-concept relevance scores for both images with Concept Relevance Propagation, and keeps only the unique concepts that appear in one image but not the other; explanation length is the number of those concepts. Grouping concepts by cumulative relevance quartiles (top 25%, 25–50%, 50–75%, bottom 25%) yields median lengths of roughly 17, 40, 78, and 100, and the ANOVA test confirms the differences are not chance. The robustness finding is nuanced: a 180-degree rotation leaves the top three relevance ranges statistically unchanged (only the very low range shifts, $p = 0.0002$), while a 10-degree rotation shortens explanations in the strong and low ranges, and Gaussian noise significantly lengthens explanations in all four ranges. The paper interprets this as support for $H_1$ and partial support for $H_2$, and frames the noise sensitivity as the model losing focus on a small discriminative concept set.","pith_inferences":["An implication the paper leaves implicit is that the entire length distribution rests on the choice of a single foil image; varying the foil selection, such as averaging over several near neighbours, would show whether the monotone trend is a property of concept relevance or an artifact of that one nearest-neighbor pairing.","The noise result suggests explanation length could double as a detector of model uncertainty or shortcut reliance, but the paper does not test this link.","The teapot-versus-vase setup could be extended to other visually similar class pairs; if the quartile length ordering replicates, the method would be a general recipe for simple contrastive explanations rather than a dataset-specific observation."],"forward_implications":["Restricting explanations to the top relevance quartile can reduce median explanation length from about 100 concepts to about 17 while still reporting the same contrast.","Explanation length can serve as a measurable, model-internal proxy for explanation complexity in concept-based methods.","Robustness evaluation should include small geometric perturbations: a 10-degree rotation changes explanation length in the strong and low relevance ranges, while a 180-degree rotation leaves the top ranges unchanged.","Gaussian noise should be treated as a serious stress test for concept-based explanation stability, since it significantly lengthens explanations in every relevance range."],"supporting_citations":[{"why":"Supplies Concept Relevance Propagation, the technique used to extract per-concept relevance scores from the network's forward and backward passes.","marker":"[1]"},{"why":"Provides the concept-based contrastive explanation framework and the use of quantiles over relevance distributions that this paper implements.","marker":"[9]"},{"why":"Motivates using embedding similarity for selecting contrastive images, the core idea implemented in Algorithm 1.","marker":"[10]"},{"why":"Provides VGG16, the network architecture that is fine-tuned and then probed for explanations.","marker":"[27]"},{"why":"Supplies ImageNet, the source dataset from which the teapot and vase subsets are drawn.","marker":"[6]"},{"why":"Defines the ILSVRC 2012–2017 challenge dataset and class hierarchy used for the experiments.","marker":"[25]"},{"why":"Underpins the contrastive 'why P rather than Q?' framing and the cognitive rationale for using foils.","marker":"[19]"},{"why":"Justifies treating shorter, concise explanations as preferable, which motivates length as the complexity metric.","marker":"[14]"},{"why":"Supplies the ANOVA test that establishes the significance of length differences across relevance ranges.","marker":"[11]"},{"why":"Supplies the paired t-test that evaluates length differences between original and augmented images.","marker":"[29]"}],"fun_headline_variants":["High relevance shortens contrastive explanations","Concept relevance drives explanation brevity","Strong concepts, short explanations","Relevance cuts explanation length, noise breaks it"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the single opposite-class image the model judges most similar is a representative foil; if a different foil image were chosen, the set of unique concepts and every measured explanation length could change, and with them the R1 and R2 results.","fun_headline_variants_meta":{"raw":{"variants":["High relevance shortens contrastive explanations","Concept relevance drives explanation brevity","Strong concepts, short explanations","Relevance cuts explanation length, noise breaks it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000217,"raw_usage":{"total_tokens":1450,"prompt_tokens":972,"completion_tokens":478,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":428}},"tokens_in":588,"tokens_out":478,"duration_ms":5153,"temperature":1.0,"reasoning_tokens":428,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:26:47.153767+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the explanation algorithm on the same test images while varying the foil: use the second-most-similar image, a random image from the contrast class, or an average over several near neighbours, and recompute explanation lengths within the four relevance ranges. If the monotone increase from very strong to very low relevance disappears, reverses, or becomes non-significant under alternative foil selection, then the paper's central length result is an artifact of the single nearest-neighbor pairing rather than a general property of concept relevance.","supporting_citations":[{"cited_title":"When a Relation Tells More Than a Concept: Exploring and Evaluating Classifier Decisions with CoReX","cited_arxiv_id":"2405.01661","evidence_quote":"Provides the concept-based contrastive explanation framework and the use of quantiles over relevance distributions that this paper implements."},{"cited_title":"In: Julián, V., Ca- macho, D., Yin, H., Alberola, J.M., Nogueira, V.B., Novais, P., Tallón-Ballesteros, A.J","cited_arxiv_id":null,"evidence_quote":"Motivates using embedding similarity for selecting contrastive images, the core idea implemented in Algorithm 1."}],"review_version":1}