{"id":"d7756c7b-baf6-46a4-a249-52c9c2bfe222","arxiv_id":"2505.04647","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ChannelExplorer turns activation channel summaries into scatterplots, Jaccard similarity matrices, and heatmaps, giving users a way to explore class separability in neural networks.","lead":"ChannelExplorer is an interactive visualization tool that shows how different parts of a deep neural network separate or confuse image classes by summarizing neuron activations. A generalist might read it to see a practical way to debug models and datasets without changing the network.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Scalar summaries discard spatial activation structure; if location matters for class separability, all three main views inherit the loss, and Section 5.3 never validates its central invariance assumption.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing premise I find: scalar channel summaries discard spatial location, and the paper's validity depends on that being harmless for class separability. I agree with the reader's CONDITIONAL verdict and do not move it, but I would make the condition explicit: the authors should validate the summarization assumption quantitatively, because all three primary views, and therefore the four use cases and the pruning scenario, flow through it. The other flagged issues, such as the unresolved 'see ??' in Section 5.3.1, the lack of exact mAP values in Section 7.3, and the convenience-sample user study in Section 7.5, are real but secondary; they affect evidence quality without undermining the core mechanism as directly as the spatial invariance assumption. This is not a rejection: a positive result from the proposed synthetic test would substantially de-risk the central claim, and the tool's open-source promise and detailed system description are genuine support. The condition is that the summary-to-full-activation separability correlation be demonstrated, not merely asserted.","tokens_in":21181,"tokens_out":5282,"duration_ms":58572,"concrete_test":"Construct a synthetic two-class dataset in which every image contains the same multichannel feature patches but arranged in complementary spatial layouts (e.g., feature A top-left vs. bottom-right), so per-channel sums, L2-norms, and max values are identical across classes. Run a trained CNN, compute ChannelExplorer's per-channel scalar summaries and the resulting UMAP embeddings and Jaccard matrix, and measure class separability (linear SVM or k-NN accuracy and silhouette score) on summary vectors versus full flattened activation tensors. If summary-space separability is near chance while raw activations are clearly separable, the Section 5.3 assumption fails for those cases and the three main views are not trustworthy. A complementary check: recompute the Figure 7 comparison with a quantitative separability metric instead of visual inspection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"ChannelExplorer's central claim, that its three coordinated views expose class separability, rests on the summarization assumption stated in Section 5.3: 'This activation magnitude can be identified with only the intensity of output activation channels, regardless of the spatial location of the high-intensity pixels.' Every downstream view consumes this scalar summary S(I): the Scatterplot embeds vectors of per-channel scalars, the Jaccard matrix selects top-A-eta channels by scalar value, and the Heatmap's ordering and stripe interpretation use the same scalars. The assumption is not established. CNNs are translation-covariant: a feature appearing at different image locations lights the same channel at different spatial positions, so two classes can have identical per-channel intensity totals while remaining perfectly separable by where activations occur (e.g., object parts arranged differently, or a channel that fires on the object versus on the background). Figure 7's comparison is qualitative, and Section 5.3.2's dismissal of raw activations rests on a pixel-wise distance that is not shown to be the right baseline. The raw overlay available on click is details-on-demand after summary-driven navigation, so it cannot rescue the overviews if the summaries conflate classes. The paper needs a quantitative check that separability in summary space tracks separability in the full activation tensor; without it, the headline use cases are conditional on an unvalidated premise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ChannelExplorer, an interactive visual analytics system for exploring class separability in image-based deep neural networks. The system summarizes each activation channel into a scalar value via four summarization functions, then presents three coordinated views: a Scatterplot View embedding per-image channel-summary vectors, a Jaccard Similarity View measuring overlap between top-activated channel sets, and a Heatmap View ordering channels by class-pairwise distance or other metrics. The paper claims that these views help experts identify inter- and intra-class confusion, derive class hierarchies, find mislabeled images, determine which channels matter, and locate latent states in generative models such as Stable Diffusion. The contributions are demonstrated through four use cases on InceptionV3, SRResNet, and Stable Diffusion, plus a nine-participant user study and a released open-source implementation.","tokens_in":21364,"tokens_out":4576,"duration_ms":45884,"significance":"If the central premise holds, ChannelExplorer addresses a genuine gap: it provides a scalable, model-agnostic, channel-level debugging workflow for image-based networks, going beyond layer-level or metric-level tools. The paper's concrete strengths are its open-source implementation, publicly accessible demo, support for multiple architectures (CNNs, GANs, ResNet, Stable Diffusion), and four use cases spanning classification and generation. The user study, while small, is a reasonable first step. However, the significance is conditional on validating the core summarization assumption, because all three primary views consume scalar channel summaries; if spatial activation structure matters for class separability, the headline use cases would rest on an unvalidated premise. The quantitative pruning claim also needs stronger evidence before the claimed model-refinement benefit can be accepted.","major_comments":[{"comment":"The central claim of the paper depends on the assertion in Section 5.3 that activation magnitude can be identified with intensity alone, regardless of the spatial location of high-intensity pixels. This premise is load-bearing: the Scatterplot View embeds vectors of per-channel summaries, the Jaccard Similarity View selects top-A_eta channels by scalar summary, and the Heatmap View orders and interprets channels using the same scalars. CNNs are translation-covariant, so two classes can have identical per-channel intensity totals while remaining separable by where activations occur. The paper does not validate this assumption quantitatively; Figure 7 is a qualitative comparison, and Section 5.3.2's dismissal of raw activations relies on a pixel-wise distance baseline that is not shown to be the right alternative. Please add a quantitative check that class separability in summary space tracks separability in the full activation tensor, for example by comparing k-nearest-neighbor or clustering accuracy computed on summary vectors versus raw channel vectors across layers, for at least the InceptionV3/ImageNet case and one generative case. Without such a check, the use cases are conditional on an unvalidated premise.","section":"Section 5.3, Eq. (1)"},{"comment":"The pruning claim in Section 7.3 is not quantitatively supported. The text reports removing 74% of channels without affecting mAP and a 12.1% CPU inference-time improvement, with a 50% GPU improvement after removing 70% of channels, but it gives no exact mAP values, no baseline mAP, no repeated runs, no error bars, and no statistical test. It also does not specify the precise criterion for stopping channel removal or the variance across model initializations. Because this is one of the four headline use cases and is used to argue for model refinement, please provide a table or figure with exact mAP values for baseline and pruned models, standard deviations over repeated runs, and the corresponding numbers for the two VGG16 baselines, together with a precise description of the pruning protocol.","section":"Section 7.3"},{"comment":"The user evaluation is described as showing that variations of DR methods, KMeans clustering, and cluster perception do not affect the decision to identify target layers or channels, but the study as reported did not systematically vary these factors and measure decision consistency; participants appear to have used a fixed tool configuration with a facilitator explanation. In T4, all participants selected channels in the bottom half of the heatmap, which may reflect the default ordering or the facilitator's guidance rather than independent identification of misleading channels. Please either temper the findings to what the study actually supports or add a controlled comparison with alternative configurations, inter-participant agreement metrics, and a baseline condition without the tool.","section":"Section 7.5"},{"comment":"The outputs of the Jaccard Similarity View and Heatmap View depend on several user-chosen parameters: eta in A_eta = ceil(eta k), the activation threshold in the summarization function, the default summarization function, and the channel-ordering metric. The paper justifies the defaults by qualitative observation ('we observed ... worked to our expectations') and does not provide a sensitivity analysis. Since the confusion hierarchy and the channel-contribution claims build on these choices, please report how Jl and the heatmap ordering vary with eta and with the choice of summarization function, or provide a principled selection criterion grounded in quantitative separability measures.","section":"Sections 5.3.2 and 5.3.3"}],"minor_comments":[{"comment":"In the L2-norm definition, the second summation runs over x=1 to h; the index should be y=1 to h.","section":"Section 5.3, Eq. (2)"},{"comment":"There is a dangling cross-reference in the text: '(see ?? in the supplement)' should be resolved to a specific section or figure before publication.","section":"Section 5.3.1"},{"comment":"The caption mentions 'Geometric Threshold function' while the text refers to 'Sum of threshold'; please unify the terminology.","section":"Figure 5"},{"comment":"There is a typo: 'interferference time' should be 'inference time'.","section":"Section 7.3"},{"comment":"The notation A_eta is typeset ambiguously; please define it explicitly as A_eta = ceil(eta k) with a clear subscript.","section":"Section 5.3.2"},{"comment":"The reference for OpenAI Microscope [63] lacks author and year information; please complete the citation.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for TVCG and the system appears to be a useful contribution if the summarization premise can be validated. The main risks are the unvalidated spatial-invariance assumption in Section 5.3 and the unsupported quantitative details in the Section 7.3 pruning claim. Both are addressable with additional experiments, so I view this as a major-revision rather than a reject."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ChannelExplorer is a real piece of work: the authors have built a tool, released the code, and demonstrated four use cases that look plausible. The new content is the integration—scalar summaries of activation channels feeding three coordinated views (scatterplot, Jaccard similarity, heatmap) with pseudo-class support and per-layer confusion hierarchies. The paper does a good job of situating itself against a crowded interpretability literature and is unusually honest about limitations.\n\nThe soft spots are real but not fatal. The load-bearing assumption is in Section 5.3: that a channel's activation can be summarized by intensity alone, ignoring spatial location. Every downstream view consumes that scalar. The paper doesn't validate it; the comparison in Figure 7 is qualitative and the dismissal of raw activations in 5.3.2 relies on a pixel-wise distance baseline that may not be the right one. This is a testable claim and the paper would be much stronger with a quantitative check that class separability in summary space tracks separability in the full activation tensor.\n\nOther issues are minor to moderate. The pruning results in 7.3 have no exact mAP values, no repeated runs, no error bars. The user study is a convenience sample of nine grad students with an author as facilitator. There's an unresolved 'see ??' placeholder in 5.3.1. And the default summarization/ordering metrics were selected post hoc to produce the expected stripe patterns—that's a mild concern, since it's a form of fitting to the display.\n\nNone of these sink the paper. The use cases are concrete and the code is out. The approach is plausible for a class of debugging tasks. I'd send it to peer review, with a request that the authors validate the summary assumption and tighten the pruning evaluation. For a visualization venue, the contribution is sufficient.\n\nThis is for readers who care about interpretability tooling for CNNs and related models; it's not a conceptual breakthrough but it's a useful, honest piece of systems work.","headline":"A credible open-source visualization tool for activation-channel analysis, but its core summary assumption needs validation before the headline use cases can be trusted.","tokens_in":21994,"tokens_out":3351,"would_cite":false,"duration_ms":30651,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ChannelExplorer claims that summarizing every activation channel into a scalar and showing those summaries in three coordinated views lets experts find class confusion, mislabeled images, and channel contributions in any image-based…","keywords":["deep neural networks","activation channels","class separability","visual analytics","explainable AI","Jaccard similarity","dimensionality reduction","model debugging"],"falsifier":"Take two classes whose images have identical per-channel intensity histograms but features in different spatial positions; if a classifier separates them while ChannelExplorer's three views show complete overlap, the intensity-only summary has lost the signal the tool claims to preserve.","tokens_in":20880,"feed_emoji":"🧠","tokens_out":5323,"duration_ms":50284,"temperature":0.7,"pith_summary":"This paper introduces ChannelExplorer, a visual analytics tool that turns the activation channels of any image-based neural network layer into compact scalar summaries and displays them in three coordinated views: a scatterplot embedding, a Jaccard similarity matrix, and a channel heatmap. The aim is to let machine-learning experts see, layer by layer, which classes the model confuses, why it confuses them, and which activation channels carry class-distinctive features. The authors argue this works across CNNs, GANs, ResNet, and Stable Diffusion, and demonstrate it by deriving ImageNet class hierarchies, catching mislabeled images, pruning channels without hurting accuracy, and locating generated images in a diffusion model's latent space. A sympathetic reader would take the paper's central claim to be that activation-channel summaries are a sufficient substrate for practical model and data debugging.","feed_headline":"Three views expose where neural nets get confused","feed_subtitle":"ChannelExplorer maps activation channels to scatterplots, similarity matrices, and heatmaps for class-by-class debugging.","key_machinery":"The load-bearing device is the activation-channel summarization function $S(I)$, which maps a channel image $I \\in \\mathbb{R}^{w \\times h}$ to a scalar; the default sums thresholded absolute intensities with the threshold from Otsu's method. This collapses thousands of spatial activations into one number per channel per image. The resulting vectors feed the Jaccard similarity coefficient $J^l_{ij} = |S^l_i \\cap S^l_j| / |S^l_i \\cup S^l_j|$ over the top-activated channel sets, the dimensionality-reduction embedding, and the channel-ordering metric $\\zeta^l_i$, the sum of class-pairwise Euclidean distances between channel summaries. The assumption that intensity alone encodes feature presence is what lets all three views work.","core_discovery":"For an image-based layer, the paper proposes treating each activation channel as a feature detector whose presence can be summarized by a single number computed from pixel intensity alone. Four summarization functions are offered, with a thresholded sum chosen as default. Those summaries feed a scatterplot embedding with cluster hulls, a Jaccard similarity matrix over the top activated channels per image, and a heatmap of channel summaries ordered by class-pairwise distance. The paper claims that reading these three views together reveals inter-class and intra-class confusion, automatically generates per-layer class hierarchies, exposes mislabeled inputs, identifies channels that contribute little to the task, and locates where outputs sit in latent space; the four use cases and the expert evaluation are offered as evidence.","pith_inferences":["Beyond the paper's examples, a controlled comparison against global average pooling would test whether spatial layout truly adds nothing: if the same views reproduce the findings with location discarded entirely, the intensity-only assumption is strongly supported.","The method's logic extends naturally to transformer patches if each patch embedding is treated as a pseudo-channel, but the paper's own limitation section notes attention layers are currently unsupported, so that extension is an inference rather than a claim.","The pruning result suggests a cheap data-driven channel-selection heuristic: order channels by class-pairwise distance and remove the low end, which could be evaluated on models beyond InceptionV3 without retraining.","Because the summaries ignore location, two classes differing only by the spatial arrangement of identical features would appear fused; probing that boundary directly would clarify how far class separability can be judged from intensity alone."],"forward_implications":["If the summaries are faithful, a user can prune the bottom channels in the heatmap's ordering: the paper reports removing 74% of channels in a layer without changing mAP and gaining 12.1% inference-time improvement on CPU.","The class-confusion hierarchy can be turned into model changes: retraining the final layer on new super-classes and subclasses raised the ImageNet classifier from 1000 to 1002 classes at unchanged accuracy.","Scatterplot sub-clusters that persist across layers are a practical signal for mislabeled data, as shown by tiger images hiding in the tiger-cat class.","For generative models, distances among activation summaries in a chosen block can position a new output relative to a set of prompts, giving a practical quality signal."],"supporting_citations":[{"why":"Establishes that activation channels align with interpretable visual features, the premise ChannelExplorer's channel summaries build on.","marker":"[7]"},{"why":"Pioneers dimensionality-reduction projection of activations in a 2D atlas, the conceptual basis for the Scatterplot View.","marker":"[13]"},{"why":"Shows class hierarchies derived from confusion can regularize CNNs; ChannelExplorer generalizes this to per-layer pseudo-classes.","marker":"[4]"},{"why":"Supplies Otsu's thresholding used by the default sum-of-threshold summarization function.","marker":"[64]"},{"why":"Provides X-means clustering used to draw cluster hulls around embedded activation summaries.","marker":"[65]"},{"why":"Defines the Jaccard coefficient that underpins the Activation Jaccard Similarity View.","marker":"[37]"},{"why":"Provides t-SNE, one of the dimensionality-reduction methods compared for the scatterplot.","marker":"[55]"},{"why":"Provides UMAP, the default dimensionality-reduction method for the scatterplot.","marker":"[57]"}],"fun_headline_variants":["Three channel views reveal class confusion in one glance","Activation maps: scatterplots, Jaccard, heatmaps debug models","Find mislabeled data and class hierarchy with ChannelExplorer","See neural confusion via activation channel summaries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the claim that a single scalar per channel, computed from activation intensity alone, preserves enough information to judge class separability regardless of where high activations sit in the image.","fun_headline_variants_meta":{"raw":{"variants":["Three channel views reveal class confusion in one glance","Activation maps: scatterplots, Jaccard, heatmaps debug models","Find mislabeled data and class hierarchy with ChannelExplorer","See neural confusion via activation channel summaries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000643,"raw_usage":{"total_tokens":2918,"prompt_tokens":870,"completion_tokens":2048,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":1983}},"tokens_in":486,"tokens_out":2048,"duration_ms":15195,"temperature":1.0,"reasoning_tokens":1983,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:46:36.391333+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take two classes whose images have identical per-channel intensity histograms but features in different spatial positions; if a classifier separates them while ChannelExplorer's three views show complete overlap, the intensity-only summary has lost the signal the tool claims to preserve.","supporting_citations":[{"cited_title":"Pelleg and A","cited_arxiv_id":null,"evidence_quote":"Provides X-means clustering used to draw cluster hulls around embedded activation summaries."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides t-SNE, one of the dimensionality-reduction methods compared for the scatterplot."}],"review_version":1}