{"id":"83817d14-ba66-4ef8-9d0c-bf3fce6f25ba","arxiv_id":"2501.17822","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Deep sparse and binary Fisher vectors usually outperform pooling, Deep Sets, memory networks, and focal attention for single-vector WSI retrieval on four TCGA sites.","lead":"This paper compares several ways to compress thousands of patch features from a whole-slide pathology image into one searchable vector. On four TCGA cancer datasets, Fisher-vector aggregation, especially sparse and binary versions, usually retrieves the most relevant slides and is fastest.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing standard deviations undermine the 'best overall' ranking; the paper promises 5-fold variance but Tables 1-4 report only point estimates, and winning margins are often 1-2%.","rationale":"The reader's verdict is CONDITIONAL, and my analysis supports keeping that verdict. The reader identified unequal supervision as the weakest assumption; I agree that the unspecified training protocols are a legitimate concern. However, I found the more immediately decisive issue to be the complete absence of the promised standard deviations. The claim is a ranking claim, and the observed differences are small enough that they could easily be within cross-validation noise. This is not a speculative hidden assumption; it is a direct contradiction between Section 3.2, which promises standard deviations, and the results tables, which omit them. This gap alone is sufficient to prevent the central claim from being authoritative. The paper does have strengths: a clear experimental setup across four TCGA cohorts, a reasonable set of baselines, and an ablation study for the regularization parameter alpha. But without variance or significance testing, the headline recommendation is not robustly supported. The test I propose is minimal and fully within the authors' capability: report the held-out fold results and run a paired comparison. If the differences survive that check, the claim would be substantially strengthened. Until then, the verdict should remain CONDITIONAL, and I would not move it to ACCEPT or REJECT based on this pass.","tokens_in":9582,"tokens_out":4545,"duration_ms":49023,"concrete_test":"Report the per-fold accuracy, Macro F1, and weighted F1 for Tables 1-4. For each dataset, compute the difference between the best-performing Fisher-vector variant and the best non-Fisher baseline, and apply a paired test across the five folds (e.g., paired t-test or Wilcoxon signed-rank). If the difference is not significant (p>0.05) or falls within one standard deviation of the folds, the statement that deep Fisher Vectors 'achieve the best overall performance' is not established by these data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that deep sparse and binary Fisher Vectors 'may achieve the best overall performance' depends on small accuracy differences across methods. Section 3.2 explicitly states: 'We compute the average and standard deviation from the five splits and use these as performance indicators.' However, Tables 1-4 report only averages, with no standard deviations, no per-fold values, and no significance tests. The observed gaps are often tiny: on Bladder, Deep Sparse Fisher Vector accuracy is 0.859 vs. Fisher Vector 0.857 and Deep Sets (Mean) 0.846; on Breast, Deep Binary Fisher Vector is 0.695 vs. Fisher Vector 0.692; on Colon, Deep Sets (Sum) actually beats all Fisher variants (0.843 vs. 0.825 for Fisher Vector). Without variance estimates, it is impossible to determine whether any of these differences exceed fold-to-fold noise. The ranking may therefore reflect random variation rather than a true superiority of the Fisher-vector family. This is a load-bearing gap because the paper's only quantitative support for its headline recommendation is this ranking. The supervision asymmetry noted by the reader is a related concern, but even if all methods received identical label supervision, the missing variance would still leave the central claim unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript benchmarks ten aggregation schemes for converting a set of patch-level deep embeddings into a single whole-slide-image (WSI) vector, evaluating them with k-NN retrieval on four TCGA datasets (bladder, breast, kidney, colon). The methods include simple pooling, Deep Sets, memory networks, focal attention, GMM-based Fisher Vector, and the authors' previously proposed deep sparse and binary Fisher Vector. The paper also compares against Yottixel's median-of-minimum distance as a non-aggregating baseline. The central claim is that the deep Fisher Vector family achieves the best overall retrieval performance in most settings, with sparse and binary variants additionally providing storage and speed benefits. The conclusion suggests that deep sparse and binary Fisher Vectors may be the preferred aggregation choice for WSI search.","tokens_in":9843,"tokens_out":3817,"duration_ms":39475,"significance":"If the ranking were robust, this would be a practically useful comparison for the computational pathology community, which frequently needs to choose an aggregation strategy for WSI-level retrieval and classification. The paper has clear strengths: it covers a broad set of aggregation families, includes a non-aggregation retrieval baseline, uses four public datasets, and reports a timing comparison for the binary representation. It also makes a specific, falsifiable recommendation. However, the empirical support for the headline claim is currently incomplete: the promised variance information is absent, the supervision protocol is asymmetric across methods, and the per-dataset rankings are not fully consistent with the stated conclusion. These issues are fixable but require additional analysis and reporting before the central claim can be accepted.","major_comments":[{"comment":"Section 3.2 states: \"We compute the average and standard deviation from the five splits and use these as performance indicators.\" Yet Tables 1-4 report only point estimates, with no standard deviations and no per-fold values. This is a load-bearing omission because the ranking that drives the paper's conclusion rests on small margins: on Bladder, Deep Sparse Fisher Vector is 0.859 versus 0.857 for Fisher Vector; on Breast, Deep Binary Fisher Vector is 0.695 versus 0.692 for Fisher Vector; on Kidney, Deep Binary Fisher Vector is 0.802 versus 0.799 for Fisher Vector. Without variance estimates or significance tests, the reader cannot determine whether these differences exceed fold-to-fold noise.","section":"Section 3.2 and Tables 1-4"},{"comment":"The supervision is not balanced across methods. Section 2.5 and Fig. 3 show that the deep Fisher Vector VAE is trained with a classification loss using WSI subtype labels, injecting label information into the embedding. Section 3.2 does not specify whether Deep Sets, Memory Network, Focal Attention, and GMM Fisher Vector were trained with equivalent label supervision, the same amount of hyperparameter tuning, or the same number of training epochs. If only the deep Fisher Vector receives subtype labels, the observed advantage could reflect the presence of supervision rather than the aggregation scheme. The revision should document the full training protocol for every baseline, or add an unsupervised/self-supervised condition for the deep Fisher Vector.","section":"Section 2.5 and Section 3.2"},{"comment":"The cross-dataset generalization claim in Section 5 is not fully supported by Tables 1-4. On Colon, Deep Sets (Sum) achieves accuracy 0.843, which is higher than all Fisher Vector variants (Fisher Vector 0.825, Deep Sparse 0.807, Deep Binary 0.809); Deep Sets (Max) at 0.839 also beats the Fisher variants. On Breast, the advantage of Deep Binary Fisher Vector over Fisher Vector is 0.003, and on Bladder the advantage of Deep Sparse Fisher Vector over Fisher Vector is 0.002. The conclusion that the Fisher Vector family has \"robust ability\" and \"versatility\" requires an explicit aggregation of results across datasets and metrics, such as average ranks, plus a significance test; the current narrative relies on \"in most cases\" without quantifying how often or by how much.","section":"Section 4 and Section 5, Tables 1-4"},{"comment":"The value of k in the k-NN search is never specified. Accuracy and F1 scores in a k-NN retrieval evaluation depend heavily on k, and without this number the results in Tables 1-4 cannot be reproduced or properly interpreted. Please report the k value used for each table, and preferably show sensitivity to k.","section":"Section 3.2"}],"minor_comments":[{"comment":"The bladder dataset is described as \"The skin data contains 457 cases,\" which appears to be a typo for \"The bladder data.\" Please correct it.","section":"Section 3.1"},{"comment":"The conclusion states that the methods were evaluated \"for different datasets and two different deep models,\" but Section 3.1 and the results only describe DenseNet as the feature extractor. Please either add the second model or correct the sentence.","section":"Section 5"},{"comment":"The ablation text says the study was performed \"on skin and lung datasets,\" but Tables 6 and 7 report results for breast and colon. Please align the text with the actual datasets.","section":"Section 4.1"},{"comment":"The timing comparison in Table 5 does not report hardware, number of runs, or whether the times are averaged; moreover, Yottixel is evaluated only at dimension 3,000 while the Fisher Vector methods are evaluated at 30,000, 3,000, and 300. Please add the experimental conditions so the speed claim is interpretable.","section":"Table 5"},{"comment":"There are several typographical and grammatical issues, e.g., \"a set of pacthes\" in the Introduction, \"the search speed will increased\" in the Introduction, and inconsistent capitalization of \"Colon\" in dataset names. A careful proofreading pass is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The recommended method is the authors' own previously published deep Fisher Vector [10], and the current protocol does not document whether the baselines received equivalent supervision. I would ask the editor to ensure the revision clarifies this point explicitly, since without it the comparison is vulnerable to a supervision confound. The missing variance information is the other main barrier; both are addressable in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nHere's my read of arXiv:2501.17822.\n\nThe useful core is the head-to-head benchmark. It's the first comparison I know of that puts GMM Fisher vector, deep sparse/binary Fisher vector, Deep Sets, memory networks, focal attention, pooling, and Yottixel's median-of-minima on the same four TCGA sites for WSI retrieval. That's a legitimate service to the field, and the search-time table for sparse vs binary vs Yottixel is practically informative. The ablation on the α regularization for the deep Fisher vector variants is also a nice touch.\n\nThe problem is that the paper's main recommendation doesn't survive contact with its own tables. Section 3.2 says averages and standard deviations are computed over five splits, but Tables 1–4 report only point estimates. Winning margins are frequently one or two points: bladder 0.859 vs 0.857, breast 0.695 vs 0.692, kidney 0.802 vs 0.799. On colon, Deep Sets (Sum) beats every Fisher variant (0.843 vs 0.825). Without variance or significance tests, \"in most cases deep Fisher vector variations achieve the best\" is not supported. This is not a side issue; the ranking is the paper's only quantitative evidence.\n\nThe supervision asymmetry compounds it. The deep Fisher vector is trained with subtype labels via a classification loss, while the baselines' training protocols are not described. If the others were trained without labels, the comparison is partly supervised vs unsupervised. That alone could explain much of the gap. The paper needs to state what supervision each baseline received, or run a version where all methods get the same labels.\n\nThere are minor inconsistencies too: the conclusion mentions \"two different deep models\" but the experiments use DenseNet only, and the ablation text says skin/lung while the tables show breast/colon.\n\nI'd give credit for the honest hedge in the conclusion (\"may achieve the best overall performance\") and for citing the authors' own prior method without hiding it. But the citation pattern is fine; the missing variance is the real blocker.\n\nThis paper deserves a serious referee because the benchmark is useful and the flaws are fixable, but it should not be accepted as is. Major revision: add per-fold results or std devs, significance tests, equal-supervision controls, and release code/data.\n\nBest,\n[You]","headline":"Useful benchmark of WSI aggregation schemes, but missing variance and uneven supervision leave the headline ranking unsupported.","tokens_in":10340,"tokens_out":2732,"would_cite":false,"duration_ms":27641,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Deep Fisher vectors win WSI retrieval benchmark","keywords":["Histopathology","Whole Slide Image","WSI","Image Representation","WSI Search","Set Representation Learning","Single-Vector","Aggregation"],"falsifier":"A controlled experiment that trains Deep Sets, memory networks, focal attention, and the deep Fisher vector on identical patch embeddings with the same label supervision and the same hyperparameter tuning budget, and then removes the classification loss from the Fisher vector; if the Fisher vector no longer outperforms the others, the paper's central claim is falsified.","tokens_in":9406,"feed_emoji":"🔬","tokens_out":3268,"duration_ms":33181,"temperature":0.7,"pith_summary":"The paper tries to establish which aggregation scheme should turn a whole slide image, represented as an unordered bag of patch embeddings, into a single fixed-size vector for WSI search. It benchmarks average and max pooling, Deep Sets, memory networks, focal attention, GMM Fisher vector, and deep sparse and binary Fisher vector on four TCGA cancer datasets (bladder, breast, kidney, colon) using k-NN retrieval. Its central claim is that the deep Fisher vector and its sparse and binary variants achieve the best overall accuracy and F1 scores in most settings, while also offering memory and speed advantages. If true, the practical recommendation for building single-vector WSI search indexes is to use the deep Fisher vector, with the binary version for fastest Hamming-distance search.","feed_headline":"Fisher vectors win WSI retrieval benchmark","feed_subtitle":"Across four TCGA cancer types, deep Fisher vector aggregation beats Deep Sets, memory networks, and pooling for search.","key_machinery":"The central object is the Fisher vector, defined as the gradient of a generative model's log-likelihood with respect to its parameters, aggregated over the set of patch embeddings. In the deep version, a VAE is trained with reconstruction, KL-divergence, classification, and gradient-regularization losses; the gradient of the reconstruction loss with respect to VAE parameters is power- and L2-normalized, and the top-M highest-variance dimensions are selected to form a single permutation-invariant WSI vector. Sparse and binary variants are produced by gradient sparsity and quantization regularization controlled by an alpha parameter, which the paper ablates.","core_discovery":"The paper reports that when a WSI is represented as a set of patch embeddings from a pretrained DenseNet, the deep Fisher vector family outperforms other set-aggregation baselines for k-NN WSI retrieval. Across the four datasets, deep sparse and deep binary Fisher vectors rank first or near-first on accuracy and weighted F1, with the GMM Fisher vector also strong; the median-of-minimums non-aggregating baseline is generally beaten. The binary Fisher vector additionally gives the fastest search because it supports CPU-level XOR-based Hamming distance, while the sparse version reduces storage by producing sparse embeddings.","pith_inferences":["The winning method was trained with cancer-subtype labels via a classification loss, while the paper does not document equivalent label supervision for Deep Sets, memory networks, or focal attention; the ranking could partly reflect supervision rather than aggregation quality.","A natural extension is to apply deep Fisher vector aggregation to patch embeddings from foundation models, which the paper notes are expensive but increasingly common; the same aggregation scheme may transfer without retraining.","The binary Fisher vector could be viewed as a learned hashing scheme, and its retrieval quality at lower dimensionality is worth testing against other learned hash codes.","The ablation results suggest the optimal alpha is dataset-dependent, so a practical deployment would need to tune sparsity or binarization strength per cohort rather than assuming one setting works everywhere."],"forward_implications":["If the claim holds, deep Fisher vector aggregation should be the default choice for building single-vector WSI retrieval systems from patch embeddings.","Sparse Fisher vectors reduce storage costs for WSI indexes, and binary Fisher vectors make search dramatically faster through Hamming distance computed with XOR operations.","Non-aggregating patch-to-patch matching methods like the median of minimums are no longer necessary for competitive WSI retrieval, since single-vector aggregation matches or beats them.","The performance advantage is consistent across four different cancer types, suggesting the Fisher vector captures tissue morphology patterns that generalize across primary sites.","The alpha ablation indicates that adding gradient regularization improves embedding quality while imposing sparsity or binarization, pointing to a tunable trade-off between accuracy and efficiency."],"supporting_citations":[{"why":"Introduces the Yottixel median-of-minimums approach, which the paper uses as the non-aggregating baseline for WSI retrieval.","marker":"[3]"},{"why":"Defines the deep sparse and binary Fisher vector method that is the paper's central evaluated approach.","marker":"[10]"},{"why":"Provides the GMM Fisher vector formulation used as a baseline aggregation method.","marker":"[28]"},{"why":"Introduces Deep Sets, the permutation-invariant set representation baseline compared in the benchmark.","marker":"[4]"},{"why":"Introduces memory networks for permutation-invariant set representation, another compared baseline.","marker":"[5]"},{"why":"Introduces focal attention, the attention-based MIL baseline compared in the benchmark.","marker":"[6]"},{"why":"DenseNet is the pretrained feature extractor that produces the patch embeddings aggregated by all methods.","marker":"[25]"},{"why":"Supplies the power and L2 normalization steps used to compute the deep Fisher vector embeddings.","marker":"[16]"}],"fun_headline_variants":["Fisher vectors outmatch Deep Sets, pooling in WSI search","Binary Fisher vectors enable fast, accurate WSI retrieval","Deep Fisher aggregation leads WSI search on TCGA cancers","Sparse and binary Fisher vectors win WSI retrieval race","Fisher vectors beat median-of-minimums for WSI search"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that all aggregation methods were trained and tuned with equivalent supervision and effort, but only the deep Fisher vector is documented as using subtype labels, so if the others received no matching label signal, the ranking could be an artifact of supervision rather than of aggregation quality.","fun_headline_variants_meta":{"raw":{"variants":["Fisher vectors outmatch Deep Sets, pooling in WSI search","Binary Fisher vectors enable fast, accurate WSI retrieval","Deep Fisher aggregation leads WSI search on TCGA cancers","Sparse and binary Fisher vectors win WSI retrieval race","Fisher vectors beat median-of-minimums for WSI search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000274,"raw_usage":{"total_tokens":1639,"prompt_tokens":944,"completion_tokens":695,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":613}},"tokens_in":560,"tokens_out":695,"duration_ms":6897,"temperature":1.0,"reasoning_tokens":613,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T04:31:23.961409+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled experiment that trains Deep Sets, memory networks, focal attention, and the deep Fisher vector on identical patch embeddings with the same label supervision and the same hyperparameter tuning budget, and then removes the classification loss from the Fisher vector; if the Fisher vector no longer outperforms the others, the paper's central claim is falsified.","supporting_citations":[{"cited_title":"Yottixel–an image search engine for large archives of histopathology whole slide images","cited_arxiv_id":null,"evidence_quote":"Introduces the Yottixel median-of-minimums approach, which the paper uses as the non-aggregating baseline for WSI retrieval."},{"cited_title":"Learning bi- nary and sparse permutation-invariant representations for fast and memory effi- cient whole slide image search","cited_arxiv_id":null,"evidence_quote":"Defines the deep sparse and binary Fisher vector method that is the paper's central evaluated approach."},{"cited_title":"Image classification with the fisher vector: Theory and practice","cited_arxiv_id":null,"evidence_quote":"Provides the GMM Fisher vector formulation used as a baseline aggregation method."},{"cited_title":"Learn- ing permutation invariant representations using memory networks","cited_arxiv_id":null,"evidence_quote":"Introduces memory networks for permutation-invariant set representation, another compared baseline."},{"cited_title":"Pay attention with focus: A novel learning scheme for classification of whole slide images","cited_arxiv_id":null,"evidence_quote":"Introduces focal attention, the attention-based MIL baseline compared in the benchmark."},{"cited_title":"Improving the fisher kernel for large-scale image classification","cited_arxiv_id":null,"evidence_quote":"Supplies the power and L2 normalization steps used to compute the deep Fisher vector embeddings."}],"review_version":1}