{"id":"8c00b803-6352-4ada-ad10-67d8ae3893c3","arxiv_id":"2412.09521","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A pathology-specialized LVLM with mixed-task feature enhancement and attention-guided detail completion reports higher accuracy than existing LVLMs across many diagnostic tasks.","lead":"OmniPath is a new vision-language model for pathology that combines mixed-task training with attention-guided zooming to analyze whole-slide images. The authors report it outperforms prior pathology LVLMs on cancer diagnosis, detection, and segmentation tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The strongest claim rests on unverified generalization: no patient/slide-level train–test split is stated, and some public datasets are split from the same pool, so repeated patches from one slide could memorize rather than generalize.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the evaluations do not state whether training and test sets are split at the patient or whole-slide level. Since the paper's headline contribution is empirical—an LVLM that outperforms existing methods on many pathology tasks—the validity of that comparison depends entirely on the test sets being unseen at the slide/patient level. The manuscript provides no such statement, and Appendix F even describes sampling training and testing subsets from the same public dataset (TCGA-Uniform-Tumor), which raises the risk of patch-level duplication. This is not an accusation of intentional data leakage; it is an unverified methodological precondition. The concern is concrete and testable: a split manifest plus a patient-level re-evaluation would settle it. The paper does have independent support that partially mitigates the general concern: zero-shot results on external datasets (CCRCC, MHIST, NCT-CRC, PANDA, DHMC, CAMELYON17) in Table 3, and comparisons with closed-source LVLMs in Appendix E, both suggest the model has some generalization ability. However, those results are not the basis of the strongest claim; the strongest claim is the across-the-board superiority on the internal diagnostic tasks, and that claim is exactly what the missing split policy puts at risk. Secondary weaknesses—no error bars, missing SOTA baselines, and an appendix copy-paste error—are real but less decisive. Because the reader already issued a CONDITIONAL verdict conditioned on this issue, our stress-test does not change the verdict; it sharpens the condition: the authors should release the split policy and rerun or confirm the internal-task numbers under a patient/slide-level split, or the across-the-board performance claim should not be accepted as evidence of generalization.","tokens_in":23012,"tokens_out":3782,"duration_ms":39385,"concrete_test":"Request a split manifest and run a single decisive check: obtain slide/patient identifiers for all training and test samples (self-collected and public), compute exact or near-duplicate overlap at the WSI level, and then evaluate OmniPath under a strict patient-level split where no patient contributes slides to both training and test. If accuracies on tasks like HCC-S, ICC-G, and LUNG-S drop materially (e.g., by more than 10 points) under this split, the reported numbers are contaminated by leakage. If they remain within a few points, the generalization concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 5.2 is that \"OmniPath achieves the best performance across all patch-level and slide-level pathological diagnosis tasks.\" For that claim to be about generalization, training and test sets must be disjoint at the patient or whole-slide level. The paper never states such a split. Appendix F describes self-collected WSI datasets from multiple institutions and says that for TCGA-Uniform-Tumor, \"stratified sampling to extract a subset for training and testing\" was used, without clarifying the split unit. Patch-level tasks tile WSIs; if patches from the same slide or patient appear in both training and test, reported accuracies such as HCC-S 97.09, slide-level HCC-S 98.40, ICC-G 99.08, and LUNG-S 98.72 could reflect memorization of scanner/stain/patient-specific features rather than learned diagnostic features. Baselines are fine-tuned on the same data, so leakage would affect them too, but OmniPath's PGFC mechanism additionally selects high-attention high-resolution patches at inference; if those exact high-resolution patches were seen in training, the gain is inflated. Table 6 makes the stakes concrete: removing PGFC drops HCC-S from 98.40 to 20.14, a 78-point jump attributed to adding top-attention patches. Such a jump is plausible only if the selected patches carry the true discriminative signal; without a verified patient/slide-level split, it cannot be distinguished from test-set contamination. Since no split policy, error bars, code, or data release is provided, the central comparison is not yet verifiable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"OmniPath is a pathology-specialized large vision-language model built on LLaVA-1.5, with the vision encoder replaced by SigLIP and augmented by UNI, plus a mask encoder/decoder for segmentation. The paper introduces two strategies: MGFE (mixed task-guided feature enhancement), which trains on detection/segmentation instructions to focus on lesion-level details, and PGFC (prompt-guided detail feature completion), which selects the top-S highest-attention patches from a WSI thumbnail and adds them as high-resolution inputs. The model is trained on a 490K-sample multi-task dataset covering 21 organs. Experiments compare OmniPath against LLaVA-1.5, LLaVA-Med, PA-LLaVA, Quilt-LLaVA (and an author-fine-tuned version) on patch-level and slide-level diagnostics, zero-shot classification, detection, and segmentation, plus ablations of PGFC and MGFE components. The authors report OmniPath as achieving the best accuracy across all in-house diagnostic tasks.","tokens_in":23321,"tokens_out":6669,"duration_ms":54751,"significance":"If the reported results are taken at face value, OmniPath would be a considerable step toward an all-in-one pathology LVLM that handles classification, detection, segmentation, and visual QA at high accuracy, and the PGFC mechanism is an interesting and potentially efficient way to inject WSI-level detail without a computationally heavy patch enumeration. The paper also contributes a large curated instruction-following dataset and an honest limitations section. However, the evaluation as reported is not yet sufficient to establish generalization: the absence of a stated patient/slide-level train-test split, the lack of uncertainty estimates and test-set sizes, and the exclusion of published WSI-capable baselines leave room for alternative explanations of the large accuracy numbers. The central mechanism is plausible and not mathematically circular, but its measured effect needs a more rigorous experimental protocol.","major_comments":[{"comment":"The manuscript never states the unit of the train/test split (patient, whole slide, patch, or random tile) for any of the internal diagnostic datasets. For patch-level tasks that tile WSIs, if patches from the same slide appear in both training and test sets, the reported accuracies (e.g., HCC-S 97.09% in Table 1) could reflect memorization of slide-specific artifacts rather than learned diagnostic generalization. Appendix F's description of TCGA-Uniform-Tumor 'stratified sampling to extract a subset for training and testing' does not clarify the split unit. Please specify the split policy for every dataset, and provide at least one internal validation with patient- or slide-disjoint splits to support the central claim.","section":"Section 5.2 and Appendix F"},{"comment":"No error bars, confidence intervals, sample sizes, or significance tests are reported for any accuracy, F1-score, or Dice value. Without these, it is impossible to determine whether OmniPath's advantages over Quilt-LLaVA (FT) and other baselines are statistically reliable, particularly on tasks where the gaps are modest (e.g., LUNG-G 71.98 vs. 68.01 in Table 1; DHMC 34.62 vs. 28.48 in Table 3). Report test-set sizes per task and include bootstrap or repeated-run intervals.","section":"Tables 1-5 and Appendices D-E"},{"comment":"The PGFC ablation exhibits an extraordinary sensitivity to attention-guided patch selection, with HCC-S rising from 20.14% without PGFC to 98.40% with PGFC, and ICC-G from 66.37% to 99.08%. Such a magnitude is difficult to attribute to the mechanism alone unless the selected high-resolution patches contain the true discriminative signal; without a verified patient/slide-level split, this result is also consistent with test-set contamination of the selected patches. The random-selection control (HCC-S 79.46%) helps, but it does not rule out leakage through attention-based selection that correlates with the test label. Please report the overlap statistics between selected patches and training patches and rerun this ablation on slide-disjoint splits.","section":"Table 6 ablation"},{"comment":"The manuscript discusses PathAlign, PRISM, PathAsst, and PathMMU as existing pathology LVLMs but never evaluates them. The claim in Section 5.2 that 'OmniPath achieves the best performance across all patch-level and slide-level pathological diagnosis tasks' is therefore only validated against LLaVA-derived baselines, one of which (Quilt-LLaVA) was fine-tuned by the authors, plus a small subset of closed-source models in Appendix E. Adding at least one established WSI-capable system (e.g., PathAlign or PRISM) to the comparisons, even on a limited set of tasks, would materially strengthen the claim.","section":"Section 2 and Section 5.2"},{"comment":"The PGFC algorithm states that 'we remove elements corresponding to image background regions from Ψ(N+M),ev' before selecting top-S patches, but no threshold or criterion for defining background is given. This is a free parameter that affects which patches are selected and thus the reported results; specify the detection rule (e.g., attention percentile, Otsu, or a fixed threshold) and the exact values used.","section":"Section 4.2"}],"minor_comments":[{"comment":"The training description omits several standard hyperparameters (warmup steps, weight decay, per-GPU batch size, and total trainable parameter count); adding these would improve reproducibility.","section":"Section 5.1"},{"comment":"The text says attention values are extracted from 'the input layer of M' while the figure caption says 'the first layer of the LLM'; this discrepancy should be reconciled.","section":"Section 3.2 and Figure 2"},{"comment":"The NCT-CRC dataset is referenced to Macenko et al., which is a color-normalization paper rather than the source of the NCT-CRC-HE-100K dataset; the citation should be corrected to the proper Kather et al. reference.","section":"Table 3 and reference [46]"},{"comment":"Table 13 reports 108,806 samples with 'unsure' organ origin, over one fifth of the dataset; clarify how these samples are used and whether they affect organ-classification results.","section":"Appendix F"},{"comment":"There is a typo: 'In the The heatmaps generated by LLaVA-Med...' should read 'The heatmaps generated by...'.","section":"Appendix A"},{"comment":"The closed-source comparison reports only a subset of tasks; state the number of images per task and the API parameters (e.g., temperature, max tokens) used for evaluation.","section":"Appendix E"},{"comment":"The claim that CCRCC, MHIST, NCT-CRC, PANDA, DHMC, and CAMELYON17 were 'not included in the training set' is plausible, but Appendix F shows that TCGA-CRC-DX (used for MSI training) and NCT-CRC are both colorectal-cancer datasets; please confirm no patch-level overlap between training and zero-shot sets.","section":"Section 5.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically interesting and the authors have been transparent about limitations. My main concern is that the evaluation protocol does not yet back the strong 'best across all tasks' claim: the split unit is unspecified, and the PGFC ablation shows an implausibly large effect. These are fixable with additional experiments and reporting, so I recommend major revision rather than rejection. I also note that several of the authors' prior works are cited for the base tasks (refs [1]-[3]), which is appropriate but should not be the only support for key design choices."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on the OmniPath paper. It's a genuine systems contribution: a pathology LVLM that unifies classification, detection, segmentation, and VQA, with two sensible mechanisms – MGFE (multi-scale feature fusion with an extra pathology encoder plus mask decoder) and PGFC (using the model's own attention to fetch high-res patches from a WSI thumbnail). The training corpus is large and multi-institutional, and the zero-shot results on PANDA and CAMELYON17 give some external signal. To their credit, the authors include an honest appendix showing that a vanilla YOLO/nnU-Net beats them on average in detection and segmentation, and they list real limitations in the conclusion.\n\nThe soft spots are mostly about evaluation hygiene, not the method. The headline claim – best across all patch- and slide-level tasks – is not yet supportable. There are no error bars or significance tests, and the paper never states whether train/test splits are at the patient or slide level. For patch-level tasks that tile WSIs, same-slide patches in both training and test would inflate accuracy, and the PGFC ablation makes this acute: HCC-S jumps from 20.14 to 98.40 when top-attention patches are added. That could be legit – high-res detail really may be necessary – but it could also be memorization. The appendix says \"stratified sampling\" for TCGA-Uniform-Tumor without specifying the split unit, and several test sets come from the same institutions as training. That's not proof of leakage, but it is a missing sentence that decides whether the numbers mean anything.\n\nA second issue is the baseline set. PathAlign, PRISM, and PathAsst are named but not benchmarked; the strongest comparison is Quilt-LLaVA fine-tuned by the authors. That's not nothing, but it leaves the \"best across all tasks\" claim hanging. There's also a copy-paste error in the appendix (segmentation prompts identical to detection prompts) and a few typos like \"Benig\" – minor, but sloppy for a paper of this scope.\n\nBottom line: the method is plausible, the engineering is real, and the evaluations are broad enough to deserve referee time. But the authors need to specify the split policy, add error bars or at least test-set sizes, benchmark at least one slide-level pathology LVLM, and release code/data. I'd send it to review with that expectation.\n\nWho should read it: anyone building pathology LVLMs or evaluating WSI models. I'd bring it to reading group for the PGFC idea and for the cautionary tale about ablation design.","headline":"A plausible systems paper for pathology LVLMs whose headline accuracy claims currently rest on unverified split hygiene and a thin baseline set.","tokens_in":23882,"tokens_out":2747,"would_cite":false,"duration_ms":26526,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OmniPath, a pathology-specialized vision-language model, claims to outperform every comparison model on all tested diagnostic tasks by steering attention to lesion details and completing slide features with prompt-chosen high-resolution…","keywords":["pathology","large vision-language model","whole slide image","feature extraction","multi-scale features","cancer diagnosis","detection and segmentation","visual question answering"],"falsifier":"Request the split policy and rerun the patch-level and slide-level evaluations with a strict patient-level split of the same data; if HCC-S, ICC-G, or LUNG-S accuracies drop dramatically under that split, the reported figures overstate generalization. A complementary check is to run OmniPath on an external WSI cohort from an institution absent from the training list and compare its accuracy with the reported numbers.","tokens_in":22820,"feed_emoji":"🔬","tokens_out":7614,"duration_ms":115166,"temperature":0.7,"pith_summary":"This paper is trying to establish that a single vision-language model can cover the full range of pathology diagnostic work, including cancer subtyping and grading, organ and tissue recognition, invasion and metastasis identification, detection, segmentation, and visual question answering, at accuracies that beat existing pathology LVLMs on essentially every task tested, often surpassing 90%. The proposed fixes target a diagnosed weakness: existing LVLMs concentrate attention on a few key tokens that coarsely summarize the whole image and often sit outside the lesion, losing the local details that diagnosis depends on. OmniPath addresses this with two strategies, mixed task-guided feature enhancement and prompt-guided detail feature completion, trained on roughly 490K samples spanning 21 organs. If the claims hold, a single interactive model could handle tasks that previously required separate specialized systems, while keeping inference fast because only a handful of high-resolution patches are added per slide.","feed_headline":"Pathology vision-language model beats baselines on every tested task","feed_subtitle":"One model now unifies cancer subtyping, grading, detection, and segmentation in a single interactive system.","key_machinery":"Mixed task-guided feature enhancement (MGFE) combines hierarchical instruction data at tissue, structure, and cellular levels with three architecture changes: an auxiliary UNI vision encoder pretrained with DINOv2 for fine-grained pathology features, a multi-scale feature fusion that tiles high-resolution inputs and reassembles features so richer detail reaches the LLM without adding image tokens, and a mask encoder and decoder with a new <mask> token so segmentation output is generated directly. Prompt-guided detail feature completion (PGFC) reads the WSI thumbnail, takes the top-S image tokens by attention to locate key regions, extracts high-resolution patches from those regions of the original slide, pools them, and feeds them back into the LLM along with textual position descriptions, completing missing detail without slowing inference. The motivating decision-dependent image token analysis extracts the attention of the final input token over all image tokens and shows that competitors' key tokens aggregate global image semantics while OmniPath's key tokens fall inside annotated lesions.","core_discovery":"On its own terms, the paper demonstrates that OmniPath achieves the best performance across all patch-level and slide-level pathological diagnosis tasks evaluated, including cancer subtyping and grading (97.09% on hepatocellular carcinoma subtyping at patch level and 98.40% at slide level), identification of microvascular and neural invasion, pan-cancer classification across 32 types, tumor-infiltrating lymphocyte recognition, and slide-level prognosis prediction. It is also the first model in this comparison that can perform referring detection and segmentation, and it reports detecting cancer cell nuclei inside blood vessels, a reasoning task that requires relating vessels, nuclei, and cancer concepts. The paper attributes the gains to fixing a measured defect: attention analysis shows existing LVLMs channel information through a few image tokens that aggregate global semantics but miss lesion-level detail, whereas in OmniPath the high-attention tokens concentrate inside pathologist-annotated cancer regions.","pith_inferences":["The PGFC selection rule is driven by attention over the thumbnail, a mechanism that could be tested on other gigapixel-imagery domains, such as cytology or satellite imaging, where input resolution likewise exceeds model limits.","Because the model already emits bounding boxes and masks through dialogue, the same format could be chained into multi-step diagnostic workflows, such as detecting an invasive region and then grading it, without retraining.","The gap between OmniPath's zero-shot accuracy (for example, 79.15% on PANDA and 59.33% on CAMELYON17) and its trained accuracy suggests that broadening the training distribution toward external cohorts, not architecture alone, is the next lever for clinical deployment.","The reported slide-level accuracies in the 90s rest on the unstated premise that slides or patients were separated between training and test sets, so an independent audit of the split policy is the first check a clinical adopter should perform."],"forward_implications":["If the results hold, a single interactive LVLM replaces pipelines of separate classification, detection, and segmentation models for common pathology tasks, with diagnosis presented as dialogue rather than a batch output.","The PGFC ablation (average slide-level accuracy 62.81% without it versus 83.91% with it, and 71.17% with randomly chosen patches) implies that where a whole-slide model looks, not just how much it sees, drives slide-level diagnostic accuracy.","The MGFE ablation shows that the pathology-pretrained auxiliary encoder contributes more than multi-scale fusion for patch-level diagnosis, while multi-scale fusion matters more for detecting variable-size tissue structures, and both are needed for the full result.","Attention analysis suggests the model learns to focus inside pathologist-annotated lesion regions, a property that could make its outputs more auditable than a black-box classifier.","The paper's own comparison with specialized vision models reports OmniPath trailing on average for detection (68.05 versus 70.08 F1 against YOLO11) and slightly ahead for segmentation (77.80 versus 76.95 Dice against nnU-Net V2), so the unified model is not yet a wholesale replacement for dedicated detectors."],"supporting_citations":[{"why":"UNI foundation model; supplies the auxiliary vision encoder whose DINOv2-style features carry the fine-grained pathology detail in MGFE.","marker":"[5]"},{"why":"Quilt-LLaVA; the main pathology LVLM baseline, fine-tuned with the same training data for fair comparison and analyzed for attention patterns.","marker":"[19]"},{"why":"PA-LLaVA; a strong prior pathology LVLM baseline that introduced a scale-invariant connector and is compared against on diagnostic tasks.","marker":"[22]"},{"why":"LLaVA-1.5; the base architecture OmniPath builds on and the source of the base vision-language training paradigm for all compared LVLMs.","marker":"[35]"},{"why":"SigLIP; the replacement primary vision encoder (ViT-SO 384px) that, together with UNI, forms OmniPath's two-encoder vision stack.","marker":"[41]"},{"why":"DINOv2; the self-supervised training framework that gives UNI its fine-grained local features, central to the MGFE design.","marker":"[40]"},{"why":"SAM; supplies the mask decoder architecture used with the LVLM's own vision encoder for segmentation output.","marker":"[43]"},{"why":"PANDA; one of the zero-shot slide-level benchmarks (79.15% accuracy) used to argue clinical generalization.","marker":"[47]"},{"why":"CAMELYON17; zero-shot slide-level benchmark where OmniPath reports 59.33%, far above the baselines.","marker":"[49]"}],"fun_headline_variants":["OmniPath: one pathology AI for subtyping, grading, detection, segmentation","First pathology model to detect cancer cells in blood vessels","Multiscale feature extraction gives pathology AI an edge in speed and accuracy","Unified pathology AI outperforms baselines on every task tested","Pathology AI sees lesion details that other models overlook"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluations assume that slides or patients were split between training and test sets, but the paper never states this; if patches from the same slides that trained the model were also used to test it, the high accuracies could reflect memorization rather than diagnostic skill.","fun_headline_variants_meta":{"raw":{"variants":["OmniPath: one pathology AI for subtyping, grading, detection, segmentation","First pathology model to detect cancer cells in blood vessels","Multiscale feature extraction gives pathology AI an edge in speed and accuracy","Unified pathology AI outperforms baselines on every task tested","Pathology AI sees lesion details that other models overlook"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000638,"raw_usage":{"total_tokens":2910,"prompt_tokens":885,"completion_tokens":2025,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":1936}},"tokens_in":501,"tokens_out":2025,"duration_ms":14237,"temperature":1.0,"reasoning_tokens":1936,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:57:37.192696+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Request the split policy and rerun the patch-level and slide-level evaluations with a strict patient-level split of the same data; if HCC-S, ICC-G, or LUNG-S accuracies drop dramatically under that split, the reported figures overstate generalization. A complementary check is to run OmniPath on an external WSI cohort from an institution absent from the training list and compare its accuracy with the reported numbers.","supporting_citations":[{"cited_title":"Towards a general-purpose foundation model for computational pathology,","cited_arxiv_id":null,"evidence_quote":"UNI foundation model; supplies the auxiliary vision encoder whose DINOv2-style features carry the fine-grained pathology detail in MGFE."},{"cited_title":"Sigmoid loss for language image pre- training,","cited_arxiv_id":null,"evidence_quote":"SigLIP; the replacement primary vision encoder (ViT-SO 384px) that, together with UNI, forms OmniPath's two-encoder vision stack."},{"cited_title":"Segment anything,","cited_arxiv_id":null,"evidence_quote":"SAM; supplies the mask decoder architecture used with the LVLM's own vision encoder for segmentation output."},{"cited_title":"Artificial intelligence for diagnosis and gleason grading of prostate cancer: The panda challenge,","cited_arxiv_id":null,"evidence_quote":"PANDA; one of the zero-shot slide-level benchmarks (79.15% accuracy) used to argue clinical generalization."},{"cited_title":"1399 h&e-stained sentinel lymph node sections of breast cancer patients: The camelyon dataset,","cited_arxiv_id":null,"evidence_quote":"CAMELYON17; zero-shot slide-level benchmark where OmniPath reports 59.33%, far above the baselines."}],"review_version":1}