REVIEW 4 major objections 5 minor 24 references
PathCoT: Chain-of-Thought Prompting for Zero-shot Pathology Visual Reasoning
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Adding simulated pathology experts and a self-check step improves zero-shot visual reasoning on path images by several accuracy points.
desk verdict Solid domain-specific prompt recipe for zero-shot pathology VQA, but the compute budget confound needs an equal-compute control. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is a four-stage prompting pipeline. In Preparation, PathCoT generates a two-part caption (question-agnostic and question-dependent descriptions) and a decision on which of the four pathology experts apply to the question. In Image Analysis, the selected experts—Cellular, Tissue, Organ, and Biomarker—produce domain-knowledge descriptions of the image, which are aggregated into expert knowledge $E$. In Summary & Answer Generation, the image, question, caption, and expert knowledge are fed to the MLLM to produce the CoT answer. In Self-Evaluation, an extra MLLM call receives the CoT answer and a direct answer, and returns the final answer with a rationale, adjudicating between the two when they conflict. The load-bearing component is this self-evaluation step (Eq. 3): when the two candidate answers differ, the model must identify which one is correct.
What would settle it
Run PathCoT on the PathMMU test sets while logging the accuracy of the self-evaluation step only for questions where $A_{\text{CoT}}$ and $A_{\text{dir}}$ differ; if the self-evaluation selects the correct candidate at or below chance on those conflicting cases, the reported gains would not survive.
Extended reading notes
Core claim
The central claim is that zero-shot pathology visual reasoning can be substantially improved by combining two mechanisms that counteract known failure modes of chain-of-thought prompting: supplying domain-specific prior knowledge and re-evaluating the answer. PathCoT avoids fine-tuning entirely. It composes a caption containing question-agnostic and question-dependent descriptions, selects relevant experts from a fixed set of four (Cellular, Tissue, Organ, Biomarker) based on the question, obtains expert knowledge $E$, and generates the chain-of-thought answer $A_{\text{CoT}} = \text{MLLM}(S)$, where $S = \{I, Q, D_{\text{cap}}, E\}$. It also obtains a direct answer $A_{\text{dir}} = \text{MLLM}(I, Q)$. A final self-evaluation call (Eq. 3) receives both candidates and returns the final answer $A$ along with a rationale $R$, selecting the correct candidate when the two disagree. The paper reports that this consistently outperforms all prior zero-shot CoT methods on PathMMU, with the largest gains coming from the expert-analysis stage (a 3.70% tiny-test improvement when added) and the self-evaluation stage (a 2.39% improvement).
Load-bearing premise
The entire gain depends on the self-evaluation call (Eq. 3) being able to pick the correct answer when the chain-of-thought answer and the direct answer disagree; the paper provides no data on how often that adjudication fails.
Editorial extensions
If this is right
- Zero-shot prompting can narrow the gap between general MLLMs and expert-level pathology reasoning without collecting domain-specific fine-tuning data.
- Question-dependent captions add 1.20% tiny-test accuracy over vanilla captions, indicating that telling the model what to look for in the image matters.
- Removing the expert-analysis stage costs 3.70% on the tiny test, the largest single ablation, showing that simulated domain expertise is the main driver of the gain.
- Self-evaluation recovers between 1.61% and 2.39% accuracy, confirming that CoT answers on pathology images are not always better than direct answers and need verification.
- The gains appear consistently across all four PathMMU subsets (PubMed, EduContent, Atlas, PathCLS) and on both tiny and full test sets.
Reading between the lines
- The self-evaluation stage could be made more reliable by ensembling several adjudication calls or training a verifier on conflicting answers; the paper does not test this.
- Because the expert roles are fixed and generic, the same prompting recipe might transfer to other medical imaging domains (e.g., radiology or dermatology) where a small set of sub-specialist viewpoints can be named.
- The two-caption design (question-agnostic plus question-dependent) could be tested as a standalone intervention on general visual QA benchmarks to see whether the gain is specific to pathology or reflects a broader prompting principle.
- Since the method uses LLaVa as the base model, a natural test is whether the gains compound with a stronger MLLM; the paper does not report a scaling curve.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes PathCoT, a zero-shot chain-of-thought prompting method for multimodal pathology visual reasoning. The method generates a two-part image caption (question-agnostic and question-dependent), selects among four predefined pathology experts (Cellular, Tissue, Organ, Biomarker) based on the question, uses the selected experts to produce domain-specific image analyses, then combines the caption and expert knowledge to obtain a CoT-based answer (Eq. 1). A separate direct answer is also generated (Eq. 2), and a final self-evaluation step (Eq. 3) chooses between the CoT-based and direct answers, optionally producing a rationale. Experiments on the PathMMU benchmark compare PathCoT with several CoT-based baselines and report consistent accuracy gains across four subsets; ablation studies examine the contribution of each stage and each expert role.
Significance. If the reported gains are robust, PathCoT is a useful contribution to zero-shot medical visual reasoning because it demonstrates that combining structured expert prompts with a self-evaluation step can improve MLLM accuracy on a challenging expert-level pathology benchmark without fine-tuning. The paper's strengths include a clearly described multi-stage prompting pipeline, ablations for each stage and each expert role, and consistent improvements over all compared baselines on both tiny test and test sets. The method is simple and could be adopted by practitioners. However, the central claim that expert knowledge and self-evaluation drive the gains is not yet isolated from the larger inference budget, and the absence of statistical error bars weakens the quantitative conclusions.
major comments (4)
- [§3.3, Table 1] The comparison against baselines does not control for inference compute. PathCoT makes multiple MLLM calls per question (caption generation, expert selection, per-expert analyses, Eq. 1, Eq. 2, and Eq. 3, with additional calls if GPT-3.5 is used), whereas MLLM-Only makes a single call and the other baselines make one or two calls. The reported accuracy advantage (e.g., 39.20 vs. 34.42 overall tiny test) may therefore reflect additional opportunities to sample the MLLM rather than the proposed prompting design. Please add an equal-compute control, such as a repeated-sampling or self-consistency baseline that uses the same number of MLLM calls and a comparable token budget but without expert roles and self-evaluation (e.g., sampling the direct answer multiple times and taking a majority vote).
- [§3.4, Table 2] The ablation rows also vary the number of MLLM calls: the 'w/o Analysis' row removes the expert-analysis calls, and the 'w/o Self-Evaluation' row removes the Eq. 3 call. The attributed contributions of expert analysis (+3.70/+2.55) and self-evaluation (+2.39/+1.61) are therefore conflated with a reduction in inference budget. Please provide an ablation in which the removed calls are replaced by additional generic calls (e.g., repeated direct-answer sampling) so that the effect of the prompt content is separated from the effect of extra compute.
- [§3.1, Tables 1–3] No error bars, confidence intervals, or repeated-run statistics are reported. Given that the tiny test sets contain only 921 questions overall (and 177–281 per subset), the reported differences, particularly the smaller ablation gaps (e.g., 1.20% caption gain, 1.61% self-evaluation gain on the test set), may be within run-to-run variance. Please report multiple runs with standard deviations, or at least a statistical significance test (e.g., bootstrap or McNemar's test) over the paired questions.
- [§2.5, Eq. (3)] The method's gains depend on the MLLM's self-evaluation being reliable when A_CoT and A_dir disagree, but the paper provides no analysis of this decision process. Please report, either on the full test set or on a sample, how often A_CoT and A_dir disagree, how often Eq. (3) selects the correct candidate in those cases, and how often it selects an incorrect candidate. This would directly substantiate the claimed mitigation of answer divergence and also reveal failure modes that a practitioner should expect.
minor comments (5)
- [§3.1] The text lists the four subsets as PubMed, SocialPath, Atlas, and PathCLS, but Tables 1–3 use 'EduContent' instead of 'SocialPath'. Please align the subset names between the text and the tables.
- [Abstract] In the sentence 'LLMs often underperforms because they lack domain-specific information', the verb should agree with the plural subject; change 'underperforms' to 'underperform'.
- [§2.2] The prompt example writes 'HE' where the later text uses 'H&E'; please use the standard 'H&E' consistently.
- [§3.1] The implementation details are sparse: it would be helpful to state the LLaVA checkpoint used, the number of inference calls per question, the token budget, and whether GPT-3.5 is queried for every question or only for specific stages (e.g., expert selection).
- [General] The paper does not mention code or data release; providing the prompt templates and evaluation scripts would improve reproducibility.
Circularity Check
No significant circularity: PathCoT is an externally benchmarked prompting pipeline with no fitted parameters, no self-citation chain, and no prediction that reduces to its inputs by construction.
full rationale
PathCoT is an empirical prompting paper evaluated on the external PathMMU benchmark. The core equations define a pipeline: A_CoT = MLLM(S) in Eq. (1), A_dir = MLLM(I, Q) in Eq. (2), and R, A = MLLM(I, Q, A_CoT, A_dir) in Eq. (3). No fitted parameter, learned weight, or domain-specific statistic is estimated from the test set; each stage is a prompted inference call. The self-evaluation stage uses the same MLLM to arbitrate between two candidate answers, but this is a design choice, not a derivation in which an output is fed back into the input by definition. The paper contains no self-citation chain and invokes no uniqueness theorem. Concerns about unmatched inference compute between PathCoT and its baselines are a fairness or correctness issue, not circularity, because the claimed gains are empirical and could be tested by an equal-budget control rather than being forced by construction. No circular step is therefore identified, and the circularity score is 0.
Assumptions & free parameters
assumptions (3)
- ad hoc to paper The four expert roles (Cellular, Tissue, Organ, Biomarker) are sufficient to cover the pathology visual analysis space relevant to the benchmark questions.
- domain assumption GPT-3.5 can generate useful expert selection and guidance for pathology image analysis.
- domain assumption The MLLM's self-evaluation, when given both candidate answers, is more reliable than either answer alone.
Cite this review
Pith. "Pith review of PathCoT: Chain-of-Thought Prompting for Zero-shot Pathology Visual Reasoning." pith.science (2026). https://pith.science/paper/4WXTHKUJ
@misc{pith2026250701029,
author = {Pith},
title = {Pith review of: PathCoT: Chain-of-Thought Prompting for Zero-shot Pathology Visual Reasoning},
year = {2026},
howpublished = {\url{https://pith.science/paper/4WXTHKUJ}},
note = {Machine review of arXiv:2507.01029}
}
read the original abstract
With the development of generative artificial intelligence and instruction tuning techniques, multimodal large language models (MLLMs) have made impressive progress on general reasoning tasks. Benefiting from the chain-of-thought (CoT) methodology, MLLMs can solve the visual reasoning problem step-by-step. However, existing MLLMs still face significant challenges when applied to pathology visual reasoning tasks: (1) LLMs often underperforms because they lack domain-specific information, which can lead to model hallucinations. (2) The additional reasoning steps in CoT may introduce errors, leading to the divergence of answers. To address these limitations, we propose PathCoT, a novel zero-shot CoT prompting method which integrates the pathology expert-knowledge into the reasoning process of MLLMs and incorporates self-evaluation to mitigate divergence of answers. Specifically, PathCoT guides the MLLM with prior knowledge to perform as pathology experts, and provides comprehensive analysis of the image with their domain-specific knowledge. By incorporating the experts' knowledge, PathCoT can obtain the answers with CoT reasoning. Furthermore, PathCoT incorporates a self-evaluation step that assesses both the results generated directly by MLLMs and those derived through CoT, finally determining the reliable answer. The experimental results on the PathMMU dataset demonstrate the effectiveness of our method on pathology visual understanding and reasoning.
Figures
Reference graph
Works this paper leans on
-
[1]
Blip-2: Bootstrapping language-imagepre-trainingwithfrozenimageencodersandlargelanguagemodels
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-imagepre-trainingwithfrozenimageencodersandlargelanguagemodels. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023
work page 2023
-
[2]
Visual instruction tuning.Advances in neural information processing systems, 36, 2024
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2024
2024
-
[3]
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning.arXiv preprint arXiv:2305.06500, 2023
arXiv 2023
-
[4]
Minigpt- 4: Enhancing vision-language understanding with advanced large language models
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt- 4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023
arXiv 2023
-
[5]
Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023
arXiv 2023
-
[6]
Llava-ultra: Large chinese language and vision assistant for ultrasound
Xuechen Guo, Wenhao Chai, Shi-Yan Li, and Gaoang Wang. Llava-ultra: Large chinese language and vision assistant for ultrasound. InProceedings of the 32nd ACM International Conference on Multimedia, pages 8845–8854, 2024
work page 2024
-
[7]
Seowoo Lee, Jiwon Youn, Hyungjin Kim, Mansu Kim, and Soon Ho Yoon. Cxr- llava: a multimodal large language model for interpreting chest x-ray images.Eu- ropean Radiology, pages 1–13, 2025
work page 2025
-
[8]
Yuxuan Sun, Chenglu Zhu, Sunyi Zheng, Kai Zhang, Lin Sun, Zhongyi Shui, Yun- long Zhang, Honglin Li, and Lin Yang. Pathasst: A generative foundation ai as- sistant towards artificial general intelligence of pathology. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5034–5042, 2024
work page 2024
Show all 24 references
-
[9]
Quilt-llava: Visual instruction tuning by extracting localized narratives from open-source histopathology videos
Mehmet Saygin Seyfioglu, Wisdom O Ikezogwo, Fatemeh Ghezloo, Ranjay Kr- ishna, and Linda Shapiro. Quilt-llava: Visual instruction tuning by extracting localized narratives from open-source histopathology videos. InProceedings of the IEEE/CVF Conference on Computer Vision and P...
2024
-
[10]
A multimodal generative ai copilot for human pathology.Nature, 634(8033):466–473, 2024
Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Melissa Zhao, Aaron K Chow, Kenji Ikemura, Ahrong Kim, Dimitra Pouli, Ankush Patel, et al. A multimodal generative ai copilot for human pathology.Nature, 634(8033):466–473, 2024
2024
-
[11]
Large language models are zero-shot reasoners.Advances in neural information processing systems, 35:22199–22213, 2022
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners.Advances in neural information processing systems, 35:22199–22213, 2022. 10 JJ Zhou et al
2022
-
[12]
Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022
2022
-
[13]
Automatic chain of thought prompting in large language models.arXiv preprint arXiv:2210.03493, 2022
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models.arXiv preprint arXiv:2210.03493, 2022
2022 arXiv
-
[14]
Multimodal chain-of-thought reasoning in language models.arXiv preprint arXiv:2302.00923, 2023
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models.arXiv preprint arXiv:2302.00923, 2023
2023 arXiv
-
[15]
Good questions help zero-shot image reasoning.arXiv preprint arXiv:2312.01598, 2023
Kaiwen Yang, Tao Shen, Xinmei Tian, Xiubo Geng, Chongyang Tao, Dacheng Tao, and Tianyi Zhou. Good questions help zero-shot image reasoning.arXiv preprint arXiv:2312.01598, 2023
2023 arXiv
-
[16]
Ddcot: Duty- distinct chain-of-thought prompting for multimodal reasoning in language models
Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, and Sibei Yang. Ddcot: Duty- distinct chain-of-thought prompting for multimodal reasoning in language models. Advances in Neural Information Processing Systems, 36:5168–5191, 2023
2023
-
[17]
Cocot: Contrastive chain-of-thought prompting for large multimodal models with multiple image inputs.arXiv preprint arXiv:2401.02582, 2024
DaoanZhang,JunmingYang,HanjiaLyu,ZijianJin,YuanYao,MingkaiChen,and Jiebo Luo. Cocot: Contrastive chain-of-thought prompting for large multimodal models with multiple image inputs.arXiv preprint arXiv:2401.02582, 2024
2024 arXiv
-
[18]
Composi- tional chain-of-thought prompting for large multimodal models
Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Composi- tional chain-of-thought prompting for large multimodal models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14420–14431, 2024
2024
-
[19]
Pathmmu: A massive multimodal expert-level benchmark for understanding and reasoning in pathology
Yuxuan Sun, Hao Wu, Chenglu Zhu, Sunyi Zheng, Qizi Chen, Kai Zhang, Yunlong Zhang, Dan Wan, Xiaoxiao Lan, Mengyue Zheng, et al. Pathmmu: A massive multimodal expert-level benchmark for understanding and reasoning in pathology. InEuropean Conference on Computer Vision, pages 56...
2024
-
[20]
Cares: A comprehensive benchmark of trustworthiness in medical vision language models.Advances in Neural Information Processing Systems, 37:140334–140365, 2025
Peng Xia, Ze Chen, Juanxi Tian, Yangrui Gong, Ruibo Hou, Yue Xu, Zhenbang Wu, Zhiyuan Fan, Yiyang Zhou, Kangyu Zhu, et al. Cares: A comprehensive benchmark of trustworthiness in medical vision language models.Advances in Neural Information Processing Systems, 37:140334–140365, 2025
2025
-
[21]
Mc-cot: A modular collaborative cot framework for zero-shot medical-vqawithllmandmllmintegration.arXiv preprint arXiv:2410.04521,2024
Lai Wei, Wenkai Wang, Xiaoyu Shen, Yu Xie, Zhihao Fan, Xiaojin Zhang, Zhongyu Wei, and Wei Chen. Mc-cot: A modular collaborative cot framework for zero-shot medical-vqawithllmandmllmintegration.arXiv preprint arXiv:2410.04521,2024
-
[22]
A survey on mixture of experts.arXiv preprint arXiv:2407.06204, 2024
Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts.arXiv preprint arXiv:2407.06204, 2024
2024 arXiv
-
[23]
Pytorch: An imperative style, high-performance deep learning library.Advances in neural information processing systems, 32, 2019
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gre- gory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library.Advances in neural information processing systems, 32, 2019
2019
-
[24]
Cantor: Inspiring multimodal chain-of-thought of mllm
Timin Gao, Peixian Chen, Mengdan Zhang, Chaoyou Fu, Yunhang Shen, Yan Zhang, Shengchuan Zhang, Xiawu Zheng, Xing Sun, Liujuan Cao, et al. Cantor: Inspiring multimodal chain-of-thought of mllm. InProceedings of the 32nd ACM International Conference on Multimedia, pages 9096–9105, 2024
2024
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.