{"id":"b0bdad99-9f02-426f-997e-6a8c16b3ed8c","arxiv_id":"2507.01029","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A chain-of-thought prompting method with four pathology experts and a self-evaluation step improves zero-shot accuracy on the PathMMU pathology visual-reasoning benchmark.","lead":"PathCoT is a new way to prompt a multimodal AI model for answering pathology image questions: it first breaks the analysis into four expert roles, then asks the model to choose between its direct answer and its step-by-step answer. The approach improves accuracy on a standard pathology benchmark without any fine-tuning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PathCoT's reported gains are not isolated from its larger inference budget: comparisons and ablations vary the number of MLLM calls, so the contributions of expert-knowledge and self-evaluation remain untested.","rationale":"The reader's weakest assumption was the reliability of the Eq. 3 self-evaluation call. I see the compute confound as the more upstream concern: self-evaluation is only one of several extra calls whose individual effects are measured by removing calls, never by holding the number of calls fixed. The paper does provide useful ablations and consistent subset-level gains, so this is not grounds for rejection; but the causal attribution ('by integrating expert knowledge and self-evaluation') is not established. The reader's rationale already lists the inference-compute mismatch as a condition, so I would keep the same CONDITIONAL verdict and add the equal-compute control to the list of requested resubmission checks.","tokens_in":8527,"tokens_out":14728,"duration_ms":151810,"concrete_test":"Run a compute-matched control on the 921-question tiny test: give LLaVa the same number of inference calls and roughly the same output-token budget as PathCoT per question, but replace the expert-knowledge and self-evaluation prompts with generic prompts (e.g., 'Describe the image' followed by 'Answer the question', repeated with different generic phrasings), aggregating by majority vote. If this generic control reaches or beats PathCoT's 39.20% tiny-test accuracy, the reported advantage is explained by inference budget rather than by expert-knowledge and self-evaluation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that expert-knowledge and self-evaluation drive PathCoT's accuracy gain, but no comparison or ablation controls for inference compute. MLLM-Only and most baselines use one or two model calls, while PathCoT uses at least five: caption generation, expert selection, per-expert analyses, summary answer A_CoT (Eq. 1), direct answer A_dir (Eq. 2), and the self-evaluation call (Eq. 3), with GPT-3.5 possibly adding more. The ablations also remove calls: 'w/o Analysis' (Table 2, row 3) drops all expert-analysis calls, and 'w/o Self-Evaluation' (row 4) drops the Eq. 3 call. The attributed gains (+2.39/+1.61 for self-evaluation, +3.70/+2.55 for expert analysis) therefore conflate prompt content with extra opportunities to sample the MLLM. A generic equal-compute control (same number of calls, same token budget, without expert roles or self-evaluation) is absent. If such a control matches PathCoT, the reported superiority is an artifact of budget rather than of the proposed prompting design.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PathCoT, a zero-shot chain-of-thought prompting method for multimodal pathology visual reasoning. The method generates a two-part image caption (question-agnostic and question-dependent), selects among four predefined pathology experts (Cellular, Tissue, Organ, Biomarker) based on the question, uses the selected experts to produce domain-specific image analyses, then combines the caption and expert knowledge to obtain a CoT-based answer (Eq. 1). A separate direct answer is also generated (Eq. 2), and a final self-evaluation step (Eq. 3) chooses between the CoT-based and direct answers, optionally producing a rationale. Experiments on the PathMMU benchmark compare PathCoT with several CoT-based baselines and report consistent accuracy gains across four subsets; ablation studies examine the contribution of each stage and each expert role.","tokens_in":8770,"tokens_out":2288,"duration_ms":23902,"significance":"If the reported gains are robust, PathCoT is a useful contribution to zero-shot medical visual reasoning because it demonstrates that combining structured expert prompts with a self-evaluation step can improve MLLM accuracy on a challenging expert-level pathology benchmark without fine-tuning. The paper's strengths include a clearly described multi-stage prompting pipeline, ablations for each stage and each expert role, and consistent improvements over all compared baselines on both tiny test and test sets. The method is simple and could be adopted by practitioners. However, the central claim that expert knowledge and self-evaluation drive the gains is not yet isolated from the larger inference budget, and the absence of statistical error bars weakens the quantitative conclusions.","major_comments":[{"comment":"The comparison against baselines does not control for inference compute. PathCoT makes multiple MLLM calls per question (caption generation, expert selection, per-expert analyses, Eq. 1, Eq. 2, and Eq. 3, with additional calls if GPT-3.5 is used), whereas MLLM-Only makes a single call and the other baselines make one or two calls. The reported accuracy advantage (e.g., 39.20 vs. 34.42 overall tiny test) may therefore reflect additional opportunities to sample the MLLM rather than the proposed prompting design. Please add an equal-compute control, such as a repeated-sampling or self-consistency baseline that uses the same number of MLLM calls and a comparable token budget but without expert roles and self-evaluation (e.g., sampling the direct answer multiple times and taking a majority vote).","section":"§3.3, Table 1"},{"comment":"The ablation rows also vary the number of MLLM calls: the 'w/o Analysis' row removes the expert-analysis calls, and the 'w/o Self-Evaluation' row removes the Eq. 3 call. The attributed contributions of expert analysis (+3.70/+2.55) and self-evaluation (+2.39/+1.61) are therefore conflated with a reduction in inference budget. Please provide an ablation in which the removed calls are replaced by additional generic calls (e.g., repeated direct-answer sampling) so that the effect of the prompt content is separated from the effect of extra compute.","section":"§3.4, Table 2"},{"comment":"No error bars, confidence intervals, or repeated-run statistics are reported. Given that the tiny test sets contain only 921 questions overall (and 177–281 per subset), the reported differences, particularly the smaller ablation gaps (e.g., 1.20% caption gain, 1.61% self-evaluation gain on the test set), may be within run-to-run variance. Please report multiple runs with standard deviations, or at least a statistical significance test (e.g., bootstrap or McNemar's test) over the paired questions.","section":"§3.1, Tables 1–3"},{"comment":"The method's gains depend on the MLLM's self-evaluation being reliable when A_CoT and A_dir disagree, but the paper provides no analysis of this decision process. Please report, either on the full test set or on a sample, how often A_CoT and A_dir disagree, how often Eq. (3) selects the correct candidate in those cases, and how often it selects an incorrect candidate. This would directly substantiate the claimed mitigation of answer divergence and also reveal failure modes that a practitioner should expect.","section":"§2.5, Eq. (3)"}],"minor_comments":[{"comment":"The text lists the four subsets as PubMed, SocialPath, Atlas, and PathCLS, but Tables 1–3 use 'EduContent' instead of 'SocialPath'. Please align the subset names between the text and the tables.","section":"§3.1"},{"comment":"In the sentence 'LLMs often underperforms because they lack domain-specific information', the verb should agree with the plural subject; change 'underperforms' to 'underperform'.","section":"Abstract"},{"comment":"The prompt example writes 'HE' where the later text uses 'H&E'; please use the standard 'H&E' consistently.","section":"§2.2"},{"comment":"The implementation details are sparse: it would be helpful to state the LLaVA checkpoint used, the number of inference calls per question, the token budget, and whether GPT-3.5 is queried for every question or only for specific stages (e.g., expert selection).","section":"§3.1"},{"comment":"The paper does not mention code or data release; providing the prompt templates and evaluation scripts would improve reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is a straightforward empirical prompting study. The main concern is not circularity but the uncontrolled inference budget, which is fixable with additional control experiments. The lack of error bars is also fixable. If the authors add an equal-compute baseline and reliability analysis of self-evaluation, the paper would be significantly stronger. The manuscript fits the journal's scope as an application of LLM prompting to medical vision-language tasks."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a well-executed domain adaptation of known prompting tricks. PathCoT swaps general-purpose CoT for four pathology expert roles, adds a two-part caption, and finishes with a self-evaluation step that picks between the CoT answer and a direct answer. On PathMMU, that recipe beats the strongest zero-shot baseline by about 4-5 points, and the gain shows up on every subset and both test splits.\n\nWhat is new here is not any single component—expert-role prompting, captioning, and self-evaluation all exist in earlier CoT work. The contribution is the specific assembly for pathology, and the paper does it carefully. The ablations are thorough: removing any stage hurts, removing any expert hurts, and the two case studies show exactly how the reasoning chain helps and where it could go wrong.\n\nThe soft spots are real but not fatal. The most serious is the inference-compute mismatch. PathCoT makes several extra MLLM calls: caption, expert selection, per-expert analyses, CoT summary, direct answer, and self-evaluation. The baselines make one or two. The ablations that remove analysis or self-evaluation also remove calls, so the attributed gains conflate prompt structure with additional sampling. An equal-compute control—same number of MLLM calls and token budget, without the expert roles or self-evaluation—would settle whether the recipe itself is what helps. It is missing.\n\nSecond, there are no error bars or repeated runs. With differences of 2-5 points, stochastic variation cannot be ruled out. The consistent direction across subsets helps, but a serious referee should ask for variance across seeds.\n\nThird, reproducibility is thinner than it should be: no code and no verbatim prompts. The prompt descriptions are detailed, but exact wording matters in this line of work, and GPT-3.5 is used as an auxiliary LLM without a clear account of where and how.\n\nThe self-evaluation mechanism is itself a bit of a learned bet: it relies on the MLLM to pick the better candidate when the two answers disagree. The ablation says it helps on average, so I do not think this is a flaw, but it is worth probing.\n\nWho this is for: people working on medical VQA or prompting recipes for specialized domains. They will get a solid baseline and a reminder that compute and prompt content are easily confounded in this genre.\n\nMy verdict: this deserves a serious referee. The core result is credible, the ablations are above the norm, and the compute-control issue is addressable with one additional experiment. I would send it to review and ask for variance estimates and an equal-compute control.","headline":"Solid domain-specific prompt recipe for zero-shot pathology VQA, but the compute budget confound needs an equal-compute control.","tokens_in":9271,"tokens_out":3332,"would_cite":false,"duration_ms":30459,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding simulated pathology experts and a self-check step improves zero-shot visual reasoning on path images by several accuracy points.","keywords":["pathology visual reasoning","chain-of-thought prompting","zero-shot reasoning","multimodal large language models","self-evaluation","expert knowledge","PathMMU","visual question answering"],"falsifier":"Run PathCoT on the PathMMU test sets while logging the accuracy of the self-evaluation step only for questions where $A_{\\text{CoT}}$ and $A_{\\text{dir}}$ differ; if the self-evaluation selects the correct candidate at or below chance on those conflicting cases, the reported gains would not survive.","tokens_in":8335,"feed_emoji":"🔬","tokens_out":4726,"duration_ms":39525,"temperature":0.7,"pith_summary":"PathCoT argues that multimodal language models can answer pathology multiple-choice questions far more accurately without any fine-tuning, by structuring their prompt as a chain of thought enriched with domain knowledge. The method first generates a two-part caption of the pathology image, then asks four simulated pathology experts—cellular, tissue, organ, and biomarker—to analyze the image from their specialties, selecting only the experts relevant to the question. The model then produces an answer by reasoning over the caption and expert knowledge, and a final self-evaluation step compares this chain-of-thought answer with a direct answer, selecting the more reliable one. On the PathMMU benchmark, PathCoT reports an overall tiny-test accuracy of 39.20%, compared with 34.42% for the strongest baseline, and improves the large test set from 32.85% to 36.75%.","feed_headline":"PathCoT lifts zero-shot pathology reasoning accuracy to 39.2%","feed_subtitle":"Adding four pathology experts and a self-evaluation step beats prior chain-of-thought prompts on the PathMMU benchmark.","key_machinery":"The machinery is a four-stage prompting pipeline. In Preparation, PathCoT generates a two-part caption (question-agnostic and question-dependent descriptions) and a decision on which of the four pathology experts apply to the question. In Image Analysis, the selected experts—Cellular, Tissue, Organ, and Biomarker—produce domain-knowledge descriptions of the image, which are aggregated into expert knowledge $E$. In Summary & Answer Generation, the image, question, caption, and expert knowledge are fed to the MLLM to produce the CoT answer. In Self-Evaluation, an extra MLLM call receives the CoT answer and a direct answer, and returns the final answer with a rationale, adjudicating between the two when they conflict. The load-bearing component is this self-evaluation step (Eq. 3): when the two candidate answers differ, the model must identify which one is correct.","core_discovery":"The central claim is that zero-shot pathology visual reasoning can be substantially improved by combining two mechanisms that counteract known failure modes of chain-of-thought prompting: supplying domain-specific prior knowledge and re-evaluating the answer. PathCoT avoids fine-tuning entirely. It composes a caption containing question-agnostic and question-dependent descriptions, selects relevant experts from a fixed set of four (Cellular, Tissue, Organ, Biomarker) based on the question, obtains expert knowledge $E$, and generates the chain-of-thought answer $A_{\\text{CoT}} = \\text{MLLM}(S)$, where $S = \\{I, Q, D_{\\text{cap}}, E\\}$. It also obtains a direct answer $A_{\\text{dir}} = \\text{MLLM}(I, Q)$. A final self-evaluation call (Eq. 3) receives both candidates and returns the final answer $A$ along with a rationale $R$, selecting the correct candidate when the two disagree. The paper reports that this consistently outperforms all prior zero-shot CoT methods on PathMMU, with the largest gains coming from the expert-analysis stage (a 3.70% tiny-test improvement when added) and the self-evaluation stage (a 2.39% improvement).","pith_inferences":["The self-evaluation stage could be made more reliable by ensembling several adjudication calls or training a verifier on conflicting answers; the paper does not test this.","Because the expert roles are fixed and generic, the same prompting recipe might transfer to other medical imaging domains (e.g., radiology or dermatology) where a small set of sub-specialist viewpoints can be named.","The two-caption design (question-agnostic plus question-dependent) could be tested as a standalone intervention on general visual QA benchmarks to see whether the gain is specific to pathology or reflects a broader prompting principle.","Since the method uses LLaVa as the base model, a natural test is whether the gains compound with a stronger MLLM; the paper does not report a scaling curve."],"forward_implications":["Zero-shot prompting can narrow the gap between general MLLMs and expert-level pathology reasoning without collecting domain-specific fine-tuning data.","Question-dependent captions add 1.20% tiny-test accuracy over vanilla captions, indicating that telling the model what to look for in the image matters.","Removing the expert-analysis stage costs 3.70% on the tiny test, the largest single ablation, showing that simulated domain expertise is the main driver of the gain.","Self-evaluation recovers between 1.61% and 2.39% accuracy, confirming that CoT answers on pathology images are not always better than direct answers and need verification.","The gains appear consistently across all four PathMMU subsets (PubMed, EduContent, Atlas, PathCLS) and on both tiny and full test sets."],"supporting_citations":[{"why":"Supplies the PathMMU benchmark used for all experiments and the evidence that current MLLMs underperform on pathology reasoning.","marker":"[19]"},{"why":"MC-CoT, the strongest zero-shot CoT baseline for medical VQA, provides the direct comparison and the modular collaboration idea extended here.","marker":"[21]"},{"why":"MM-CoT introduces the two-stage rationale-then-answer paradigm that PathCoT builds on and is one of the baselines.","marker":"[14]"},{"why":"Qvix demonstrates the value of descriptive information in zero-shot image reasoning, supporting the caption-generation stage, and is a baseline.","marker":"[15]"},{"why":"DDCoT breaks problems into sub-questions, providing a baseline and a contrast to unified expert analysis.","marker":"[16]"},{"why":"CCoT uses scene graphs for visual reasoning, one of the CoT baselines PathCoT must beat.","marker":"[18]"},{"why":"Establishes zero-shot CoT prompting, the base paradigm that PathCoT extends to multimodal pathology reasoning.","marker":"[11]"},{"why":"LLaVa is the MLLM backbone used for all methods and ablations in the paper.","marker":"[2]"}],"fun_headline_variants":["PathCoT: expert-guided CoT for zero-shot pathology reasoning","Zero-shot pathology reasoning boosted by expert knowledge and self-check","PathCoT blends expert knowledge with self-evaluation for pathology AI","Expert-knowledge prompting sharpens zero-shot pathology visual reasoning","PathCoT: zero-shot pathology reasoning with expert priors and self-validation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire gain depends on the self-evaluation call (Eq. 3) being able to pick the correct answer when the chain-of-thought answer and the direct answer disagree; the paper provides no data on how often that adjudication fails.","fun_headline_variants_meta":{"raw":{"variants":["PathCoT: expert-guided CoT for zero-shot pathology reasoning","Zero-shot pathology reasoning boosted by expert knowledge and self-check","PathCoT blends expert knowledge with self-evaluation for pathology AI","Expert-knowledge prompting sharpens zero-shot pathology visual reasoning","PathCoT: zero-shot pathology reasoning with expert priors and self-validation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0008,"raw_usage":{"total_tokens":3571,"prompt_tokens":1051,"completion_tokens":2520,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":667,"completion_tokens_details":{"reasoning_tokens":2431}},"tokens_in":667,"tokens_out":2520,"duration_ms":17429,"temperature":1.0,"reasoning_tokens":2431,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:36:24.017972+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PathCoT on the PathMMU test sets while logging the accuracy of the self-evaluation step only for questions where $A_{\\text{CoT}}$ and $A_{\\text{dir}}$ differ; if the self-evaluation selects the correct candidate at or below chance on those conflicting cases, the reported gains would not survive.","supporting_citations":[{"cited_title":"Pathmmu: A massive multimodal expert-level benchmark for understanding and reasoning in pathology","cited_arxiv_id":null,"evidence_quote":"Supplies the PathMMU benchmark used for all experiments and the evidence that current MLLMs underperform on pathology reasoning."},{"cited_title":"Ddcot: Duty- distinct chain-of-thought prompting for multimodal reasoning in language models","cited_arxiv_id":null,"evidence_quote":"DDCoT breaks problems into sub-questions, providing a baseline and a contrast to unified expert analysis."},{"cited_title":"Composi- tional chain-of-thought prompting for large multimodal models","cited_arxiv_id":null,"evidence_quote":"CCoT uses scene graphs for visual reasoning, one of the CoT baselines PathCoT must beat."},{"cited_title":"Large language models are zero-shot reasoners.Advances in neural information processing systems, 35:22199–22213, 2022","cited_arxiv_id":null,"evidence_quote":"Establishes zero-shot CoT prompting, the base paradigm that PathCoT extends to multimodal pathology reasoning."}],"review_version":2}