{"id":"72069441-63dc-4151-998b-b261d47cde85","arxiv_id":"2608.07651","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A three-step agentic workflow with LLM function calling and reflection improved glaucoma classification, CDR estimation, and repeatability over LLM-alone baselines, approaching specialist-level accuracy.","lead":"An AI framework that lets a language model call specialized deep learning tools improved glaucoma detection from retinal photos, cutting error and run-to-run inconsistency relative to language models used alone. The approach reached near-specialist accuracy on two public datasets, suggesting tool-using orchestration may matter more than raw model size in medical imaging AI.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Agentic gains over LLM-alone track the SwinV2-Tiny classifier (ORIGA 68%→69-73%; RIM-ONE 88%→85-88%); without a static tool-fusion baseline, the claim that agentic orchestration drives the improvement is unsupported.","rationale":"I read the paper as an engineering demonstration that an LLM coordinating specialist tools can outperform the LLM alone. That directional result is consistently observed, and the inclusion of DL-only numbers in the text is honest evidence. However, the central explanatory claim—that the agentic architecture, and in particular the reflection step, is responsible for the gains—is not supported by the reported comparisons. The DL classifier alone accounts for nearly all of the accuracy improvement: on ORIGA it reaches 68% vs. 69-73% for the agent, and on RIM-ONE-v3 it reaches 88% vs. 85-88%. The only evidence that the reflection step adds value is the qualitative error-correction examples and the '71.7% vs 54.5%' arbitration accuracy, but no aggregate ablation or baseline is provided. Because the paper's title and discussion generalize to a 'paradigm shift toward orchestrated multi-agent systems,' the marginal contribution of the LLM orchestration is load-bearing. The proposed static baseline would settle whether the agentic LLM layer is necessary or whether a deterministic fusion of the specialist tools reproduces the results. This does not change the reader's CONDITIONAL verdict, but it sharpens the condition: the authors should report this ablation before claiming architectural benefit. I partially agree with the reader's weakest assumption, which focused on tool transferability and specialist noise; the attribution gap is adjacent but distinct.","tokens_in":15546,"tokens_out":7004,"duration_ms":67629,"concrete_test":"Implement a non-agentic baseline for all 16 conditions using the same four DL tools with fixed deterministic rules (e.g., glaucoma if SwinV2-Tiny probability >0.5 or SegFormer vCDR>0.6, with quality-based abstention) and no LLM reflection. Compare its accuracy, specialist κ, and run-to-run κ to Tables 2-3. If the static baseline matches or exceeds the agentic results, the load-bearing claim that the agentic workflow's orchestration/reflection drives the gains is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the agentic architecture, not model scale, explains the improvement over LLM-alone. But the 'Agentic AI vs. DL tools' section reports that the single SwinV2-Tiny classifier alone achieves 68% accuracy on ORIGA and 88% on RIM-ONE-v3, while the agentic pipeline achieves 69-73% and 85-88%, respectively. Thus the 16-47 percentage-point gains relative to LLM-alone are almost entirely explained by the deterministic DL classifier, not by the LLM's function-calling or reflection. On RIM-ONE-v3 the agentic pipeline is at or slightly below the classifier alone, and run-to-run kappa is 0.85-0.96 rather than 1.0, so the stochastic reflection step adds variability without a clear accuracy benefit. The abstract's 'within 6 points of the specialist' also relies on comparing AI accuracy to dataset labels, while the specialist agrees with those labels only 79% on ORIGA; AI-specialist agreement on ORIGA is only κ=0.38-0.55. The paper never reports an ablation in which the LLM reflection is replaced by a fixed rule over the same DL outputs, so the marginal contribution of the agentic orchestration is unmeasured.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an agentic AI framework for glaucoma detection from fundus photographs, in which an LLM first gives an initial assessment, then invokes four specialized deep learning tools (QAModel, FundaQ-8, SwinV2-Tiny, SegFormer-B0) via function calling, and finally reflects on the combined evidence to produce a structured output. The authors evaluate two LLMs (Gemini 2.5 Flash, GPT-5.4 mini) on 100-image subsets of ORIGA and RIM-ONE-v3 under cropped and uncropped conditions, with each condition run twice. They report that the agentic workflow improves classification accuracy by 16-47 percentage points over LLM-alone, brings accuracy within 6 points of a fellowship-trained specialist on ORIGA and matches 88% on RIM-ONE-v3, reduces CDR estimation error, and raises run-to-run kappa from near-random to near-perfect. The paper argues that the agentic architecture, not model scale, drives the gains.","tokens_in":15793,"tokens_out":3863,"duration_ms":38126,"significance":"If the central claim holds, the paper provides a practical template for grounding LLM-based medical image interpretation in deterministic, auditable tool outputs, directly addressing reproducibility and hallucination concerns that are currently central to clinical LLM deployment. The use of open-source DL tools, explicit reporting of per-condition accuracy/sensitivity/specificity/F1, and the inclusion of run-to-run consistency as a safety metric are strengths; the paper also honestly discusses several limitations, including the RIM-ONE quality overclassification issue and the single-specialist reference standard. The main significance hinges on whether the agentic orchestration itself adds value over simply using the DL tools, which is not yet established.","major_comments":[{"comment":"The standalone DL classifier SwinV2-Tiny achieves 68% accuracy on ORIGA and 88% on RIM-ONE-v3, which is essentially identical to the agentic pipeline's 69-73% and 85-88%. Because the 16-47 percentage-point gains over LLM-alone are thus almost entirely explained by the deterministic DL tool outputs, the claim that agentic orchestration is the driver of improvement is not supported without a static-fusion baseline. The authors should add an ablation in which the LLM reflection is replaced by a fixed rule over the same tool outputs (e.g., majority vote, or priority to segmentation-derived CDR), and compare its accuracy and consistency to the agentic workflow. If the agentic version does not exceed that baseline, the abstract's attribution of gains to the agentic architecture must be revised.","section":"Results, 'Agentic AI vs. DL tools'"},{"comment":"No confidence intervals, significance tests, or effect-size measures are provided for any accuracy, sensitivity, specificity, or kappa difference. With n=100 per dataset and only two runs per condition, the 1-5 percentage-point differences among agentic conditions (e.g., ORIGA 69% vs 73%) are within plausible sampling variability. Report bootstrap confidence intervals for proportions and kappas, use paired McNemar tests for accuracy comparisons against the same images, and report Run-2 accuracy values rather than only Run-1 accuracy, so that readers can assess the stability of the headline improvements.","section":"Results and Table 3"},{"comment":"The statement that agentic AI performs 'within 6 points of the specialist' is based on comparing both AI and the specialist to dataset labels. On ORIGA, however, the specialist agrees with dataset labels in only 79% of cases, and direct AI-specialist agreement on ORIGA is only kappa = 0.38-0.55 (fair to moderate by Landis-Koch). The near-specialist framing in the abstract and conclusion should be tempered by explicitly presenting the direct AI-specialist kappa values alongside the accuracy comparison, especially for ORIGA, and by acknowledging that a single specialist grading with 3 excluded 'cannot determine' images per dataset is a weak reference standard for the specialist-comparison claim.","section":"Results, 'Glaucoma Classification Performance' and Table 3"},{"comment":"The exclusion of images graded 'cannot determine' (n=3 per dataset) is stated, but the paper does not report how those exclusions affect the dataset-label-based accuracy comparisons or the specialist-agreement analyses. Because the excluded images are exactly those where the specialist was uncertain, their removal could inflate both AI-specialist agreement and the accuracy of any predictor that tends to avoid committing on hard images. The authors should report the number of excluded images in each analysis and provide a sensitivity analysis (e.g., treating excluded images as incorrect for both AI and specialist) to show that the main conclusions are robust to this exclusion.","section":"Methods, 'Evaluations and Repeatability Analysis'"}],"minor_comments":[{"comment":"The footnote explaining the RIM-ONE-v3 Cropped poor-recall denominators (17 vs 18) is confusing: the denominator for recall should be the number of specialist-graded poor images, which is fixed at 18 for RIM-ONE-v3; clarify that one image was excluded because the predictor labeled it ungradable, and state how this affects the reported recall values.","section":"Table 5 footnote"},{"comment":"Two different figures are both labeled 'Supplemental Figures S2' (one showing classification performance, one showing predicted vs specialist CDR). Renumber them sequentially to avoid ambiguity.","section":"Supplemental Figures"},{"comment":"Figure 1A refers to 'ChatGPT-5' while the study evaluates Gemini 2.5 Flash and GPT-5.4 mini; clarify whether the illustrative panels are from the current models or from prior work, and ensure the caption matches the models actually evaluated.","section":"Figure 1"},{"comment":"The column header 'ChatGPT' should be changed to 'GPT-5.4 mini' for consistency with the text, since the paper evaluates GPT-5.4 mini rather than the base ChatGPT model.","section":"Tables 2 and 3"},{"comment":"The phrase 'paradigm shift' in the conclusion is editorializing and goes beyond the evidence presented; a more measured statement about the potential of orchestrated systems would be appropriate given the limitations acknowledged in the same section.","section":"Discussion"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim is plausible but the missing static-fusion ablation is a genuine load-bearing gap: the paper's own Table 2 data show the DL classifier alone nearly matches the agentic pipeline, so the attribution of the gains to 'agentic orchestration' is currently unsupported. The two-run design without confidence intervals further weakens the quantitative claims. The manuscript would be substantially stronger if the authors add the ablation and statistical reporting; without those, the abstract's strongest formulations are not justified. The paper is within the scope of the journal and the topic is timely, but I would not accept it in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this paper reports a real, consistent result. An LLM that calls a few deterministic DL tools for fundus quality, glaucoma classification, and disc/cup segmentation beats bare LLMs by 16 to 47 points on ORIGA and RIM-ONE-v3, and it corrects two distinct failure modes: systematic positive bias and run-to-run randomness. That is useful, and the run-to-run kappa framing as a clinical safety metric is a good addition to evaluation practice.\n\nWhat is actually new is the specific application of function-calling plus reflection to glaucoma screening and the explicit measurement of identical-run consistency. The components are not new—radiology agents already do this—but the ophthalmic setting and the consistency framing are. The study is clean and honest: two LLMs, two datasets, two crops, a masked specialist as an extra reference, clearly reported tables, and a limitations paragraph that names the single specialist, the two runs, the undocumented QAModel training data, and the RIM-ONE overclassification of poor-quality images.\n\nThe soft spot is load-bearing. The 'Agentic AI vs. DL tools' section shows the SwinV2-Tiny classifier alone achieves 68% accuracy on ORIGA and 88% on RIM-ONE-v3, while the agentic pipeline achieves 69–73% and 85–88%. So the 16–47 point gain over LLM-alone is almost entirely the deterministic classifier, not the LLM's tool selection or reflection. The paper argues reflection corrects classifier false positives and shows plausible examples, but the measured margin over the classifier is 1–5 points on ORIGA and slightly negative on RIM-ONE. There is no static baseline—a fixed rule over the same tool outputs, or a no-reflection agent—so the marginal contribution of orchestration is unquantified. The 'within 6 points of the specialist' claim also needs the caveat that the specialist agrees with dataset labels only 79% on ORIGA, and AI-specialist kappa is 0.38–0.55 there. The tool-transfer worry is real but secondary; the tools performed adequately on these datasets, so the missing ablation is the principal gap.\n\nThese are fixable gaps. The directional finding holds up, the paper is transparent, and the consistency metric is worth citing. It just does not support the strong conclusion that 'architectural design rather than model scale' drives the gains, or that this signals a paradigm shift. The RIM-ONE overclassification of poor-quality images is an acknowledged transfer problem and a useful caution.\n\nWho is this for? People building tool-augmented medical imaging systems and anyone testing LLM-versus-agent claims will find it a useful reference. I would send it to peer review—it deserves referee time—but I would demand a static-fusion control and uncertainty quantification before publication. Bring it to reading group if you want to discuss ablation design.","headline":"Real and consistent gains, but the improvement over LLM-alone is driven mainly by the injected SwinV2-Tiny classifier, not the agentic orchestration.","tokens_in":16361,"tokens_out":3476,"would_cite":true,"duration_ms":32864,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An agentic workflow—LLM plus four specialized tools plus a reflection step—raises glaucoma classification accuracy by 16–47 percentage points over LLM alone and comes within 6 points of a specialist.","keywords":["agentic AI","large language models","glaucoma detection","fundus photography","deep learning tools","function calling","reflection","run-to-run consistency"],"falsifier":"Run the same 16 conditions on a third, held-out fundus dataset with three masked glaucoma specialists and compare agentic versus LLM-alone accuracy, specialist agreement, and run-to-run $\\kappa$; the central claim is contradicted if the agentic accuracy advantage over LLM-alone falls below the reported 16-point margin or if run-to-run $\\kappa$ does not rise above fair agreement.","tokens_in":15359,"feed_emoji":"👁️","tokens_out":10830,"duration_ms":87403,"temperature":0.7,"pith_summary":"This paper argues that the usual fix for LLM unreliability in medical imaging—bigger models or fine-tuning—is not the only path: an LLM acting as an orchestrator around deterministic specialized tools can correct its failure modes. On 200 fundus photographs from two public datasets, the agentic workflow raised glaucoma classification accuracy by 16 to 47 percentage points over the same LLMs reading images alone, reaching within 6 points of a fellowship-trained glaucoma specialist and matching the specialist's 88% accuracy on RIM-ONE-v3. The workflow also cut cup-to-disc ratio error by 15–50%, raised correlation with specialist grading from weak ($r = 0.12$–$0.39$) to moderate-strong ($r = 0.59$–$0.84$), and improved run-to-run agreement from near-random ($\\kappa$ as low as $-0.01$) to near-perfect ($\\kappa$ up to $0.96$). If true, this means architectural design, not model scale, can be the main driver of clinically usable LLM performance in screening.","feed_headline":"Agentic AI lifts glaucoma detection up to 47 points over LLMs alone","feed_subtitle":"Orchestrated deep-learning tools fix LLM overdiagnosis and near-random run-to-run agreement.","key_machinery":"The central mechanism is the three-step agentic workflow, which turns the LLM from direct predictor into orchestrator: the LLM first reads the image and gives an initial impression, then native function calling invokes four deterministic open-source tools—QAModel and FundaQ-8 for image quality, SwinV2-Tiny for glaucoma probability, and SegFormer-B0 for optic disc/cup segmentation with vertical cup-to-disc ratio—and finally the LLM reflects, integrating its initial impression with tool outputs into a single structured report. This anchors predictions in reproducible measurements while preserving the LLM's clinical reasoning.","core_discovery":"The paper's central claim is that an agentic AI workflow—structured as LLM initial assessment, function calling to four specialized open-source deep learning tools (QAModel and FundaQ-8 for quality, SwinV2-Tiny for glaucoma classification, and SegFormer-B0 for disc/cup segmentation), and LLM reflection—overcomes the three LLM limitations of weak ophthalmic grounding, non-determinism, and hallucinated confidence. Across both LLMs, both datasets, and both field-of-view conditions, the workflow improved glaucoma classification accuracy by 16 to 47 percentage points over LLM-alone, corrected GPT-5.4 mini's systematic positive bias (sensitivity 95–100%, specificity 0–5%) and Gemini 2.5 Flash's stochastic variability, and raised run-to-run $\\kappa$ from as low as $-0.01$ to as high as $0.96$. It reached within 6 points of a masked fellowship-trained glaucoma specialist on ORIGA (73% vs. 79%) and matched the specialist's 88% accuracy on RIM-ONE-v3. The authors take this as evidence that a modular orchestrated system, rather than a larger monolithic LLM, can make LLM-based image analysis clinically reproducible.","pith_inferences":["A natural extension the paper does not test is that the same orchestration pattern could correct LLM bias in other image-based screening tasks (diabetic retinopathy, AMD, skin lesions) wherever a deterministic classifier and a segmentation/quality tool already exist.","The paper's arbitration analysis implies morphological evidence (segmentation-derived CDR) may be more reliable than classifier probability scores when tools conflict; a targeted ablation could test this by withholding either tool output from the reflection step.","The RIM-ONE quality over-classification suggests the quality tools carry dataset-specific illumination biases; an implication is that the framework would need per-site calibration of its quality thresholds before deployment on new cameras or populations."],"forward_implications":["If the reported gains hold, glaucoma screening systems can approach specialist-level accuracy using low-cost LLM tiers plus a small suite of open-source tools, instead of relying on flagship models.","The jump in run-to-run agreement (from $\\kappa$ near $-0.01$ to up to $0.96$) means an LLM-based reader could satisfy the reproducibility requirement for clinical workflows.","Because the specialized tools always receive the full fundus image, agentic accuracy stays stable when the LLM sees only a disc-centered crop, removing a preprocessing dependency that hurt LLM-alone reading.","The reflection step can outperform the DL classifier alone (69–73% vs. 68% on ORIGA) by correcting classifier false positives, so the system acts as an intelligent integrator rather than a pass-through."],"supporting_citations":[{"why":"Prior LLM-alone baseline on the same two datasets; its documented 30–35% run-to-run disagreement and modest accuracy motivate the failure modes the agentic workflow targets.","marker":"[16]"},{"why":"One of the two evaluation datasets; provides 100 fundus photos with SiMES clinical labels.","marker":"[30]"},{"why":"The second evaluation dataset; provides 100 fundus photos with specialist-adjudicated labels.","marker":"[31, 32]"},{"why":"Supply the quality-regression tool QAModel that gives a deterministic, bounded 0–1 image quality signal during function calling.","marker":"[34, 35]"},{"why":"Supply the FundaQ-8 tool that scores eight clinical image-quality attributes, the second deterministic quality signal.","marker":"[36, 37]"},{"why":"Supply the SwinV2-Tiny glaucoma classifier whose probability output anchors the LLM's final glaucoma decision.","marker":"[38, 39]"},{"why":"Supply the SegFormer-B0 disc/cup segmentation whose vCDR output anchors quantitative CDR estimates.","marker":"[40, 41]"},{"why":"One of the two LLM backbones; the agentic gain holds for it and it is the model whose stochastic variability is corrected.","marker":"[28]"},{"why":"The other LLM backbone; its systematic positive bias under LLM-alone is corrected by the agentic workflow.","marker":"[29]"}],"fun_headline_variants":["Agentic AI fixes LLM glaucoma blind spots, gains up to 47 points","Orchestrated AI tools beat LLM-only glaucoma reads by 47 points","LLM plus specialized tools lifts glaucoma accuracy 16-47 points","Agentic workflow tames LLM glaucoma hallucinations and drift","Multi-agent AI matches specialist in glaucoma detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework's gains assume the four open-source deep learning tools give trustworthy outputs on ORIGA and RIM-ONE-v3 images, even though SwinV2-Tiny and SegFormer-B0 were trained on REFUGE and QAModel's training set is not publicly documented; if those tools transfer poorly or carry dataset-specific biases, the agentic grounding inherits or amplifies them.","fun_headline_variants_meta":{"raw":{"variants":["Agentic AI fixes LLM glaucoma blind spots, gains up to 47 points","Orchestrated AI tools beat LLM-only glaucoma reads by 47 points","LLM plus specialized tools lifts glaucoma accuracy 16-47 points","Agentic workflow tames LLM glaucoma hallucinations and drift","Multi-agent AI matches specialist in glaucoma detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000208,"raw_usage":{"total_tokens":1547,"prompt_tokens":1232,"completion_tokens":315,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":848,"completion_tokens_details":{"reasoning_tokens":226}},"tokens_in":848,"tokens_out":315,"duration_ms":3683,"temperature":1.0,"reasoning_tokens":226,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:26:17.687522+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 16 conditions on a third, held-out fundus dataset with three masked glaucoma specialists and compare agentic versus LLM-alone accuracy, specialist agreement, and run-to-run $\\kappa$; the central claim is contradicted if the agentic accuracy advantage over LLM-alone falls below the reported 16-point margin or if run-to-run $\\kappa$ does not rise above fair agreement.","supporting_citations":[{"cited_title":"March 17, 2026; Available from: https://openai.com/index/introducing-gpt-5-4-mini-and-nano/","cited_arxiv_id":null,"evidence_quote":"The other LLM backbone; its systematic positive bias under LLM-alone is corrected by the agentic workflow."},{"cited_title":"Origa-light: An online retinal fundus image database for glaucoma analysis and research","cited_arxiv_id":null,"evidence_quote":"One of the two evaluation datasets; provides 100 fundus photos with SiMES clinical labels."}],"review_version":1}