{"id":"05e934ae-6c0c-4f75-a345-9a7c8da37aff","arxiv_id":"2412.01233","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper recommends starting LLM use in radiology with prompt optimization and retrieval augmentation, fine-tuning only when needed, and preferring locally hosted open models with human expert evaluation.","lead":"This paper is a review that compiles best practices for using large language models in radiology, covering prompting, retrieval-augmented generation, fine-tuning, evaluation, privacy, and regulation. A generalist reader would read it to understand the practical choices and risks when bringing AI text tools into medical imaging, and why the authors recommend escalating from simple prompting to fine-tuning only as a last step.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Escalation ladder's ordering lacks direct radiology evidence; a head-to-head benchmark across adaptation levels is needed before the central recommendation is treated as settled.","rationale":"The reader identified transferability of general-domain best practices as the weakest assumption. I agree and sharpen it: the load-bearing issue is not merely that recommendations may transfer imperfectly, but that the paper's central escalation ladder assumes an ordinal cost-effectiveness relationship among prompting, RAG, and fine-tuning. The manuscript does not provide direct radiology-specific head-to-head evidence for that ordering; Section 4.3 relies on a general-domain citation (ref. 120) for the trade-off, and the claim that prompting only moderately influences radiology-specific abilities is stated without a dedicated radiology comparison. Additionally, the ladder is text-centric, while a central radiology task is image interpretation; the paper handles this by discussing multimodal LMMs separately, but the conclusion's recommended path does not integrate that discussion into the ladder. This is a limitation of evidence rather than an internal contradiction, so it does not warrant rejecting the review. The authors hedge appropriately and the individual recommendations are broadly reasonable and consistent with current practice. The two unresolved citation placeholders ('?, 90' and '?, 64') and the mismatched 'BLOOM(27)' reference are real editorial defects that reinforce the conditional status, but they are not the central scientific concern. The proposed benchmark would settle whether the ladder's ordering is empirically justified for radiology or should be made task-dependent.","tokens_in":25904,"tokens_out":3430,"duration_ms":35397,"concrete_test":"Run a head-to-head benchmark on three representative radiology tasks (report summarization, critical finding extraction, and chest X-ray impression generation) using two or three model families, contrasting four configurations: prompt-only, prompt + RAG with curated guidelines, LoRA fine-tuning, and full fine-tuning. Evaluate on a fixed held-out set with radiologist expert preference plus an automated clinical metric (e.g., RadGraph F1 or MEDCON), and repeat at a second institution. If for any task fine-tuning from the start outperforms the best prompt/RAG configuration by a prespecified margin (e.g., 5% in expert preference), the conclusion's universal ordering should be revised to a task-dependent recommendation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central recommendation in Section 5 is an ordered ladder: optimized prompting, then in-context learning and retrieval-augmented generation, then tools, then fine-tuning 'if necessary.' This ladder presupposes a cost-effectiveness ordering in which text-level, in-context interventions should be tried before model-weight adaptation, and that this ordering transfers to radiology tasks. That premise is not directly supported in the manuscript. Section 4.3 ('Balancing prompting, retrieval-augmented generation and fine-tuning') states that 'Prompting will only moderately influence concrete radiology-specific abilities,' but the only comparative citation for the prompting-vs-fine-tuning choice is ref. 120, a general-domain knowledge-injection study, not a radiology study. The ladder is also text-centric: for the core radiology task of image interpretation, prompt optimization and text RAG cannot supply visual competence; the model must be multimodal or paired with a vision bridge, and the paper only advises readers to 'keep an eye' on large multimodal models. If, for many real radiology tasks, fine-tuning or multimodal model selection is required from the start, the recommended escalation from cheap text-level methods loses its force. The authors do hedge ('may vary across different LLM types and versions'), but the ordinal structure of the recommendation is still presented as general guidance without head-to-head radiology evidence for each rung.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript is a narrative review of best practices for using and adapting large language models (LLMs) in radiology. It covers technical foundations (transformers, training pipeline, multimodal extensions), possible applications (clinical, operational, research, education), desiderata, limitations (confabulation, privacy, bias, reproducibility), adaptation methods (prompting, in-context learning, retrieval-augmented generation, instruction tuning, alignment, parameter-efficient fine-tuning, quantization, distillation), and evaluation strategies (automatic metrics, radiology-specific benchmarks, human expert assessment). The central recommendation, stated in Section 5, is an iterative escalation ladder: start with optimized prompting of general-purpose models, then enrich context through in-context learning and retrieval-augmented generation, enable access to tools, and finally fine-tune if necessary, while preferring local open models and maintaining ongoing human evaluation.","tokens_in":26033,"tokens_out":3879,"duration_ms":36352,"significance":"If the recommendations are accepted, this review could serve as a practical entry point for radiology teams without dedicated machine-learning resources, helping them decide how to deploy LLMs for language-oriented tasks in radiology. The paper is timely and broad, and it has several concrete strengths: Table 1 and Table 2 are actionable and clearly organized; the authors explicitly hedge that effectiveness depends on the model and may change over time; practical cost estimates for pretraining and fine-tuning are useful; and the review covers a wide range of recent radiology-specific and general-domain sources. As a narrative review rather than a primary study, its value lies in synthesis and guidance, and the authors generally distinguish between established practice, emerging evidence, and suggestions based on practical experience.","major_comments":[{"comment":"The manuscript contains unresolved citation placeholders: 'radiological images.?, 90' in Section 4.1 and 'Model Cards and Versioning.?, 64' in Section 4.1.1. These incomplete citations undermine the evidence base for statements about multimodal LLM performance and model cards, and they are not acceptable in a submitted manuscript. Please replace the '?' placeholders with the intended reference keys or remove the citations.","section":"Section 4.1 (Multimodality paragraph) and Section 4.1.1"},{"comment":"The central recommendation is an ordered escalation ladder: prompting, then in-context learning/RAG, then tools, then fine-tuning. The manuscript states in Section 4.3 that 'Prompting will only moderately influence concrete radiology-specific abilities,' but the only comparative citation for the prompting-versus-fine-tuning choice is ref. 120, a general-domain knowledge-injection study. No radiology-specific head-to-head evidence is provided for the ordering. Because the ladder is the paper's main takeaway, the authors should either cite radiology-specific comparisons (e.g., prompting versus fine-tuning for report summarization or information extraction) or explicitly reframe the ordering as a resource-based heuristic rather than an evidence-based effectiveness ranking, and discuss the evidence gap. The existing hedges about model dependence are helpful but do not directly address the ordinal claim.","section":"Section 4.3 and Section 5 (Central recommendation)"},{"comment":"The proposed ladder is text-centric and does not clearly account for the core radiology task of image interpretation. The paper acknowledges that large multimodal models are needed for image-based tasks and advises readers to 'keep an eye' on them, but the central recommendation does not integrate multimodal model selection or vision bridges into the escalation path. For many real-world radiology tasks, prompt optimization and text-based RAG cannot supply visual competence; the appropriate starting point may be selecting a multimodal model or pairing a text LLM with a vision module. Please clarify how the escalation ladder applies to language-only tasks versus image-interpretation tasks, and incorporate the multimodality discussion into the central recommendation.","section":"Section 4.3 and Section 5"}],"minor_comments":[{"comment":"There is a typo: 'particulary' should be 'particularly.'","section":"Section 2.2"},{"comment":"The example JSON in the few-shot prompting row has mismatched braces, making the example harder to parse; consider formatting it as a complete, valid JSON object.","section":"Table 1, Few-shot prompting row"},{"comment":"The phrase 'The presented method groups increase in complexity' is awkward; consider rewording to 'The presented groups of methods increase in complexity.'","section":"Figure 5 caption"},{"comment":"Several author names contain stray spacing or OCR-like artifacts (e.g., 'LLaV A' for LLaVA, 'V o' in reference author names); a careful proofreading pass is needed.","section":"Section 4.1 and elsewhere"},{"comment":"The cost calculation is written as '$33/8 GPUs/h', which is ambiguous; clarify whether the $33 is per GPU-hour or for a group of GPUs, and make the arithmetic explicit.","section":"Footnote 2 in Section 2.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a review and not a primary research paper, so the standard of evidence for the central recommendation should be judged accordingly; nevertheless, the ordinal escalation ladder is presented as general guidance and the evidence gap identified in Section 4.3 is worth addressing directly. The unresolved citation placeholders suggest the arXiv version was not camera-ready. The self-citations are used as examples in a survey and the core recommendations rely on independent external references, so I do not see a circularity problem."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a review paper, and it should be judged as one. The core value is the practical escalation ladder: start with prompt optimization, then in-context examples and RAG, then fine-tuning if necessary, while avoiding expensive pretraining and preferring locally hosted open models. That framing is genuinely useful for radiology departments trying to get from demos to deployments. The paper does what a good review should: it consolidates a scattered literature, gives concrete guidance in Tables 1 and 2, includes cost examples that help teams budget, and emphasizes human expert evaluation and privacy throughout. The recommendations are mainstream and consistent with the cited sources. Credit is deserved for the clear structuring; this is better than the typical survey.\n\nThe soft spots are real but not fatal. First, the manuscript is not finished. Section 4.1 contains an unresolved placeholder ('radiological images.?, 90'), Section 4.1.1 has another ('reproducibility.?, 64'), and the in-text reference to 'BLOOM(27)' points to Reed et al.'s generalist agent paper rather than a BLOOM model. These need to be fixed before this can be treated as a reliable resource; as is, the carelessness undermines the authority the review needs.\n\nSecond, the stress-test concern about the ladder's ordering has some teeth. Section 4.3 admits that 'prompting will only moderately influence concrete radiology-specific abilities,' and the only direct comparative citation for prompting-vs-fine-tuning is a general-domain knowledge-injection study, not radiology. Moreover, the ladder is text-centric: for image interpretation, text prompting and text RAG cannot conjure visual competence, so the model must be multimodal or paired with a vision bridge. The authors hedge ('may vary across different LLM types and versions'), and each individual step is reasonable, but the ordinal structure is presented as general guidance without head-to-head radiology evidence. That is a limitation worth noting in the review, not grounds for rejection.\n\nThere are no invented entities or fabricated results; the self-citations (RadAdapt, CheXagent, RaLEs, Almanac, etc.) appear as examples alongside independent literature, which is normal for a review. The difficulty ratings in Table 2 are subjective but explicitly labeled as the authors' estimates.\n\nWho gets value: radiology researchers, clinicians, and ML engineers planning to use LLMs, plus anyone teaching this material. It deserves a serious referee, but the referee should require the citation repairs and a frank acknowledgment that the ladder's ordering is based on general-domain evidence and practical judgment, not a validated radiology benchmark. I would send it to peer review with a conditional: fix the placeholders, add a sentence of caution about the missing comparative evidence, and it can be a genuinely useful contribution.","headline":"A solid, well-organized review of LLM best practices in radiology whose practical escalation ladder is useful and appropriately hedged, but the manuscript is unfinished (citation placeholders, a broken reference) and the ladder's ordering lacks direct radiology evidence.","tokens_in":26712,"tokens_out":2083,"would_cite":true,"duration_ms":21593,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This review argues that radiology departments should not pretrain their own large language models, but instead follow a staged escalation from prompting to retrieval-augmented generation to fine-tuning only when needed.","keywords":["large language models","radiology","prompt engineering","retrieval-augmented generation","fine-tuning","LoRA","model evaluation","privacy"],"falsifier":"A controlled comparison on a representative radiology task in which a small fine-tuned model clearly beats a well-prompted large model with retrieval augmentation, at equal or lower cost, would undercut the recommended order of escalation.","tokens_in":25601,"feed_emoji":"🩻","tokens_out":6538,"duration_ms":53194,"temperature":0.7,"pith_summary":"This review argues that radiology departments do not need to pretrain their own large language models. The authors recommend an iterative escalation: start with prompt optimization on general-purpose models, add in-context examples and retrieval-augmented generation to supply relevant knowledge, enable tools, and only fine-tune if the simpler steps are insufficient. The payoff is that radiology teams can deploy capable language tools for reporting, summarization, and communication while controlling compute cost, privacy risk, and confabulation. The recommendation rests on a synthesis of recent LLM research and radiology-specific studies rather than a new experiment.","feed_headline":"For radiology AI, prompt first, fine-tune last","feed_subtitle":"A review of current evidence says general-purpose LLMs should be prompted, grounded, and only then fine-tuned.","key_machinery":"The organizing mechanism is the iterative optimization ladder (Figure 5). It orders adaptation techniques by increasing complexity and cost: prompting, in-context learning, retrieval-augmented generation, tool access, and fine-tuning. Each higher rung permanently modifies the model or adds persistent infrastructure, so the recommendation is to stop at the lowest rung that meets the task requirement, with prompt engineering remaining relevant even after fine-tuning.","core_discovery":"The central claim is that the most effective path to using large language models in radiology follows a cost-ordered adaptation ladder. Its rungs are: iteratively refine the prompt; include a few task examples (in-context learning); ground the model with retrieved external knowledge (retrieval-augmented generation); give the model access to tools; and finally, if needed, fine-tune with curated data, preferably via parameter-efficient methods like LoRA. The paper further claims that pretraining from scratch is impractical for most radiology labs, that open locally hosted models should be favored over closed hosted models for privacy and reproducibility, and that evaluation must combine automated metrics with radiologist review and continue after deployment.","pith_inferences":["The paper does not say how to choose the threshold that triggers a move up the ladder; an operational rule would be to escalate only when the current rung fails an agreed clinical benchmark.","The open-vs-closed model recommendation is driven by governance concerns rather than head-to-head accuracy evidence; head-to-head radiology benchmarks could change the balance.","As multimodal LLMs improve, the ladder may effectively start at a higher rung, with image-aware prompting already covering tasks like draft report generation without any fine-tuning.","The ladder can be tested prospectively: fix an evaluation set, measure each rung's gain, and check whether the gains plateau before fine-tuning."],"forward_implications":["Radiology AI projects can skip the multi-million-dollar pretraining step and begin with prompted general-purpose models already available today.","Factual reliability can be improved without retraining by grounding outputs in retrieved guidelines and reports, which also makes outputs easier to verify.","Privacy-sensitive deployment is feasible with locally hosted open models, since imaging data and notes do not need to be sent to an external API.","Fine-tuning, when needed, can be carried out cheaply with LoRA-style methods, making customization attainable for smaller labs.","Evaluation that pairs automated scores with radiologist review becomes the gatekeeper for deciding when to escalate to the next rung."],"supporting_citations":[{"why":"Establishes that general-purpose LLMs can follow instructions and learn from in-context examples, the foundation for the prompting-first rung.","marker":"10"},{"why":"Defines retrieval-augmented generation, the mechanism for grounding outputs in external knowledge.","marker":"118"},{"why":"Shows lightweight domain adaptation of an LLM can improve radiology report summarization, evidence for the fine-tuning rung.","marker":"12"},{"why":"Demonstrates that adapted LLMs can outperform human experts on clinical text summarization, supporting the value of fine-tuning.","marker":"52"},{"why":"Provides LoRA, the parameter-efficient fine-tuning method that makes the last rung affordable.","marker":"78"},{"why":"Documents the Llama 2 open model family, a basis for the recommendation to deploy local open models for privacy.","marker":"64"},{"why":"A radiology-specific primer that the paper draws on for practical use guidance.","marker":"5"}],"fun_headline_variants":["Radiology LLMs: prompt first, fine-tune last","Prompt, ground, then fine-tune: radiology LLM ladder","Cost-ordered LLM adaptation for radiology","For radiology AI, prompt before fine-tuning","Rad LLMs: refine prompts, then fine-tune"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire ladder assumes that techniques validated in general-domain LLM research and a small set of early radiology studies will transfer reliably to real radiology workflows across different models, hospitals, and tasks.","fun_headline_variants_meta":{"raw":{"variants":["Radiology LLMs: prompt first, fine-tune last","Prompt, ground, then fine-tune: radiology LLM ladder","Cost-ordered LLM adaptation for radiology","For radiology AI, prompt before fine-tuning","Rad LLMs: refine prompts, then fine-tune"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000188,"raw_usage":{"total_tokens":1292,"prompt_tokens":863,"completion_tokens":429,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":347}},"tokens_in":479,"tokens_out":429,"duration_ms":3844,"temperature":1.0,"reasoning_tokens":347,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:33:14.832653+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled comparison on a representative radiology task in which a small fine-tuned model clearly beats a well-prompted large model with retrieval augmentation, at equal or lower cost, would undercut the recommended order of escalation.","supporting_citations":[],"review_version":1}