{"id":"143694d6-039d-4b41-8bbc-39e992a55e74","arxiv_id":"2501.11803","paper_version":6,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A mostly automated radiotherapy planning pipeline produces 3,730 public treatment plans across head-and-neck and lung cancer, with code and a challenge baseline.","lead":"This paper presents an automated radiotherapy planning pipeline that generates large volumes of treatment plans, and releases a dataset of 3,730 plans for head-and-neck and lung cancer, over ten times larger than the leading public alternative. It matters because AI training for radiotherapy has been bottlenecked by scarce, inconsistent plan data, and this work aims to remove that bottleneck.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Quality claim is circular and unquantified: scorecard convergence is presented as proof of clinical comparability, while the paper itself disclaims definitive comparison (Sec. 4.2, Sec. 5).","rationale":"The reader's CONDITIONAL verdict already captures this weakness; my stress test adds precision by locating the circularity. The paper is an honest engineering contribution: it releases 3,730 DICOM plans plus code and a challenge, and the scale claim versus OpenKBP is verifiable from Table 1. The limitation section is unusually candid and should be weighed in the verdict; it explicitly concedes that the DVH/scorecard comparison cannot definitively support superiority, which conflicts with the abstract's unqualified 'comparable' claim. My proposed test is feasible because original clinical plans exist for the same TCIA patients; it would settle whether the quality claim is a real clinical equivalence or only internal scorecard consistency. The concern is not that the pipeline is fraudulent or useless; rather, the central assertion is currently supported by anecdote and a circular metric, so the verdict should remain CONDITIONAL with the quality claim downgraded until independent evaluation exists.","tokens_in":17832,"tokens_out":3602,"duration_ms":39573,"concrete_test":"Select 50 released cases (25 HaN, 25 Lung) with their original clinically approved plans, score both AIRTP and clinical plans with the same Varian scorecard and with two independent blinded dosimetrists using an external rubric (e.g., RTOG/EORTC constraints), and compare paired non-inferiority with a pre-registered margin. Also report how many plans were removed or re-planned during curation. If AIRTP fails non-inferiority on either evaluation, the comparability claim in the abstract should be weakened to 'scorecard-consistent plans'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing claim is that AIRTP plans are 'of quality comparable to those generated manually' (Abstract). The evidence for this is not independent: Algorithm 1 iteratively optimizes toward a Varian scorecard with hand-margined DVH objectives (Figs. 10-11), and Sec. 4.2 then supports 'plan quality is high' with DVH plots from two randomly selected cases plus a note that examples were manually checked by an experienced dosimetrist. The same section states 'we cannot definitively claim that the AIRTP plan is superior to the clinical manual plan based solely on DVHs or scorecards,' and Sec. 5's limitations say the comparison 'may not be entirely fair, as the AIRTP and clinical plans follow different guidelines.' No quantitative metrics, paired dose statistics, or blinded review are reported for the 3,730 plans. Table 1 also says corner cases are removed or re-planned, but the number discarded is not disclosed; if filtering was substantial, the released data are a favorable selection rather than the pipeline's steady-state output. Because the scorecard is both optimization objective and quality metric, scoring high on it is expected and does not establish clinical acceptability; it can reward metric-gaming. The risk is that HMM-RT is internally consistent with one vendor scorecard but not demonstrably comparable to manual clinical plans.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces the Automated Iterative RT Planning (AIRTP) pipeline, which combines AI-based OAR contouring, scripted Eclipse ESAPI-based beam setup and optimization, RapidPlan-based objectives, and an iterative scorecard-guided refinement loop to generate deliverable radiotherapy plans at scale. The authors release the HMM-RT dataset (3,730 plans from 1,622 patients across nine head-and-neck and lung cohorts) to support an AAPM 2025 challenge, and report a five-fold reduction in planning time (0.1-1 hour vs. 3-6 hours manual). The central claims are that AIRTP produces treatment plans of quality comparable to manually generated plans and that HMM-RT contains more than ten times the number of plans of OpenKBP, the largest previous well-curated public dataset.","tokens_in":18054,"tokens_out":4047,"duration_ms":40553,"significance":"If the quality and scale claims hold, this work addresses a real bottleneck in radiotherapy AI research: the scarcity of large, standardized, deliverable plan data. The public release of DICOM and processed numpy data, the challenge infrastructure, the baseline model, and the pretrained weights are concrete community resources. The scale advantage (3,730 vs. 340 plans) is well supported by Table 1, and the multi-cohort, multi-institution design with variable prescription doses is a substantive step beyond OpenKBP. However, the load-bearing claim of comparability to manual plans currently rests on circular and anecdotal evidence; the paper itself concedes in Sec. 4.2 and Sec. 5 that definitive comparison is not established. Additional independent validation would be needed to convert this from a useful dataset contribution into a demonstrated clinically comparable plan-generation method.","major_comments":[{"comment":"The evidence for 'quality comparable to manual plans' is circular: the Varian scorecard is both the optimization objective in the iterative refinement (Algorithm 1, lines 8-19, and Step 3 in Sec. 3) and the quality metric reported in Table 3. High scores under scorecard-guided optimization are expected and do not by themselves establish clinical acceptability or comparability to manual plans. The paper should compare the scorecard scores of the clinical/manual plans in the same cohorts, or add an independent evaluation (e.g., blinded dosimetrist scoring, or external plan-quality metrics that were not used during optimization).","section":"Sec. 4.2, Table 3; Algorithm 1 (lines 8-19)"},{"comment":"The direct comparison to clinical plans is limited to two randomly selected head-and-neck cases with qualitative DVH overlays; no lung cases, no quantitative paired DVH statistics, and no blinded review are reported. The manuscript's own statements—'we cannot definitively claim that the AIRTP plan is superior... based solely on DVHs or scorecards' (Sec. 4.2) and the comparison 'may not be entirely fair, as the AIRTP and clinical plans follow different guidelines' (Sec. 5)—are more cautious than the abstract's claim of quality 'comparable to those generated manually.' Please provide a quantitative comparison across a larger matched set (e.g., paired DVH metrics or blinded clinical review) or soften the abstract claim to match the evidence.","section":"Sec. 4.2, Fig. 5"},{"comment":"Table 1 states that 'corner cases' are 'remove or re-plan' to make plans 'more aligned to scorecard,' but the number and characteristics of excluded or re-planned cases are not disclosed. Without this information, the released 3,730 plans may represent a favorable selection rather than the pipeline's steady-state output. Please report the pass rate at each pipeline stage, the number of plans re-planned or discarded per cohort, the criteria used, and a sensitivity analysis of the released dataset to these filtering decisions.","section":"Table 1"},{"comment":"The convergence of the iterative refinement is supported only by the statement that 'when the parameters are properly tuned, empirical evidence shows that plan quality tends to converge toward a high score' (Sec. 3, Step 3). No convergence curves, no formal stopping criterion (Algorithm 1 leaves 'MAX iteration' and the 'close enough' condition undefined), and no sensitivity analysis for the hand-tuned margins and priorities in Figs. 10 and 11 are provided. Since the scalable-quality claim depends on this convergence, please show score-versus-iteration curves for a representative set of cases and quantify sensitivity to the Dose2Obj and Scorecard2Dose constants.","section":"Sec. 3, Step 3; Algorithm 1"}],"minor_comments":[{"comment":"There are several typographical errors, including 'an novel' (Sec. 1), 'Miscorsoft' (Sec. 4.1), 'V olumetric' (Sec. 2), 'featuresmore than 10 timesnumber' (Abstract), 'coustomized' (Appendix C), and 'T otal Lung-GTV' (Fig. 6b).","section":"Throughout"},{"comment":"The text states that head-and-neck VMAT plans use 'four arcs,' while the Fig. 4 caption and Fig. 6b label show 'two arcs'; please clarify whether this is two full arcs or four partial arcs, and ensure consistency across the text and figures.","section":"Sec. 3, Step 2; Fig. 4"},{"comment":"The stopping criterion is described qualitatively ('Plan i & i+1 close enough?'); please state the exact quantitative condition used in the actual runs (e.g., score difference threshold, dose difference threshold, or fixed iteration count).","section":"Algorithm 1"},{"comment":"The footnote says the validation/testing plan quality scores are based on subsets of 41 and 50 plans, respectively; please report the mean and standard deviation of the score distribution over these subsets, and clarify whether the same subset is used for all phases.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript contains a preprint note stating it was rejected by a NeurIPS program chair; this is not appropriate in the main text of a journal submission and should be removed or moved to a neutral footnote. As a dataset/benchmark contribution, the paper's fit with the journal's scope should be confirmed by the editor. The dataset size and public release are clear strengths, but the quality-comparability claim currently exceeds the evidence; the authors should be urged to conduct an independent blinded evaluation or explicitly limit the claim to 'internally consistent, high-scorecard plans' rather than 'comparable to manual plans.'"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline: treat this as a data-resource paper, not as a clinical-quality paper. The released HMM-RT dataset is a real contribution: 3,730 plans across head-and-neck and lung, roughly ten times OpenKBP, shipped as both DICOM and processed numpy, with an AAPM challenge, baseline code, tutorials, and a pretrained model. The iterative scorecard-guided refinement loop is new engineering, and the dose-to-plan reproduction method is a useful idea, even if it composes existing commercial components.\n\nWhat it does well: the pipeline is described concretely enough to follow, the data split is clean, and the benchmark setup is practical. The authors are also honest in the limitations: they say the comparison to clinical plans may not be entirely fair, and that plans follow different guidelines. That candor counts.\n\nThe soft spots are real and mostly where the reader put them. The central claim in the abstract—quality comparable to manual plans—is not backed by the evidence in the paper. The same scorecard is used as the optimization objective in Algorithm 1 and as the quality metric in Table 3, so high scores are partly the result of optimizing toward that target. The DVH comparison in Fig. 5 is two randomly selected cases, no quantitative paired statistics or blinded review. Table 1 mentions removing or re-planning corner cases without disclosing the number; if that fraction is large, the released set is a favorable selection. The dose-reproduction section shows visual DVH agreement but reports no error metric. None of this kills the dataset, but it means the \"clinical comparability\" claim should be softened to \"internally consistent with a single vendor scorecard.\"\n\nOne thing I'd push back on lightly: the paper itself flags the comparison limitation, so the circularity is not hidden. The flaw is that the abstract and Section 4.2 still overstate what was shown.\n\nWho this is for: anyone working on dose prediction or planning AI who wants a large, standardized, public training set. That is a meaningful segment, and the data will be used.\n\nI would send this to peer review. The dataset and pipeline deserve referee time, but the quality claim needs independent evaluation—e.g., blinded review of a sample of plans, quantitative DVH metrics against clinical plans, and disclosure of the filtering rate—before the \"comparable\" language is acceptable.\n\nWant me to also flag it for the reading group? I'd say bring it in once to discuss what counts as evidence for generated-plan quality.","headline":"A valuable large public RT dataset and a credible engineering pipeline, but the 'comparable to manual plans' claim is circular and needs independent evaluation before you trust it.","tokens_in":18700,"tokens_out":2429,"would_cite":true,"duration_ms":25252,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a fully automated, iterative treatment-planning pipeline can generate radiotherapy plans of quality comparable to manual plans in 0.1 to 1 hour instead of 3 to 6 hours, and that the released HMM-RT dataset is more…","keywords":["Radiotherapy treatment planning","Automated planning pipeline","Dose prediction","Treatment plan dataset","IMRT/VMAT","Head-and-neck cancer","Lung cancer","Organ-at-risk contouring"],"falsifier":"Run the pipeline on a held-out set of patients from an institution with different contouring conventions and count the plans that fail the scorecard after iteration; if the failure rate is high, or if a blinded dosimetrist review judges the automated plans clinically unacceptable despite high scores, the comparability claim would fall.","tokens_in":17564,"feed_emoji":"⚡","tokens_out":5239,"duration_ms":51626,"temperature":0.7,"pith_summary":"This paper tries to establish that the subjective, time-consuming work of manual radiotherapy treatment planning can be replaced by an automated iterative pipeline that produces plans of comparable quality far more quickly. The authors show that by scripting contouring, beam setup, optimization, and dose calculation, then repeatedly using dose-volume histogram points and scorecard metrics to tighten the planning objectives, they can generate plans that match manual plans in quality. At this scale, they release 3,730 treatment plans for head-and-neck and lung cancer, more than ten times the size of the largest existing curated public dataset. If true, AI researchers gain a scalable source of standardized high-quality plans for training dose-prediction and planning models.","feed_headline":"Automated plans match manual radiotherapy at a fraction of the time","feed_subtitle":"Cuts planning time from 3–6 hours to under an hour and releases 3,730 plans for AI training.","key_machinery":"The load-bearing mechanism is Algorithm 1, the iterative objective-refinement loop. For each organ at risk, the pipeline samples the dose-volume histogram at multiple volume points, computes a target dose and priority from hand-tuned rules, adds scorecard-derived objectives with margins, and re-optimizes until successive plans are close. This loop converts the qualitative notion of plan quality into explicit dose objectives, making convergence toward a high scorecard score an automated process.","core_discovery":"The central claim is that dosimetrist-style iterative planning can be encoded as a closed loop: auto-contour organs, script beam setup, seed objectives from knowledge-based planning, then repeatedly read dose-volume histogram points and scorecard metrics from the current plan and convert each point into a tightened optimization objective with a safety margin, re-optimizing until the plan converges. Running this loop at scale produced 3,730 deliverable IMRT and VMAT plans across head-and-neck and lung cohorts, with DVHs comparable to clinical plans. The same loop, with objectives derived only from a predicted 3D dose, reproduces a reference plan closely enough to turn dose predictions into deliverable plans.","pith_inferences":["Beyond the paper: if the dosimetric scorecard is a valid proxy for clinical quality, the same iterative loop could be reused as an automated plan-refinement tool inside planning software, not just as a data-generation pipeline.","Beyond the paper: the hand-tuned margin and priority rules are the most manual part of the loop, so a natural extension is to learn those mapping functions from paired manual-plan data, which would remove the remaining human step.","Beyond the paper: the released dataset could test whether larger, more standardized corpora actually improve out-of-distribution dose prediction, a hypothesis the paper motivates but does not itself evaluate."],"forward_implications":["AI dose-prediction models can be trained on a standardized corpus an order of magnitude larger than existing public benchmarks, with plans produced by one consistent pipeline.","The pipeline can re-plan legacy clinical cohorts using current treatment-planning technology, removing the bias of older, lower-quality plans from AI training data.","Dose-prediction models can be evaluated by downstream deliverability: predicted dose is converted into a deliverable plan and compared by DVH, not only by voxel error.","Researchers can generate arbitrarily many plans by varying beam configurations, prescriptions, and structures, enabling controlled studies of model generalization across treatment sites and protocols."],"supporting_citations":[{"why":"The OpenKBP public benchmark (340 plans) that the paper compares its HMM-RT dataset against as the previous largest curated corpus.","marker":"[9]"},{"why":"The published head-and-neck knowledge-based planning model used to seed initial PTV and OAR optimization objectives.","marker":"[35]"},{"why":"The auto-contouring model that produces the organ-at-risk masks in Step 1 of the pipeline.","marker":"[25]"},{"why":"The official head-and-neck dosimetric scorecard whose metrics Algorithm 1 converts into optimization objectives.","marker":"[2]"},{"why":"The lung dosimetric scorecard used to derive initial lung objectives when no lung knowledge-based planning model is available.","marker":"[3]"},{"why":"The Python interface to the treatment-planning scripting API that lets the pipeline run beam setup, optimization, and dose calculation automatically.","marker":"[4]"},{"why":"Supplies the angle-plate and beam-plate input encoding and the dose-prediction loss baseline used for the challenge.","marker":"[20]"}],"fun_headline_variants":["Automated RT planning matches manual quality in minutes","Scalable auto-planner yields 3,730 plans at clinical quality","Closed-loop automation generates clinician-grade RT plans fast","RT planning: hours to minutes, quality matched","3,730-plan RT dataset from automated pipeline rivals manual"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the assumption that the hand-tuned margin rules and the dosimetric scorecard are a valid proxy for clinical quality, so that iterative convergence to high scorecard scores means convergence to clinically good plans; the paper does not report how many plans failed the scorecard and were discarded or re-planned.","fun_headline_variants_meta":{"raw":{"variants":["Automated RT planning matches manual quality in minutes","Scalable auto-planner yields 3,730 plans at clinical quality","Closed-loop automation generates clinician-grade RT plans fast","RT planning: hours to minutes, quality matched","3,730-plan RT dataset from automated pipeline rivals manual"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000423,"raw_usage":{"total_tokens":2179,"prompt_tokens":963,"completion_tokens":1216,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":1138}},"tokens_in":579,"tokens_out":1216,"duration_ms":12901,"temperature":1.0,"reasoning_tokens":1138,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:50:28.168375+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the pipeline on a held-out set of patients from an institution with different contouring conventions and count the plans that fail the scorecard after iteration; if the failure rate is high, or if a blinded dosimetrist review judges the automated plans clinically unacceptable despite high scores, the comparability claim would fall.","supporting_citations":[{"cited_title":"Babier, B","cited_arxiv_id":null,"evidence_quote":"The OpenKBP public benchmark (340 plans) that the paper compares its HMM-RT dataset against as the previous largest curated corpus."},{"cited_title":"Magliari, R","cited_arxiv_id":null,"evidence_quote":"The published head-and-neck knowledge-based planning model used to seed initial PTV and OAR optimization objectives."},{"cited_title":"Healthineers","cited_arxiv_id":null,"evidence_quote":"The auto-contouring model that produces the organ-at-risk masks in Step 1 of the pipeline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The official head-and-neck dosimetric scorecard whose metrics Algorithm 1 converts into optimization objectives."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The lung dosimetric scorecard used to derive initial lung objectives when no lung knowledge-based planning model is available."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Python interface to the treatment-planning scripting API that lets the pipeline run beam setup, optimization, and dose calculation automatically."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the angle-plate and beam-plate input encoding and the dose-prediction loss baseline used for the challenge."}],"review_version":1}