{"id":"d88fe9d6-98b3-4271-b943-3ed72210c8ee","arxiv_id":"2608.03606","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Offline-trained models, especially reward-weighted behavioral cloning, predict historical oncology trial portfolios more accurately than frontier LLM agents on a new 881-episode benchmark.","lead":"This paper builds a dataset of historical oncology drug-program decisions and trains AI models to predict the next six months of clinical trials, then compares them against frontier chatbot agents. The best trained model beats the best chatbot on identifying the right disease and trial design, especially on data after the models' training cutoff.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reward labels leak post-cutoff outcomes into training, so the post-Aug-2025 'contamination-clean' result may overstate offline models.","rationale":"Reader's weakest assumption was dataset construction fidelity. My concern is a specific, testable failure of that assumption as it applies to the strongest result: the post-Aug-2025 holdout is not temporally clean because training rewards can include outcomes after the holdout cutoff. The paper explicitly says rewards are retrospective and delayed (Section 2.4), and no reward date truncation is mentioned for temporal splits. This does not require disputing any numbers; it requires one rerun to settle. I recommend keeping the paper conditional rather than rejecting: the dataset and static comparisons are still valuable, and the leakage is fixable. Agreement is partial because the reader pointed at general label fidelity, not the reward-date truncation issue specifically.","tokens_in":15910,"tokens_out":8505,"duration_ms":86488,"concrete_test":"Retrain RW-BC on the post-Aug-2025 split after recomputing Eq. 1 with all approval and revenue events truncated at Aug 1, 2025 (a_i and rev_i set to 0 for approvals/revenue dated after the cutoff), then rerun evaluation on the same 24 windows and compare to GPT-5.4 agent. If the Indication F1 or Strict F1 gap decreases by more than half or becomes non-significant under the paired bootstrap, the claimed prospective advantage is not supported. Also report how many training examples change weight under truncation.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 2.4 Eq. 1 defines the per-trial reward as ri = 2a_i - f_i + log(1+rev_i), where a_i marks pivotal FDA approval and rev_i is attributed peak revenue. Approvals and revenue are realized after a trial launch, and the dataset is built through 2026, so for any training window with decision date before Aug-2025 the reward can incorporate approvals/revenue dated after Aug-2025. The post-Aug-2025 split (Section 2.2) holds out only decision windows; nothing in Appendix A.5 describes truncating reward events at the cutoff. RW-BC therefore up-weights pre-cutoff examples whose outcomes were observed after the test period, injecting information unavailable in a truly prospective deployment. The comparison is asymmetric: the LLM agents receive no reward labels at all. Because the headline 46.2 vs 25.0 / 14.2 vs 2.1 gap is on this 'contamination-clean' subset, this is a load-bearing threat to the central claim. It is distinct from the acknowledged retrospective-reward limitation: the issue is not confounding but temporal leakage in the training labels.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper frames oncology clinical-trial strategy as an offline decision-making problem: at each six-month window, an agent predicts the next trial portfolio of a drug program from date-gated evidence. The authors construct a dataset of 881 decision episodes across 45 programs from 31.7k public records (ClinicalTrials.gov, Drugs@FDA, SEC filings, CMS, SEER), compare four offline objectives (BC, RW-BC, LR-BC, IQL) against four frontier LLM tool agents on shared retrieval scaffolds across drug, sponsor, drug-class, and temporal splits, and report that fine-tuned offline models, especially RW-BC, outperform the agents. The headline result is on the post-August 2025 'contamination-clean' holdout, where RW-BC reaches 46.2% indication F1 and 14.2% strict F1 versus 25.0% and 2.1% for the best tool agents.","tokens_in":16142,"tokens_out":7381,"duration_ms":77988,"significance":"If the result holds, the paper provides a valuable benchmark and evidence that structured offline training on historical trial data can transfer to next-trial-portfolio prediction, complementing prior work on trial-outcome prediction. The strengths are substantial: a reproducible dataset and code release, date-gated state construction, a structured action schema, multiple held-out generalization axes, and an honest discussion of limitations (small post-cutoff sample, associational rewards, schema abstraction). However, the central comparison is compromised by a temporal-leakage issue in the reward labels used to weight training examples, which affects the 'contamination-clean' claim. Because the paper is fundamentally a benchmark paper, the absence of validation of the ground-truth construction pipeline is also a serious gap. With these addressed, the work could make a useful contribution; as written, the main quantitative claim is not yet fully supported.","major_comments":[{"comment":"The per-trial reward ri = 2ai − fi + log(1 + revi) is computed from FDA approvals and peak revenue that can be realized well after the decision window. For the post-August 2025 holdout, nothing in Appendix A.5 truncates reward events at the cutoff: training windows with decision dates before August 2025 can be weighted by outcomes observed after August 2025, injecting information that would be unavailable in a prospective deployment at that cutoff. The LLM agents receive no such outcome labels, making the comparison asymmetric. Since the headline 46.2 vs 25.0 and 14.2 vs 2.1 numbers are on the 'contamination-clean' subset, this leakage undermines the clean-holdout claim. Please either truncate reward realizations at the cutoff (or at each decision date) and retrain, or reword the claim and add an explicit caveat that the training rewards include post-cutoff information.","section":"§2.4, Eq. (1); §2.2; Appendix A.5"},{"comment":"The benchmark's validity rests on the accuracy of the approval linker and trial-date extraction, but no validation of these components is reported. No precision/recall or manual audit is given for NCT/acronym matching, no check that the sponsor-scoped fallback rules do not misattribute approvals, and no comparison against an independent source (e.g., FDA labels or company disclosures) is provided. Because every method is scored against the same ground-truth portfolios and rewards, systematic linker errors would bias all comparisons. Please add a validation subsection with, at minimum, a hand-checked sample of trials and approvals and a sensitivity analysis on the reward definition.","section":"Appendix A.2–A.4"}],"minor_comments":[{"comment":"The dagger before 'post-Aug 2025' is unexplained; either define it or remove it.","section":"Table 1"},{"comment":"The caption says 'windows-weighted means' but the weighting is not defined. Clarify whether this is an unweighted mean over windows or weighted by number of trials/portfolio size.","section":"Figure 1"},{"comment":"The statement that 'every fine-tuned model beats all four frontier agents on indication and field-match (P≈0.99–1.0)' should report the statistical test used and whether any multiple-comparison correction was applied across the eight splits.","section":"§3"},{"comment":"The R² values for state→reward (0.49) and decision→reward (0.025) are reported without details on the regression specification, features, or cross-validation. Please provide enough detail to interpret these numbers.","section":"Appendix B.3"},{"comment":"The claim that ranking is invariant for τ∈[0.50,0.75] would be better supported by a small sensitivity table rather than a one-line statement.","section":"Appendix A.10"},{"comment":"The '27k structured fields to predict' is not defined in the main text; please clarify what counts as a field (e.g., each trial field across all episodes? per trial?).","section":"Abstract"},{"comment":"LR-BC regresses onto the same rule-based reward target used by RW-BC; this is disclosed in B.3, but the main text should state this early so readers do not treat LR-BC as an independent learned-reward signal.","section":"§2.4 and Appendix A.7"}],"recommendation":"major_revision","confidential_remarks":"The dataset and release are genuinely useful, and the paper is mostly honest about its limitations. However, the temporal leakage in the reward labels is a load-bearing flaw for the headline post-August 2025 claim, and the missing validation of the ground-truth pipeline is a second concern for a benchmark paper. Both are fixable with additional experiments and analysis, so major revision seems appropriate rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the dataset and task formulation are a real contribution, the evaluation is mostly careful, and the authors are refreshingly transparent. But the headline 'contamination-clean' result is not as clean as it looks. Reward-weighted BC trains on rewards defined by Eq. 1 using FDA approvals and peak revenue, which are realized after the decision date. Nothing in the paper truncates those rewards at the decision date or at the August 2025 cutoff. So for training windows before that cutoff, the model sees outcomes from the test period — the LLM agents do not. That asymmetry is load-bearing for the 46.2 vs 25.0 headline, and it is distinct from the acknowledged retrospective-association limitation.\n\nWhat's genuinely new: a public dataset of 881 decision episodes across 45 oncology programs, a 14-field structured action schema, a date-gated retrieval scaffold, and a systematic comparison of four offline objectives against four frontier agent baselines across multiple splits. Code and data are released; splits are fixed; bootstrap CIs are reported; limitations are stated. The full-set result (n=432) showing offline models beat non-fine-tuned agents on indication F1 and field-match is more robust than the small subset and does not depend on the leakage issue.\n\nSoft spots, in proportion. First, the reward-leakage concern is real and needs an empirical answer: rerun with rewards censored at each decision date, or at least a sensitivity check on the post-cutoff subset. Second, the post-August 2025 subset has only 24 windows with wide CIs; the paper leans heavily on a paired bootstrap with p=0.017, which is suggestive, not definitive. Third, LR-BC regresses onto the same rule-based reward as RW-BC, so it is not an independent signal — the authors disclose this, so it's a ceiling rather than a flaw. Fourth, on the full set, strict F1 does not consistently favor offline models across splits, though the aggregate and field-match do.\n\nWho this is for: people building decision-support for clinical development, offline RL researchers looking for a new structured benchmark, and LLM-agent evaluators. It deserves a serious referee. I would accept it conditionally, with the reward-truncation analysis as the main requested revision. As it stands, the practical claim in the abstract — 'teach agents to plan clinical experiments' — should be read narrowly.","headline":"A genuinely useful new benchmark, but the post-cutoff 'clean' result trains on rewards that postdate the test cutoff, so the headline gap is overstated.","tokens_in":16651,"tokens_out":2321,"would_cite":true,"duration_ms":24514,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reward-weighted behavioral cloning trained on historical oncology decisions beats frontier tool-using agents at predicting next clinical-trial portfolios.","keywords":["offline reinforcement learning","clinical trial strategy","behavioral cloning","reward-weighted behavioral cloning","oncology drug development","trial portfolio prediction","LLM agents","temporal dataset"],"falsifier":"Manually reconstruct the launch portfolios for a random sample of 50 program-windows directly from ClinicalTrials.gov registration dates and FDA review documents, then recompute RW-BC's indication and strict F1 on those hand-verified windows; if the hand-built portfolios differ materially from the dataset's 1,956 launch actions, the reported ranking could be an artifact of label error rather than a real difference in planning ability.","tokens_in":15752,"feed_emoji":"💊","tokens_out":7592,"duration_ms":67026,"temperature":0.7,"pith_summary":"The paper tries to establish that clinical-trial strategy—deciding which trials a sponsor should launch next for an oncology drug—is a learnable offline decision problem, not just a knowledge-retrieval task for large language models. It assembles a temporal dataset of 881 decision episodes from 31.7k public records and uses it to fine-tune a 7B language model under four training objectives. On the contamination-clean post-August-2025 holdout, the simplest outcome-weighted imitation objective reaches 46.2% indication F1 and 14.2% strict F1, versus 25.0% and 2.1% for the best tool-using frontier agent on each metric. If this is right, retrospective public data on drug development carries a learnable strategic signal, and offline policy training could support portfolio planning rather than relying on parametric knowledge.","feed_headline":"Outcome-weighted model outplans frontier agents on trial strategy","feed_subtitle":"On a contamination-free holdout, a 7B model trained on historical decisions beats tool-using GPT, Claude, and Gemini.","key_machinery":"The load-bearing mechanism is the offline decision episode: at each six-month window, a state containing only pre-decision public evidence is mapped to a 14-field structured trial portfolio, and each launched trial receives a retrospective reward r_i = 2a_i - f_i + log(1 + rev_i) that credits FDA approvals and attributed revenue back to the window in which the pivotal trial was launched. Reward-weighted behavioral cloning converts this per-trial reward into clipped sampling weights and trains the same Qwen-2.5-7B backbone with QLoRA, so high-reward windows are oversampled. An approval linker that matches NCT identifiers and trial acronyms in FDA review PDFs is what makes this credit assignme","core_discovery":"The central discovery is that reweighting behavioral cloning by a per-trial reward built from FDA approval linkage, trial failure, and attributed revenue produces the strongest predictor of the next six-month trial portfolio, and that this predictor transfers across held-out drugs, sponsors, drug classes, and temporal windows. The effect is cleanest where pretraining contamination is impossible: on the post-August-2025 subset, RW-BC reaches 46.2% indication F1 and 14.2% strict F1 against 25.0% and 2.1% for the best tool agent on each metric, with the strict-F1 gap having a bootstrap CI excluding zero. The paper argues the improvement comes from learning a decision mapping from outcome-weight","pith_inferences":["Because the approval-linkage term dominates the reward (correlation 0.98), the method is effectively learning to imitate decisions that later became pivotal approvals; a causal counterfactual reward could push beyond this retrospective association.","The same dataset pipeline could be extended to non-oncology therapeutic areas or to phase I/II decisions, where approval-linked rewards are sparser and would need surrogate outcome signals.","The gap between frontier agents' design knowledge (decent field-match when they identify the right trial) and their decision targeting suggests the bottleneck is strategic action selection, not factual knowledge—so retrieval-heavy agent scaffolds alone are unlikely to close it.","Precise reward tuning matters less than the approval-linkage signal itself, since dropping reward terms moves training weights only slightly; future work could therefore focus on cleaning and expanding the ground-truth launch and approval linkage data rather than on more sophisticated objectives."],"forward_implications":["Fine-tuning a small open-weight model on historical decision episodes can teach it to assemble the correct indication, phase, and strategy jointly, instead of just naming the right disease the way the untrained base model does.","Offline-trained models keep most of their score in autoregressive rollout where they condition on their own previous predictions, while frontier agents degrade more sharply.","Reward-weighted behavioral cloning outperforms value-based implicit Q-learning at 881 episodes, consistent with prior offline-RL findings that simple imitation reweighted by returns can beat value-based methods when data are limited.","Naively composing offline policy with agent-retrieved evidence improved RW-BC to 59.0% indication F1 and 18.6% strict F1 on the post-cutoff subset, pointing to retrieval-augmented offline policies as a near-term direction."],"supporting_citations":[{"why":"Supplies the advantage-weighted regression objective that reward-weighted behavioral cloning directly adapts to trial portfolios.","marker":"Peng et al., 2019"},{"why":"Supplies the implicit Q-learning value-based baseline the paper compares against.","marker":"Kostrikov et al., 2021"},{"why":"Supports the paper's explanation that behavioral cloning can beat value-based offline RL at limited data scale.","marker":"Kumar et al., 2022"},{"why":"Frames offline RL as sequence modeling over logged decisions, motivating the episode-to-portfolio training setup.","marker":"Chen et al., 2021"},{"why":"Supplies the offline-trajectory-as-sequence framing used to justify predicting structured trial actions from states.","marker":"Janner et al., 2021"},{"why":"Defines the offline reinforcement learning problem setting that the paper instantiates for clinical development.","marker":"Levine et al., 2020"},{"why":"Supports reward-weighted supervised learning as a simple and effective offline objective in small-data regimes.","marker":"Emmons et al., 2022"},{"why":"Prior clinical-trial outcome prediction work that the paper positions itself against to motivate studying the upstream strategy question.","marker":"Fu et al., 2022"}],"fun_headline_variants":["Offline-trained agent outplans frontier LLMs on cancer trials","Historical data beats big AI on clinical trial strategy","7B model tops GPT, Claude, Gemini in trial planning","Reweighted behavior cloning wins trial strategy race","Outcome-weighted training beats frontier agents in trial foresight"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The whole comparison rests on the dataset correctly identifying which trials were actually launched in each six-month window and correctly crediting later FDA approvals back to the trials that supported them; if records are parsed incompletely or approvals attach to the wrong trial, every model's score is being measured against a faulty ground truth.","fun_headline_variants_meta":{"raw":{"variants":["Offline-trained agent outplans frontier LLMs on cancer trials","Historical data beats big AI on clinical trial strategy","7B model tops GPT, Claude, Gemini in trial planning","Reweighted behavior cloning wins trial strategy race","Outcome-weighted training beats frontier agents in trial foresight"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000293,"raw_usage":{"total_tokens":1558,"prompt_tokens":771,"completion_tokens":787,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":707}},"tokens_in":515,"tokens_out":787,"duration_ms":8608,"temperature":1.0,"reasoning_tokens":707,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:07:45.455187+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually reconstruct the launch portfolios for a random sample of 50 program-windows directly from ClinicalTrials.gov registration dates and FDA review documents, then recompute RW-BC's indication and strict F1 on those hand-verified windows; if the hand-built portfolios differ materially from the dataset's 1,956 launch actions, the reported ranking could be an artifact of label error rather than a real difference in planning ability.","supporting_citations":[{"cited_title":"Decision transformer: Reinforcement learning via sequence modeling","cited_arxiv_id":null,"evidence_quote":"Frames offline RL as sequence modeling over logged decisions, motivating the episode-to-portfolio training setup."},{"cited_title":"Offline reinforcement learning as one big sequence modeling problem","cited_arxiv_id":null,"evidence_quote":"Supplies the offline-trajectory-as-sequence framing used to justify predicting structured trial actions from states."},{"cited_title":"RvS : What is essential for offline RL via supervised learning? In International Conference on Learning Representations (ICLR), 2022","cited_arxiv_id":null,"evidence_quote":"Supports reward-weighted supervised learning as a simple and effective offline objective in small-data regimes."}],"review_version":1}