arxiv: 2604.16742 · v1 · submitted 2026-04-17 · 💻 cs.AI · cs.CL

Recognition: unknown

CT Open: An Open-Access, Uncontaminated, Live Platform for the Open Challenge of Clinical Trial Outcome Prediction

Jianyou Wang , Youze Zheng , Longtian Bao , Hanyuan Zhang , Qirui Zheng , Yuhan Chen , Yang Zhang , Matthew Feng

show 6 more authors

Maxim Khan Aditya K. Sehgal Christopher D. Rosin Ramamohan Paturi Umber Dube Leon Bergen

Authors on Pith no claims yet

Pith reviewed 2026-05-10 08:00 UTC · model grok-4.3

classification 💻 cs.AI cs.CL

keywords clinical trial outcome predictionprospective evaluationdecontamination pipelineopen challenge platformAI forecastingLLM web searchlive benchmarkdata leakage prevention

0 comments

The pith

CT Open creates a live platform for predicting clinical trial outcomes using only pre-public data.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents CT Open as an ongoing series of open challenges where anyone can submit forecasts for clinical trial results. Submissions are scored only after the actual outcomes become public, but the platform uses a new automated process to confirm that no results were available online at the time of each submission. This decontamination step relies on repeated AI-driven web searches to locate the first public mention of each outcome. By releasing a training set and two time-stamped test sets, the work supplies clean benchmarks for testing whether AI systems can anticipate real biomedical events. If the approach holds, it supplies a reusable template for fair evaluation of forecasting models in high-stakes domains.

Core claim

CT Open is an open-access platform that runs four clinical-trial-outcome-prediction challenges each year. Any method and any data source may be used because a fully automated pipeline repeatedly queries the web with large language models to identify the earliest public mention of each trial's result; human experts validated the pipeline's accuracy on sampled cases. The platform therefore guarantees that every evaluated prediction was made before the outcome was publicly known, and it releases an initial training set plus two time-stamped test benchmarks for Winter 2025 and Summer 2025.

What carries the argument

The decontamination pipeline, which performs iterative LLM-powered web searches to locate the earliest public mention of each trial outcome and thereby prevents leakage into the evaluation set.

If this is right

Participants may employ any prediction technique without restrictions on data sources or model types.
Four fresh challenges will be issued each year, each using trials whose outcomes are still private at submission time.
The same decontamination method can be reused to create additional uncontaminated benchmarks beyond the initial Winter and Summer 2025 sets.
Successful models on the platform could directly inform improvements in trial design and patient recruitment strategies.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same search-based decontamination idea could be applied to other forecasting domains where information timing is critical, such as economic indicators or policy outcomes.
If the platform grows, it may reduce the common problem of models inadvertently training on leaked future data.
The approach highlights the value of maintaining live, time-stamped leaderboards that automatically update once ground truth appears.
Wider adoption might encourage clinical registries to release results faster, since the pipeline already surfaces the true first-public dates.

Load-bearing premise

The iterative LLM web-search process will locate the absolute earliest public mention of every trial outcome across all possible sources.

What would settle it

An independent expert review that finds a publicly available outcome report dated earlier than the pipeline's identified date for any trial in the test benchmarks.

Figures

Figures reproduced from arXiv: 2604.16742 by Aditya K. Sehgal, Christopher D. Rosin, Hanyuan Zhang, Jianyou Wang, Leon Bergen, Longtian Bao, Matthew Feng, Maxim Khan, Qirui Zheng, Ramamohan Paturi, Umber Dube, Yang Zhang, Youze Zheng, Yuhan Chen.

**Figure 1.** Figure 1: CT Open Platform Screenshots. It has released leaderboards for Winter 2025 and Summer 2025 benchmarks. It is now accepting submissions for Summer 2026 Open. To facilitate finding a solution to this challenge, we introduce CT Open, a dynamic platform for clinical trial outcome prediction. See [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Figure 2: Agentic LLM Workflow is the only evidence available prior to the cutoff date, and, to the best of our knowledge, current LLMs are pretrained on textual data. Conservatively, the decontamination pipeline achieves an accuracy of at least 98%. As a precaution, we exclude this trial and its associated questions from the test benchmarks. 4.5 Answer Generation Pipeline Each remaining trial is associated with a s… view at source ↗

**Figure 3.** Figure 3: Prompt for the initial screening pass, used to determine whether any results exist [PITH_FULL_IMAGE:figures/full_fig_p022_3.png] view at source ↗

**Figure 4.** Figure 4: Prompt for LLM web search before the cutoff date. D Dataset Generation Prompts This section collects all prompt templates used throughout the CT Open pipeline, including those for decontamination search, trial matching, results verification, and answer generation. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_4.png] view at source ↗

**Figure 5.** Figure 5: Prompt for LLM web search after the cutoff date [PITH_FULL_IMAGE:figures/full_fig_p023_5.png] view at source ↗

**Figure 6.** Figure 6: Prompt for extracting publication dates and result summaries from URLs discovered by LLM web search [PITH_FULL_IMAGE:figures/full_fig_p023_6.png] view at source ↗

**Figure 7.** Figure 7: Prompt for rewriting trial information into search queries for the Brave Search API. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_7.png] view at source ↗

**Figure 8.** Figure 8: Prompt for Round 1 of GPT-5 verification, used to determine whether a document discusses the same clinical trial. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_8.png] view at source ↗

**Figure 9.** Figure 9: Prompt for Round 2 of GPT-5 verification, used to determine whether a document reports results for the matched trial [PITH_FULL_IMAGE:figures/full_fig_p025_9.png] view at source ↗

**Figure 10.** Figure 10: Prompt for answer verification for Endpoint/Superiority Questions. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_10.png] view at source ↗

**Figure 11.** Figure 11: Prompt for answer verification for Comparative Effect Questions. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_11.png] view at source ↗

**Figure 12.** Figure 12: Prompt for Prompt-only LLM Evaluation. F Example CT Open Questions In this section, we provide one example question for each question type (Endpoint, Superiority, Comparative Effect) we have in CT Open. See [PITH_FULL_IMAGE:figures/full_fig_p027_12.png] view at source ↗

**Figure 13.** Figure 13: Prompt for RAG Evaluation for datapoints with Matched Historical Similar Trials. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_13.png] view at source ↗

**Figure 14.** Figure 14: Prompt for RAG Evaluation for datapoints with Matched Historical Me-Too Trials. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_14.png] view at source ↗

**Figure 15.** Figure 15: Prompt for RAG Evaluation for datapoints without Matched Historical Trials. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_15.png] view at source ↗

**Figure 16.** Figure 16: Protocol used during the agent evaluation pipeline. 33 [PITH_FULL_IMAGE:figures/full_fig_p033_16.png] view at source ↗

read the original abstract

Scientists have long sought to accurately predict outcomes of real-world events before they happen. Can AI systems do so more reliably? We study this question through clinical trial outcome prediction, a high-stakes open challenge even for domain experts. We introduce CT Open, an open-access, live platform that will run four challenge every year. Anyone can submit predictions for each challenge. CT Open evaluates those submissions on trials whose outcomes were not yet public at the time of submission but were made public afterwards. Determining if a trial's outcome is public on the internet before a certain date is surprisingly difficult. Outcomes posted on official registries may lag behind by years, while the first mention may appear in obscure articles. To address this, we propose a novel, fully automated decontamination pipeline that uses iterative LLM-powered web search to identify the earliest mention of trial outcomes. We validate the pipeline's quality and accuracy by human expert's annotations. Since CT Open's pipeline ensures that every evaluated trial had no publicly reported outcome when the prediction was made, it allows participants to use any methodology and any data source. In this paper, we release a training set and two time-stamped test benchmarks, Winter 2025 and Summer 2025. We believe CT Open can serve as a central hub for advancing AI research on forecasting real-world outcomes before they occur, while also informing biomedical research and improving clinical trial design. CT Open Platform is hosted at $\href{https://ct-open.net/}{https://ct-open.net/}$

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

CT Open sets up a practical live platform and data release for clinical trial forecasting but its decontamination pipeline lacks the quantitative validation needed to fully support the no-leakage claim.

read the letter

The main thing to know is that this paper delivers a new open platform for quarterly challenges on predicting clinical trial outcomes, complete with time-stamped test sets that are meant to be free of future leakage. It also ships training data and two 2025 benchmarks right away. That resource is immediately usable for anyone working on real-world forecasting models. The decontamination pipeline that uses repeated LLM web searches to hunt for the earliest public outcome mention is the technical piece that makes the setup different from standard static benchmarks. Human expert checks on samples are mentioned as validation. This setup lets submitters bring any method or data source without the usual contamination worries, which is a clear practical win for organizing work in this area. The live aspect and the focus on trials whose results appear only after the cutoff address a genuine pain point in AI evaluation for high-stakes events. The soft spot sits in the pipeline itself. The claim that every evaluated trial had no public outcome at submission time rests on the search process catching every early mention, yet the paper gives no precision, recall, or date-error numbers for the full Winter and Summer 2025 sets. Sampled human annotations are cited, but without details on how many trials were reviewed, inter-annotator agreement, or measured false-negative rates for missed early postings, the guarantee stays partly untested. That matters because a missed prior mention would quietly add contaminated examples while false positives only shrink the set. The stress-test note correctly flags this gap. This paper is for researchers who build or evaluate forecasting systems and want a clean, ongoing benchmark they can submit to directly. Readers focused on clinical AI or trial design would also get value from the released data and the platform at ct-open.net. It deserves peer review because the contribution is a concrete new resource rather than another model tweak, even though the validation section needs concrete metrics added.

Referee Report

1 major / 2 minor

Summary. The manuscript introduces CT Open, an open-access live platform that will host four annual challenges for predicting clinical trial outcomes. It proposes a novel fully automated decontamination pipeline that employs iterative LLM-powered web searches to identify the earliest public mentions of trial outcomes, thereby ensuring that evaluated trials have no prior public results at the time of prediction. The pipeline is validated through human expert annotations, and the paper releases a training set plus two time-stamped test benchmarks (Winter 2025 and Summer 2025).

Significance. If the decontamination pipeline can be shown to reliably exclude contaminated trials, CT Open would offer a valuable, reusable benchmark resource for AI forecasting of real-world events that permits unrestricted methods and data sources. The open, recurring challenge format and public platform could accelerate progress in temporal prediction while providing secondary benefits to biomedical research and trial design. The release of concrete training and test sets is a concrete strength that supports immediate community use.

major comments (1)

The section describing the decontamination pipeline states that it is validated 'by human expert's annotations' but provides no quantitative details on sample size, precision or recall for earliest-date recovery, false-negative rate, or inter-annotator agreement. Because the central claim of guaranteed uncontaminated evaluation rests on correctly identifying that no public outcome mention existed before each challenge cutoff, the lack of these metrics leaves the pipeline's reliability unquantified and directly weakens the benchmark integrity guarantee.

minor comments (2)

Abstract: 'four challenge every year' is grammatically incorrect and should read 'four challenges every year.'
Abstract: 'human expert's annotations' should be 'human experts' annotations' to reflect plural possessive form.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their constructive feedback and recognition of CT Open's potential value as a benchmark resource. We address the single major comment below and will incorporate the requested details in the revised manuscript.

read point-by-point responses

Referee: The section describing the decontamination pipeline states that it is validated 'by human expert's annotations' but provides no quantitative details on sample size, precision or recall for earliest-date recovery, false-negative rate, or inter-annotator agreement. Because the central claim of guaranteed uncontaminated evaluation rests on correctly identifying that no public outcome mention existed before each challenge cutoff, the lack of these metrics leaves the pipeline's reliability unquantified and directly weakens the benchmark integrity guarantee.

Authors: We agree that the current description lacks the quantitative validation metrics needed to fully substantiate the decontamination pipeline's reliability. In the revised manuscript, we will expand the relevant section to report the sample size of trials reviewed by human experts, precision and recall for earliest public mention date recovery, false-negative rate for identifying pre-cutoff outcome mentions, and inter-annotator agreement statistics. These additions will directly address the concern and strengthen the evidence supporting the benchmark's uncontaminated evaluation guarantee. revision: yes

Circularity Check

0 steps flagged

No circularity: platform and pipeline are self-contained contributions

full rationale

The paper introduces a live challenge platform and an automated decontamination pipeline for identifying earliest public mentions of clinical trial outcomes. No mathematical derivations, equations, parameter fitting, or predictions are present that could reduce to inputs by construction. Validation relies on separate human expert annotations rather than self-referential steps. The central claim (uncontaminated benchmarks) rests on the described search process and external validation, not on any self-definition, fitted-input renaming, or self-citation load-bearing argument. This is a systems and data-resource paper with no derivation chain to inspect for circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the assumption that the automated search pipeline plus human validation produces truly uncontaminated test cases; no free parameters or invented physical entities are introduced.

axioms (1)

domain assumption Iterative LLM-powered web search combined with human annotation can identify the earliest public mention of trial outcomes with high reliability.
Invoked to justify the decontamination pipeline and the claim of uncontaminated evaluation.

pith-pipeline@v0.9.0 · 5628 in / 1190 out tokens · 31961 ms · 2026-05-10T08:00:09.551283+00:00 · methodology

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

DeepImagine: Learning Biomedical Reasoning via Successive Counterfactual Imagining
cs.CL 2026-04 unverdicted novelty 5.0

DeepImagine trains LLMs on counterfactual pairs from clinical trials using supervised fine-tuning and reinforcement learning to improve outcome prediction by approximating causal mechanisms.

Reference graph

Works this paper leans on

9 extracted references · 5 canonical work pages · cited by 1 Pith paper

[1]

Ezra Karger, Houtan Bastani, Chen Yueh-Han, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip E

URLhttps://arxiv.org/abs/2401.15269. Ezra Karger, Houtan Bastani, Chen Yueh-Han, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip E. Tetlock. Forecastbench: A dynamic benchmark of ai forecasting capabilities,

work page arXiv
[2]

URLhttps://arxiv.org/abs/2409.19839. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. Dynabench: Rethinking benchmar...

work page arXiv
[3]

arXiv:2104.14337 (2021), https://arxiv.org/abs/2104.14337

URLhttps://arxiv.org/abs/2104.14337. Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint arXiv:2405.17428, 2024. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, H...

work page arXiv 2024
[4]

URLhttps://arxiv.org/abs/2112.09332. OpenAI. Openai o3-mini. https://openai.com/index/openai-o3-mini/, January 2025. OpenAI release page. OpenAI. Openai gpt-5 system card, 2025. URLhttps://arxiv.org/abs/2601.03267. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa...

work page doi:10.18653/v1/2024.findings-acl.33 2025
[5]

InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing

URLhttps://arxiv.org/abs/2408.00727. Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search, 2025. URLhttps://arxiv.org/abs/2504.08066. Zhiling Yan, Dingjie Song, Zhe Fang, Yisheng Ji, Xiang Li, Quanzheng Li, an...

work page doi:10.18653/v1/ 2025
[6]

Full data feature set for tabular machine learning models . . . . . . . . . . . . . . . . . 16
[7]

Baseline Implementation Details and Hyperparameters . . . . . . . . . . . . . . . . . . . 18 C. Pipelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
[8]

Decontamination Pipeline Implementation Detail . . . . . . . . . . . . . . . . . . . . . . . . . 18
[9]

similar clinical trials

Answer Generation Pipeline Implementation Detail . . . . . . . . . . . . . . . . . . . . . . . 21 D. Dataset Generation Prompts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 E. Evaluation Prompts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...

2025