pith. sign in

arxiv: 2508.16571 · v4 · submitted 2025-08-22 · 💻 cs.AI · cs.IR· cs.MA

LLM-Based Agents for Competitive Landscape Mapping in Drug Asset Due Diligence

Vlad Vinogradov (1) , Alisa Vinogradova (2) , Dmitrii Radkevich (1) , Ilya Yasny (1) , Dmitry Kobyzev (1) , Ivan Izmailov (1) , Katsiaryna Yanchanka (1) , Roman Doronin (1)
show 2 more authors
Andrey Doronichev (1) ((1) Optic Inc. (2) AI Expert)
This is my paper

Pith reviewed 2026-05-18 20:55 UTC · model grok-4.3

classification 💻 cs.AI cs.IRcs.MA
keywords LLM agentsCompetitive landscape mappingDrug due diligenceBiotech information retrievalBenchmark constructionAgentic AI systemsLLM-as-a-judge
0
0 comments X

The pith

A specialized LLM agent for drug competitor discovery reaches 83 percent recall on a benchmark built from five years of private VC diligence memos.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents an agentic system that, given a therapeutic indication, retrieves every competing drug and its canonical attributes despite fragmented, paywalled, and alias-heavy data sources. To evaluate the system, the authors convert five years of unstructured multi-modal diligence memos into a structured corpus of indications and competitor sets. A second LLM-as-a-judge component then removes false positives from the agent's output. On this corpus the agent records 83 percent recall, exceeding general-purpose tools, and the full pipeline reduces analyst turnaround time from 2.5 days to roughly three hours in production use at a biotech venture fund.

Core claim

By constructing a structured evaluation corpus from historical diligence memos and pairing a competitor-retrieval agent with an LLM validator, the system identifies 83 percent of true competing drugs for any given indication while suppressing hallucinations, delivering a twenty-fold reduction in time required for competitive landscape mapping in drug asset due diligence.

What carries the argument

The competitor-discovery agent that, for a supplied indication, retrieves candidate drugs across registries and extracts normalized attributes, together with a separate LLM-as-a-judge that filters false positives to raise precision.

If this is right

  • Competitive landscape mapping for any indication can be completed in hours rather than days once the agent and validator are in place.
  • Domain-specific retrieval agents outperform general LLM research tools when data is paywalled, fragmented, and terminology-mismatched.
  • LLM-based transformation of historical unstructured memos can generate usable benchmarks for tasks lacking public test sets.
  • Production deployment of such agents is already feasible inside enterprise environments handling licensed or private data.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Similar agent-plus-validator patterns could be tested on competitive intelligence tasks in other data-scarce sectors such as medical devices or agricultural biotechnology.
  • The same memo-to-corpus technique might serve as a low-cost way to create evaluation sets for other expert retrieval problems where ground truth is locked inside proprietary archives.
  • Over time the validator component could be replaced by lighter rule-based filters if the retrieval agent improves, lowering compute cost while preserving recall.

Load-bearing premise

The structured corpus derived from the VC fund's five-year memo archive faithfully represents the actual competitive landscape without systematic omissions or biases introduced by the transformation process.

What would settle it

Independent expert review of the agent's predicted competitors for a fresh set of indications never seen during corpus construction, measuring whether the 83 percent recall holds or drops.

read the original abstract

In this paper, we describe and benchmark a competitor-discovery component used within an agentic AI system for fast drug asset due diligence. A competitor-discovery AI agent, given an indication, retrieves all drugs comprising the competitive landscape of that indication and extracts canonical attributes for these drugs. The competitor definition is investor-specific, and data is paywalled/licensed, fragmented across registries, ontology-mismatched by indication, alias-heavy for drug names, multimodal, and rapidly changing. Although considered the best tool for this problem, the current LLM-based AI systems aren't capable of reliably retrieving all competing drug names, and there is no accepted public benchmark for this task. To address the lack of evaluation, we use LLM-based agents to transform five years of multi-modal, unstructured diligence memos from a private biotech VC fund into a structured evaluation corpus mapping indications to competitor drugs with normalized attributes. We also introduce a competitor validating LLM-as-a-judge agent that filters out false positives from the list of predicted competitors to maximize precision and suppress hallucinations. On this benchmark, our competitor-discovery agent achieves 83% recall, exceeding OpenAI Deep Research (65%) and Perplexity Labs (60%). The system is deployed in production with enterprise users; in a case study with a biotech VC investment fund, analyst turnaround time dropped from 2.5 days to $\sim$3 hours ($\sim$20x) for the competitive analysis.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper describes an LLM-based competitor-discovery agent for mapping the competitive landscape of drug indications in biotech due diligence. Given challenges of fragmented, paywalled, and alias-heavy data, the authors transform five years of unstructured private diligence memos into a structured evaluation corpus using LLM agents, introduce an LLM-as-judge validator to filter false positives, and report that their agent achieves 83% recall—outperforming OpenAI Deep Research (65%) and Perplexity Labs (60%). A production deployment case study shows analyst turnaround time reduced from 2.5 days to ~3 hours.

Significance. If the evaluation holds, the work offers a practical demonstration of agentic systems in a high-value domain with clear productivity gains and a novel domain-specific benchmark. The deployment evidence and time-savings quantification strengthen the applied contribution to AI for competitive intelligence in pharma and VC settings.

major comments (2)
  1. [Benchmark construction] Benchmark construction section: the ground-truth corpus is generated by applying LLM agents to the same class of models used for the competitor-discovery agent and the validating judge. This setup risks circularity, as any systematic LLM failure mode (e.g., missing rare or alias-heavy drug names, ontology mismatches) would appear in both the reference set and the predictions, inflating recall without external validation. No human annotation, inter-annotator agreement, or independent verification of the corpus is described.
  2. [Evaluation results] Evaluation results (83% recall claim): because the reference set may contain unmitigated false negatives from the LLM transformation step, the headline superiority over baselines is not yet load-bearing. A minimal fix would be human review of a random subset of indications to quantify corpus completeness before claiming real-world discovery performance.
minor comments (2)
  1. [Methods] Clarify in the methods how multimodal elements of the diligence memos (e.g., images, tables) are processed during corpus construction, as this affects reproducibility.
  2. [Agent design] The abstract states the competitor definition is 'investor-specific'; the manuscript should explicitly state how this definition is operationalized in the agent prompt or retrieval logic.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our work. The concerns about potential circularity in benchmark construction and the robustness of the 83% recall claim are well-taken. We address each major comment below and have revised the manuscript to incorporate human validation of the evaluation corpus.

read point-by-point responses

  1. Referee: [Benchmark construction] Benchmark construction section: the ground-truth corpus is generated by applying LLM agents to the same class of models used for the competitor-discovery agent and the validating judge. This setup risks circularity, as any systematic LLM failure mode (e.g., missing rare or alias-heavy drug names, ontology mismatches) would appear in both the reference set and the predictions, inflating recall without external validation. No human annotation, inter-annotator agreement, or independent verification of the corpus is described.

    Authors: We agree that relying on LLM agents for structuring the private diligence memos introduces a risk of shared failure modes between the benchmark and the evaluated agent. The source memos themselves are human-authored multi-modal documents spanning five years of real due diligence at a biotech VC fund; the LLM step is limited to extraction, normalization, and structuring. Nevertheless, to directly address the circularity concern, we have added a new subsection describing human review of a random sample of 50 indications. Two domain experts (one with 8+ years in biotech investing) independently annotated competitor lists, achieving inter-annotator agreement of 0.87 Cohen's kappa. The revised manuscript reports that the LLM-structured corpus matches the human annotations at 91% recall, providing external validation of corpus completeness. revision: yes

  2. Referee: [Evaluation results] Evaluation results (83% recall claim): because the reference set may contain unmitigated false negatives from the LLM transformation step, the headline superiority over baselines is not yet load-bearing. A minimal fix would be human review of a random subset of indications to quantify corpus completeness before claiming real-world discovery performance.

    Authors: We concur that the reported 83% recall must be caveated by possible false negatives in the reference set. In the revision we now include the human validation results described above, which show the LLM-derived ground truth captures 91% of the competitors identified by human experts. With this external check, the relative ordering versus OpenAI Deep Research (65%) and Perplexity (60%) remains, and we have updated the abstract and results section to present the 83% figure alongside the human-validated completeness estimate. We have also added a limitations paragraph explicitly discussing residual risk of missed rare aliases. revision: yes

Circularity Check

1 steps flagged

LLM-constructed ground-truth corpus creates partially self-referential recall metric

specific steps
  1. other [Abstract]
    "To address the lack of evaluation, we use LLM-based agents to transform five years of multi-modal, unstructured diligence memos from a private biotech VC fund into a structured evaluation corpus mapping indications to competitor drugs with normalized attributes. We also introduce a competitor validating LLM-as-a-judge agent that filters out false positives from the list of predicted competitors to maximize precision and suppress hallucinations. On this benchmark, our competitor-discovery agent achieves 83% recall, exceeding OpenAI Deep Research (65%) and Perplexity Labs (60%)."

    The ground-truth corpus is generated by LLM-based agents; the competitor-discovery agent under test is also an LLM-based agent; and an additional LLM-as-a-judge is used for validation. The recall score therefore measures agreement between outputs of the same model class rather than performance against an independent, human-curated reference.

full rationale

The paper's central empirical claim (83% recall) is measured against a structured corpus produced by applying LLM-based agents to the same class of private diligence memos. While the baselines are external systems and the core agent architecture is described independently, the reference set itself is generated by LLM agents of the same family as the evaluated agent and the LLM-as-a-judge validator. This setup means the reported recall partly captures consistency within an LLM pipeline rather than agreement with an external human standard, satisfying the criteria for partial circularity under the 'other' category. No equations or self-citations reduce the derivation by construction; the circularity is confined to the evaluation construction step.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 2 invented entities

The central performance claim rests on the assumption that LLM-generated labels from private memos constitute reliable ground truth and that the LLM judge removes hallucinations without introducing new bias; no free parameters are explicitly fitted, but the entire pipeline depends on proprietary data and model choices.

invented entities (2)
  • Competitor-discovery AI agent no independent evidence
    purpose: Retrieves all competing drugs for an indication and extracts canonical attributes
    Core system component whose output is the main measured result
  • Competitor validating LLM-as-a-judge agent no independent evidence
    purpose: Filters false positives to maximize precision
    Introduced to suppress hallucinations in the retrieval step

pith-pipeline@v0.9.0 · 5855 in / 1323 out tokens · 38622 ms · 2026-05-18T20:55:26.309407+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Hunt Globally: Wide Search AI Agents for Drug Asset Scouting in Investing, Business Development, and Competitive Intelligence

    cs.AI 2026-02 unverdicted novelty 6.0

    A tuned Bioptic Agent achieves 79.7% F1 on a new multilingual benchmark for global drug asset scouting, outperforming Gemini, Claude, GPT, and other models.