{"id":"19c0ec38-98d0-4da6-a25e-f77dd360bcbd","arxiv_id":"2507.02975","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A simple rubric assigns Green, Yellow, or Red badges based on whether an LLM answer is directly supported by retrieved context, and the authors report coverage rates for three retrieval systems.","lead":"This paper introduces a three-question rubric and a traffic-light badge system for judging whether an LLM's biomedical answer is grounded in the retrieved evidence. The authors test it on nearly 3,000 physician questions, reporting that combining PubMed-based retrieval with a proprietary real-world evidence library yields a high-confidence badge for over 70% of queries.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The unvalidated LLM-as-a-judge is the load-bearing measurement instrument; all headline coverage rates depend on it, and the paper's own Discussion concedes model-specific bias risk.","rationale":"The reader's weakest assumption identifies exactly the right soft spot. The framework's badge aggregation logic is clear and the source comparison draws on real physician questions, but the entire quantitative contribution is mediated by one LLM prompt with no calibration. This is not an internal inconsistency; the paper is honest about the limitation, and the rubric is transparent enough that a validation study can settle it. I do not see a reason to reject the paper outright: the method is lightweight, the prompt is included, and the data come from a real clinical question bank. The conditional verdict is appropriate. Secondary concerns (the 2,972 vs 2,942 sample discrepancy, the 'all questions were submitted to three sources' sentence versus the Perplexity subset of 1,739, and the oracle interpretation of 'combined') would not by themselves change the verdict, but they should be cleaned up. The proposed human-annotation check is the minimum experiment that would upgrade the empirical claim from conditional to credible.","tokens_in":10760,"tokens_out":4711,"duration_ms":56669,"concrete_test":"Draw a stratified random sample of ~300 question-context-answer triples, covering all three sources and oversampling cases where the LLM judge assigned Yellow or Red. Two physician annotators, blinded to the badge, independently apply the three criteria from Supplemental Figure S1. Compute Cohen's kappa for each criterion and for the Green/Yellow/Red badge. Recompute the source-level rates and the 72.7% 'at least one Green' rate using the adjudicated human labels. If kappa is below 0.6 for any criterion, or if the human-labeled combined coverage differs from 72.7% by more than 5 percentage points, the headline percentages are not established without recalibrating or replacing the judge.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Every reported coverage rate (System 44.6%, Alexandria 50.1%, Perplexity 21.3%, and the combined 'at least one Green' 72.7%) is a direct output of a single LLM-as-a-judge applying the three binary criteria in Supplemental Figure S1. No human-annotation study, inter-annotator agreement statistic, judge model identifier, or error analysis is reported, so there is no evidence that the judge's True/False labels correspond to the intended semantic criteria. The manuscript itself concedes in the Discussion: 'the evaluation relies on a single LLM-as-a-judge to assess response quality, which, while efficient and interpretable, may introduce model-specific bias or inconsistency.' A systematic bias in any one criterion changes the badge distribution; for instance, a lenient 'answer_grounded_in_context' threshold would inflate Green rates across all sources, while a style-specific bias favoring Alexandria's structured RWE summaries would manufacture the 50.1% vs 44.6% gap and the 72.7% complementarity result. The central empirical claim therefore rests on an uncalibrated measurement instrument, and the limitation admitted in the paper does not remove the dependency.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces \"Answered with Evidence,\" a framework that uses a single LLM-as-a-judge to apply three binary criteria (context directly answers the question, context is related to the question, answer is grounded in the context) and assigns Green/Yellow/Red badges to LLM answers. The authors apply the framework to a sample of 2,972 (or 2,942, per the tables) physician-submitted questions, comparing Alexandria, a proprietary real-world-evidence library, with two PubMed-based systems (System and Perplexity). They report Green-badge rates of 50.1% for Alexandria, 44.6% for System, and 21.3% for Perplexity, and claim that at least one evidence source produced a Green-badged answer in 72.7% of cases, concluding that combining published and custom real-world evidence enables reliable answers to over 70% of biomedical queries.","tokens_in":10917,"tokens_out":7230,"duration_ms":82739,"significance":"If the reported rates were trustworthy, the framework would be a useful and interpretable evaluation instrument, and the complementarity finding would be a meaningful argument for hybrid retrieval systems that combine published literature with real-world evidence. The paper has several strengths: it uses real physician-submitted questions, the badge rubric is simple and transparent, the full judge prompt is provided in Supplemental Figure S1, and the discussion contains a concrete falsifiable prediction about a roughly 55% ceiling for PubMed-based systems. However, the central empirical claims are currently not established, because all headline percentages are produced by an unvalidated single LLM judge, and the combined 72.7% figure is an oracle-style upper bound rather than the performance of any achievable system. The 50.1% versus 44.6% comparison also lacks confidence intervals or significance testing. With the addition of human validation, judge-model disclosure, corrected statistics, and more careful framing of the combined figure, the contribution could become a solid evaluation framework for evidence-grounded biomedical question answering.","major_comments":[{"comment":"The three binary criteria are applied solely by an unspecified single LLM-as-a-judge, and every reported coverage rate (System 44.6%, Alexandria 50.1%, Perplexity 21.3%, and the combined 72.7%) is a direct output of this judge. No human annotation study, inter-annotator agreement statistic, judge model identifier, temperature setting, or error analysis is reported. The Discussion concedes that a single LLM judge \"may introduce model-specific bias or inconsistency,\" but this limitation is load-bearing rather than peripheral: a lenient threshold on any criterion would inflate Green rates across all sources, and a style-specific bias favoring Alexandria's structured summaries could manufacture the observed 5.5-percentage-point gap and the complementarity result. The manuscript should identify the judge model and settings, validate the judge against human expert annotations on a random sample (for example, per-criterion Cohen's kappa and a confusion matrix), and report error patterns. Without such validation, the headline numbers do not yet measure evidence grounding.","section":"Materials and Methods, Evidence Evaluation; Supplemental Figure S1"},{"comment":"The Methods state that a sample of 2,972 questions was selected, but Tables 1a, 1b, 2b, and 2d all sum to 2,942 cases, and Table 2d sums to 1,739 cases. This discrepancy must be resolved because it affects every percentage in the paper. In addition, the Results claim that \"all questions were submitted to the three evaluated evidence sources,\" which contradicts the Methods statement that only a randomly selected subset of 1,739 questions was evaluated with Perplexity. The subset definitions and denominators should be stated consistently in every table and figure.","section":"Materials and Methods, first paragraph; Tables 1a, 1b, 2b, 2d"},{"comment":"The 72.7% figure is defined as \"at least one evidence source provided a Green-badged answer\" and is therefore a union of three independent systems, achievable only with an oracle that knows in advance which source will answer correctly. The Abstract's statement that \"Combined, these sources enabled reliable answers to over 70% of biomedical queries\" and the Conclusion's \"nearly 75%\" substantially overstate what was measured; no single pipeline or even a fixed ensemble was shown to achieve this rate. The Discussion separately reports 70.2% for the System plus Alexandria pair, which is a different quantity. The paper should distinguish an oracle upper bound from an achievable system, report the achievable system-level coverage, and remove language suggesting that a deployed combined system attained 72.7%.","section":"Abstract; Results, Intersource agreement; Discussion; Conclusion"},{"comment":"The Green badge is assigned when the retrieved context directly answers the question and the answer is grounded in that context, but the rubric never verifies whether the context's claims are factually correct. Consequently, a Green badge can be assigned to an answer that is confidently wrong if the retrieved context is also wrong. The manuscript repeatedly uses \"reliable answers,\" \"high-confidence responses,\" and \"evidence-supported answers\" interchangeably with Green-badged; the Background even says the framework evaluates \"correctness,\" which the three criteria do not assess. The authors should explicitly state that the badge measures grounding in and relevance of the retrieved context, not factual correctness, and should temper the reliability language accordingly.","section":"Materials and Methods, Badge assignment; Background; Conclusion"},{"comment":"All comparisons between sources are reported as point estimates without confidence intervals or significance tests. The 50.1% versus 44.6% difference for Alexandria versus System is small relative to the uncertainty introduced by an unvalidated judge, and the agreement percentages in Tables 2a-2d also lack uncertainty measures. The authors should provide confidence intervals (for example, via the bootstrap) for each coverage rate and for the difference between systems, in addition to the requested human validation of the judge.","section":"Results, Table 1"}],"minor_comments":[{"comment":"The caption contains a typo: \"sournces\" should be \"sources.\"","section":"Results, Table 1 caption"},{"comment":"The text reports \"Yellow/Red agreement\" and \"overall agreement across all badge categories,\" but the tables do not define how Yellow and Red are collapsed in these agreement calculations. The exact three-category agreement is much lower than the reported figures (for example, Table 2c's exact diagonal is about 34.6%, not 56.3%), so the agreement metric should be defined precisely and consistently.","section":"Results, Intersource agreement; Tables 2a-2d"},{"comment":"The Background says the framework evaluates \"correctness, evidentiary grounding, and citation fidelity,\" but the Methods and rubric do not assess citation fidelity at all. Either remove this claim or add a citation-verification criterion.","section":"Background"},{"comment":"Figure 3 is referenced only once and is not described in detail; the reader cannot tell whether it reports the two-source 70.2% figure or the three-source 72.7% figure. The figure should be described and its denominator stated.","section":"Discussion, Figure 3"},{"comment":"The commercial platform is referred to both as \"System\" and \"SystemAI,\" and the reference points to system.com; this is confusing because \"System\" is also used as a generic term for the PubMed-based pipeline. Use a consistent, unambiguous name throughout.","section":"Materials and Methods, Evidence Sources"}],"recommendation":"major_revision","confidential_remarks":"The paper's comparative claim favors Alexandria, a system developed and operated by the authors' organization, and the question set comes from Atropos Health's own service. This is not by itself disqualifying, but the paper would benefit from an explicit conflict-of-interest statement and from a clear statement that the evaluation is an internal benchmark. The absence of judge validation and the oracle-union headline figure are the main technical obstacles to publication; both are fixable within the scope of a revision, so I am not recommending rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper does one genuinely useful thing — it runs thousands of real physician questions through two PubMed-RAG systems and a proprietary real-world evidence library, then scores them with a transparent three-criterion badge rubric and reports detailed agreement matrices. The coverage rates (44.6% for System, 50.1% for Alexandria, 21.3% for Perplexity, and roughly 70% for 'at least one source green') are a real empirical measurement, not something I've seen before. The rubric is simple enough to reuse, and the prompt is included in the supplement.\n\nWhere it hurts: the only measurement instrument for every one of those percentages is a single LLM-as-a-judge applying the same binary criteria. There is no human validation, no inter-annotator agreement statistic, no judge model identifier, and no error analysis. The paper admits this in the Discussion — 'the evaluation relies on a single LLM-as-a-judge... may introduce model-specific bias or inconsistency' — but an admission does not calibrate the instrument. If the judge is lenient on 'answer_grounded_in_context' or has a style preference for structured RWE summaries, the 50.1-vs-44.6 gap and the 72.7% complementarity result could be artifacts. I am not saying they are; I am saying the headline numbers do not yet measure evidence support.\n\nSecond: the 'combined' figure is a union bound, not an achievable single system. The paper mostly says 'at least one source,' which is correct, but the abstract's 'Combined, these sources enabled reliable answers to over 70%' invites readers to infer a system that does not exist. That framing needs to be tightened.\n\nThird: 'evidence-supported' does more work than the rubric supports. A green badge means the answer is grounded in whatever context was retrieved, not that the context is true or clinically valid. That is a reasonable internal-grounding check, but it is not 'founded in evidence' in the strong sense the title implies.\n\nMinor: Methods says 2,972 questions; the tables sum to 2,942. Small, but it should be reconciled. Also, the authors work for Atropos, the company behind Alexandria, and the paper has no conflict-of-interest statement; a referee will want one.\n\nWho this is for: people building or evaluating biomedical RAG systems, especially those thinking about adding real-world evidence libraries. It is worth a serious referee, but the coverage claims should not be relied on until a human-agreement study — or at least a labeled calibration subset — is added. I would not cite the coverage rates as established, but I might bring it to a reading group to discuss why LLM-as-judge validation matters.","headline":"Genuinely new empirical comparison of biomedical RAG sources, but every headline number depends on an unvalidated LLM judge and the 'combined' figure is a union bound.","tokens_in":11499,"tokens_out":4344,"would_cite":false,"duration_ms":46893,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces a three-criterion badge framework for judging whether LLM answers to biomedical questions are grounded in retrieved evidence, and reports that combining published literature with custom real-world evidence yields…","keywords":["evidence grounding","biomedical question answering","retrieval-augmented generation","LLM-as-a-judge","real-world evidence","clinical decision support","answer reliability","evidence badges"],"falsifier":"Hand-score a random sample of 200 question-context-answer triples with at least two physician reviewers using the paper's three criteria, and measure agreement with the LLM judge; if agreement falls below roughly 80% or errors are concentrated in one evidence source, the reported Green rates would not be a valid measure of evidence support.","tokens_in":10493,"feed_emoji":"🩺","tokens_out":6536,"duration_ms":73574,"temperature":0.7,"pith_summary":"The paper introduces Answered with Evidence, a rubric for deciding whether an LLM answer to a biomedical question is actually supported by the evidence it was given. It applies the rubric to 2,942 physician-submitted questions run through two published-literature retrieval systems and a custom real-world evidence library. The headline results are that the best published-literature system earns a Green badge on 44.6% of questions, the custom library on 50.1%, and at least one of the three sources on 72.7%, with no single source reaching about 55%. The paper argues this means retrieval of existing literature alone is not enough: closing the evidence gap will require generating and integrating novel real-world evidence.","feed_headline":"Combined evidence sources answer 72.7% of medical queries","feed_subtitle":"Published literature alone reaches about 45 percent; adding real-world evidence closes most of the gap.","key_machinery":"The central object is the Answered with Evidence rubric: three binary judgments—whether the retrieved context directly answers the question, whether the context is at least topically related, and whether the LLM answer is grounded only in that context—combined into a Green badge when all three are true, a Yellow badge when the context is related and the answer is grounded but the context does not directly answer, and a Red badge otherwise. An LLM-as-a-judge applies the rubric through a structured prompt that returns three booleans and a short explanation. The badges are what carry the argument: they turn an abstract quality like grounding into a countable, comparable label, which is how the paper derives its coverage and novelty statistics.","core_discovery":"The central claim is that evidence grounding of LLM answers can be measured by three binary questions—does the retrieved context directly answer the question, is it at least related, and does the answer stay strictly inside that context—and that these judgements can be collapsed into Green, Yellow, and Red badges. Using an LLM-as-a-judge to apply them to nearly three thousand real clinician and researcher questions, the paper reports that a custom library of structured real-world evidence grounds 50.1% of answers, a stronger published-literature retrieval system grounds 44.6%, and a second grounds 21.3%. The key finding is combined coverage: at least one source produces a Green answer in 72.7% of cases, and the custom library plus the stronger published-literature system alone cover 70.2%, while only 6.3% of cases are Green from all three sources. These numbers are offered as evidence that published and real-world evidence are complementary, and that systems able to generate novel evidence in real time will be needed to go beyond the single-source ceiling.","pith_inferences":["A natural extension is to validate the LLM judge against physician reviewers on a sample of triples before trusting any badge-rate headline; until then the exact percentages should be treated as provisional.","The complementarity pattern suggests a routing or ensemble design that sends each question to the source most likely to hold direct evidence could push combined coverage above the reported 72.7%.","A testable follow-up would check whether question type, such as causal versus epidemiological, predicts which source turns Green, and whether adding a third independent evidence type closes the remaining roughly 27%.","The gap between the two published-literature systems, if real, implies that swapping retrieval backends under the same answer generator would be a clean ablation to identify what drives grounding success."],"forward_implications":["If the framework and numbers are right, no single retrieval source, published or custom, directly grounds more than about half of real physician questions.","Combining published literature with custom real-world evidence raises the share of high-confidence answers to roughly 70%, so complementary evidence sources are necessary rather than optional.","Published-literature-only systems may be near a ceiling around 55% even as LLMs improve, making real-time generation of novel evidence the main lever for further gains.","The two published-literature systems differ sharply, 21.3% versus 44.6% Green, indicating that retrieval architecture and response strategy matter as much as the underlying corpus.","Yellow badges identify evidence gaps that could be converted into new real-world studies, giving the framework a prescriptive use beyond evaluation."],"supporting_citations":[{"why":"Supplies the definition of evidence-based medicine that motivates the need for grounded answers.","marker":"1"},{"why":"An existing medical LLM evaluation benchmark that does not assess evidence grounding, the gap the framework positions itself against.","marker":"14"},{"why":"Retrieval-augmented generation, the method the compared evidence-answering systems are built on.","marker":"18"},{"why":"Prior on-demand consultation service from which the physician-submitted question corpus derives.","marker":"17"},{"why":"Earlier argument for learning from patients like the one at hand, the rationale for a custom real-world evidence library.","marker":"16"},{"why":"Prior work answering real-world clinical questions with LLM retrieval, supporting the task and framing.","marker":"8"},{"why":"One of the published-literature retrieval systems whose answers are benchmarked in the study.","marker":"13"},{"why":"The second retrieval system benchmarked, configured to academic sources only.","marker":"12"}],"fun_headline_variants":["Combined sources ground 72.7% of medical LLM answers","Evidence framework: 72.7% coverage when sources combine","Custom plus PubMed evidence answers 72.7% of queries","LLM medical answers: 72.7% covered by combined evidence","Evidence combos lift LLM answers to 72.7% from 44.6%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All headline percentages depend on a single LLM judge correctly and consistently applying the three criteria to every answer, without validation against human expert ratings.","fun_headline_variants_meta":{"raw":{"variants":["Combined sources ground 72.7% of medical LLM answers","Evidence framework: 72.7% coverage when sources combine","Custom plus PubMed evidence answers 72.7% of queries","LLM medical answers: 72.7% covered by combined evidence","Evidence combos lift LLM answers to 72.7% from 44.6%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000521,"raw_usage":{"total_tokens":2518,"prompt_tokens":935,"completion_tokens":1583,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":1486}},"tokens_in":551,"tokens_out":1583,"duration_ms":13478,"temperature":1.0,"reasoning_tokens":1486,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:22:17.279242+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hand-score a random sample of 200 question-context-answer triples with at least two physician reviewers using the paper's three criteria, and measure agreement with the LLM judge; if agreement falls below roughly 80% or errors are concentrated in one evidence source, the reported Green rates would not be a valid measure of evidence support.","supporting_citations":[{"cited_title":"Using Aggregate Patient Data at the Bedside via an On-Demand Consultation Service","cited_arxiv_id":null,"evidence_quote":"Prior on-demand consultation service from which the physician-submitted question corpus derives."},{"cited_title":"It is time to learn from patients like mine","cited_arxiv_id":null,"evidence_quote":"Earlier argument for learning from patients like the one at hand, the rationale for a custom real-world evidence library."},{"cited_title":"Answering real-world clinical questions using large language model, retrieval-augmented generation, and agentic systems","cited_arxiv_id":null,"evidence_quote":"Prior work answering real-world clinical questions with LLM retrieval, supporting the task and framing."},{"cited_title":"Accessed June 30, 2025","cited_arxiv_id":null,"evidence_quote":"One of the published-literature retrieval systems whose answers are benchmarked in the study."},{"cited_title":"Perplexity AI","cited_arxiv_id":null,"evidence_quote":"The second retrieval system benchmarked, configured to academic sources only."}],"review_version":1}