{"id":"433ce5ee-07d3-4ed4-ab4a-08f8e66c7402","arxiv_id":"2509.00414","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"MedSEBA is a RAG-based medical question-answering system that provides cited key arguments, per-study stance labels, and temporal consensus visualization, evaluated by a 10-person user study.","lead":"This paper presents MedSEBA, an AI system that searches PubMed, generates cited summaries of medical evidence, labels whether studies support or refute the question, and charts how consensus changes over time. A small user study of 10 participants rated the system usable, but the evaluation lacks baselines and statistical detail.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LLM-generated stance labels and summary citations are never validated against ground truth, so the system's core 'evidence-based' claim and consensus visualizations rest on unvalidated GPT-4o outputs; the user study measures subjective satisfaction, not correctness.","rationale":"The paper is a system demonstration, and its architecture is coherent: PubMed retrieval, semantic filtering, metadata enrichment, LLM-based synthesis, and visualization are all described clearly. The choice of 20 documents is grounded in the authors' prior experiments, and the use of PubMed as a source is reasonable. However, the load-bearing assumption is that GPT-4o's stance labels and synthesized claims are trustworthy enough to call the system 'evidence-based.' The reader's weakest assumption identifies exactly this gap. The user study, while positive, measures perceived usability and trustworthiness rather than factual correctness; SUS scores and Likert items cannot establish that the stance labels are right or that the cited abstracts truly support the summary claims. The paper's defense of GPT-4o over DeBERTa-v3 is not a validation, and no external benchmark or expert-annotated evaluation is provided. This is not an internal inconsistency, but it is a correctness risk: if GPT-4o systematically mislabels nuanced medical abstracts, the consensus visualization would actively mislead users while appearing credible. The proposed check—expert annotation of stance labels and citation-support on a fixed retrieval set—would directly settle whether the concern lands. Because the reader already reached a CONDITIONAL verdict on this basis, my stress-test does not change the verdict; it reinforces it.","tokens_in":8209,"tokens_out":3317,"duration_ms":44406,"concrete_test":"Build a gold-standard evaluation set: take 30–50 medical questions spanning diverse topics; freeze MedSEBA's retrieval stage and obtain its top-20 PubMed abstracts for each. Have two or three medical experts independently label each abstract as support/refute/neutral with respect to the question. Compare GPT-4o's stance labels to the expert labels via accuracy, macro-F1, and Cohen's kappa. Additionally, have experts check each bracketed reference in the generated summaries by reading the cited abstract and judging whether the summary claim is actually supported. If stance-label agreement with experts is below a pre-registered threshold (e.g., kappa < 0.6 or accuracy < 0.85) or citation-support precision is below ~0.9, the evidence-based and consensus claims are not supported by the current evaluation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"MedSEBA's central value proposition is that it provides evidence-based, traceable answers and a faithful view of research consensus. That claim depends on two LLM-produced artifacts: (1) the GPT-4o stance label assigned to each retrieved abstract, and (2) the GPT-4o summary with per-claim bracketed references. Neither is checked against any ground truth. Section 3.1 defends GPT-4o only by noting that a DeBERTa-v3 NLI model over-predicts the neutral class; this shows that one baseline is weak, not that GPT-4o labels are correct. Section 4's user study asks participants whether stance labels are 'sensible' (80% positive) and whether summaries are understandable/informative, but subjective agreement cannot establish factual accuracy. The consensus bar chart, temporal support/refute plots, and citation-based answer are all downstream of these unvalidated labels, so a systematic stance-labeling error would propagate into the system's main outputs while user satisfaction remains high. The paper itself concedes weak points in summary completeness and sentence selection (Section 4), but it never addresses whether generated references actually support the claims they are attached to. Therefore, the 'evidence-based' label is currently an unverified assertion rather than a demonstrated property of the system.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"MedSEBA is a system demonstration paper describing an interactive AI pipeline that answers medical questions by retrieving 50 PubMed papers via an expanded Boolean query, semantically narrowing to 20 via BMRetriever, and using GPT-4o to generate a structured summary with per-document stance labels (support/refute/neutral), highlighted evidence sentences, and temporal visualizations of research consensus. The paper reports a small user study (n=10, two medical experts and eight CS graduate students) with an average SUS score of 81.7 versus the 68 benchmark (claimed t-test p<0.01) and positive Likert responses on relevance, stance-label sensibility, and summary quality. The central claims are that the system provides 'evidence-based' answers grounded in trustworthy studies and that users find it usable and trustworthy.","tokens_in":8567,"tokens_out":4273,"duration_ms":53172,"significance":"If the claimed properties are substantiated, MedSEBA would be a useful prototype for medical literature synthesis and stance-aware retrieval, and the public code/repository plus demonstration video are valuable for the demo-track community. The architecture is well-motivated, combining established retrieval components (SciSpacy, BMRetriever, PubMed, iCite, Semantic Scholar) with LLM synthesis and interactive visualization. The paper is honest about some weaknesses, such as summary completeness and sentence selection in Section 4. However, the strongest claims—'evidence-based' and the faithful temporal evolution of consensus—currently rest on unvalidated GPT-4o outputs and a small, non-representative user study. The strengths are real but the evidence presented does not yet establish the central value proposition at the level claimed.","major_comments":[{"comment":"The stance labels assigned by GPT-4o are never validated against any ground truth. The only comparative evidence is that a DeBERTa-v3 NLI model over-predicts the neutral class; this shows a baseline deficiency, not that GPT-4o labels are correct. Because the stacked bar charts, temporal support/refute visualizations, and the synthesized 'consensus' view are all downstream of these labels (Section 3.3, Figure 5), an unmeasured stance-label error rate directly threatens the paper's 'evidence-based' claim. Please add an evaluation against a labeled stance/claim-verification dataset or a human-annotated sample with inter-annotator agreement, reporting per-class precision/recall for support/refute/neutral.","section":"Section 3.1 (Answer Synthesis) and Section 3.3"},{"comment":"The claim that answers are 'grounded' because each statement cites bracketed references is not checked for reference-support correctness. The prompt in Table 1 instructs the model to cite exact studies, and the user study asks whether summaries are understandable/informative, but there is no audit of whether the cited abstracts actually entail the claims they are attached to. The paper itself concedes in Section 4 that summary completeness and relevant-sentence selection are weak points. A manual audit of, say, 20 generated answers—measuring what fraction of [n] citations point to abstracts that genuinely support the adjacent sentence—is needed before 'grounded' and 'evidence-based' can be claimed.","section":"Section 3.1 and Section 4"},{"comment":"The temporal charts are presented as showing 'how the research consensus evolved through time,' but they are computed from the 20 documents selected by current-query semantic similarity, not from a representative or longitudinal sample of the literature on the topic. This selection bias means the year-by-year support/refute distribution may reflect retrieval artifacts rather than the true evolution of consensus. Please either qualify the visualization as 'selected retrieved studies' or add a more systematic sampling method to support the consensus-evolution claim.","section":"Section 3.3 (Visual Plots)"},{"comment":"The statistical reporting is insufficient for the strength of the claims. The SUS average of 81.7 versus the 68 benchmark is reported as significant with p<0.01, but no standard deviation, confidence interval, or effect size is given, and the sample is only 10 in-house colleagues. The Likert percentages are said to be significant with a binomial test at p<0.05, but it is not stated which statements individually reached significance or how the test was applied. There is also no baseline system comparison, so the results cannot distinguish the contribution of MedSEBA's features from the general usefulness of an LLM-generated summary over PubMed abstracts. At minimum, report per-item counts, the SUS distribution, and temper the wording of 'significantly better' accordingly.","section":"Section 4 (User Study)"}],"minor_comments":[{"comment":"Typo: 'how has the research on the given hypothesis has developed through time' should be 'how the research ... has developed.' Also, 'support/refute labels for each featured year' is ambiguous; clarify that the years are those of the retrieved papers, not all years in the literature.","section":"Section 3.3"},{"comment":"The sentence 'The final stage of the pipeline focuses on enriching the selected 20 papers and synthesizing their content into an evidence-based answer' uses 'evidence-based' before evidence of correctness is provided; consider 'study-grounded' or similar.","section":"Section 3.1"},{"comment":"The term 'SciSpacy' should be spelled as 'scispaCy' or 'SciSpacy' consistently; also consider citing the specific model version used.","section":"Section 3.1"},{"comment":"The Likert statements are grouped as 'three on accuracy' and 'three on quality,' but the first three are phrased about user perception, not objective accuracy; rename the grouping to avoid overstating what was measured.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is best positioned as a system demonstration, and the core architecture and interactive features are suitable for CIKM. The main risk is overclaiming in the abstract and conclusion: 'evidence-based' and 'trustworthy' are asserted before the stance-label and citation-support outputs are validated. I recommend major revision because the gaps are addressable with additional evaluation and sharper caveats; they are not fundamental to the system's design."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: MedSEBA is a well-built system demo that combines retrieval, GPT-4o synthesis, per-paper stance labels, and temporal consensus charts into one web app. The temporal view and per-key-point citation grounding are genuinely useful extensions over plain RAG. The user study (SUS 81.7 vs 68) is positive but thin: ten participants, all in-house, no baseline, no SD or CI. The bigger issue is that the stance labels and the summary references are pure GPT-4o outputs, never validated against a gold standard. The paper's defense—that a DeBERTa NLI model over-predicts neutral—shows only that one baseline is weak, not that GPT-4o is right. So the consensus charts and the 'evidence-based' claim sit on unverified labels. The stress-test note is on target.\n\nCredit where due: the system is fully implemented and public, the pipeline is clearly described, and the paper is honest about its weak points (summary completeness, sentence selection). The chart design gives users a quick view of how research evolved, which is a real plus. The choice of 20 papers is grounded in the authors' own earlier experiments; that is cited prior work, not circular reasoning.\n\nMost of the evaluation is subjective satisfaction, which fits a usability demo. The paper just should not overclaim 'evidence-based' without some objective check. A small annotation study—say 50 queries, two annotators, agreement with stance labels—would fix the main gap. For a CIKM system demonstration, this is within scope and the flaws are addressable.\n\nBottom line: send it to peer review; a good reviewer can help tighten the claims. I would cite it as a reference point for stance-aware medical Q&A, though not as a foundational result.","headline":"A neat stance-aware medical Q&A demo, but the 'evidence-based' label rests on unvalidated GPT-4o annotations and a ten-person usability study.","tokens_in":8968,"tokens_out":2300,"would_cite":true,"duration_ms":23354,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MedSEBA builds medical answers whose every key point can be traced to a retrieved PubMed study, with each study labeled as supporting, refuting, or neutral toward the question, and charts showing how the research consensus has shifted over","keywords":["medical question answering","retrieval-augmented generation","evidence synthesis","stance detection","PubMed","large language models","information retrieval","user study"],"falsifier":"Take 50 medical questions, have two medical experts independently label the stance of each of the top-20 retrieved PubMed studies, run MedSEBA's stance classifier on the same studies, and measure agreement; if the system's labels agree with the experts no better than chance, the 'evidence-based' claim is not supported.","tokens_in":8164,"feed_emoji":"🩺","tokens_out":8467,"duration_ms":88219,"temperature":0.7,"pith_summary":"MedSEBA's central claim is that trustworthiness in medical question answering can be built by grounding large-language-model summaries in a fixed set of retrieved research papers, labeling each paper's stance toward the question, and showing how that evidence has evolved over time. The system turns a user's medical question into a PubMed query, narrows the results to the 20 most relevant studies, and has GPT-4o write a structured answer with numbered references to those studies. It then shows users the balance of support and refutation among the retrieved studies and visualizes changes in that balance year by year. A ten-participant user study, including two medical experts, rated the system highly on usability and found the answers relevant, informative, and trustworthy. If the system's assumptions hold, it offers a template for health search that surfaces conflicting evidence instead of hiding it.","feed_headline":"Medical Q&A grounds each answer in source studies and shows consensus","feed_subtitle":"Every claim links back to a study, so health answers are auditable rather than black-box.","key_machinery":"The load-bearing mechanism is the multi-stage grounding pipeline: SciSpacy rewrites the user question into a Boolean PubMed query, PubMed returns 50 candidate papers, BMRetriever's biomedical embeddings rerank them to the 20 most similar, and GPT-4o writes the answer with numbered references while separately assigning each study a stance label. The final piece is sentence-level extraction, which pulls the one abstract sentence most relevant to the question, so every synthesized claim can be checked against its source. This pipeline is what carries the paper's claim that answers are evidence-based: the language model is constrained to speak from a fixed, visible set of retrieved studies.","core_discovery":"The paper introduces MedSEBA and claims that evidence-based medical answers can be produced by combining biomedical query expansion with PubMed's relevance ranking, semantic reranking to 20 papers, GPT-4o synthesis, and per-study stance classification. The generated answer is structured into key points that cite numbered studies; below the answer, each study is shown with metadata, an AI-generated stance label (supports/refutes/neutral), and the single most relevant sentence from its abstract. Three charts display the distribution of stances and how support and refutation have evolved by year, with citations plotted against time. A ten-person user study found the system usable (SUS 81.7) and","pith_inferences":["Beyond what the paper evaluates, the same pipeline could be run on a fixed, date-stamped corpus to turn MedSEBA into a monitoring tool for emerging reversals in medical consensus.","The stance labels could be stress-tested against expert annotations; if they hold up, the consensus charts become a lightweight exploratory substitute for a systematic review.","A next testable extension would compare decision quality rather than satisfaction: do lay users answer follow-up health questions more accurately after using MedSEBA than after a standard search engine?","The sentence-extraction step could be extended to generate the stance label from the extracted sentence itself, making each label auditable by showing the exact evidence behind it."],"forward_implications":["Every key point in a generated answer carries a reference to one of the 20 retrieved studies, so the reasoning is auditable from the answer down to a specific sentence in an abstract.","The stacked stance charts give users a quantitative picture of how many of the most relevant studies support, refute, or are neutral toward the claim, rather than a single yes/no answer.","The time-series charts let researchers see whether evidence for a hypothesis is accumulating, weakening, or mixed across publication years.","Metadata such as publication year and citation count are presented alongside each study, giving users visible signals about the evidence's provenance and standing."],"supporting_citations":[{"why":"Supplies the biomedical named-entity recognition and synonym expansion that turns the user question into a Boolean PubMed query.","marker":"[20]"},{"why":"Provides the biomedical sentence transformer used to rerank the 50 candidate papers down to 20 and to extract the most relevant sentence per abstract.","marker":"[39]"},{"why":"The authors' earlier comparison of context sizes justifies selecting 20 papers as the balance between coverage and noise.","marker":"[35]"},{"why":"Shows that recency and citation count help predict claim veracity, motivating the trustworthiness metadata displayed for each study.","marker":"[34]"},{"why":"Establishes verifiability as a goal for generative search, motivating clickable references and direct evidence sentences.","marker":"[18]"},{"why":"Motivates using large language models for medical question answering and positions MedSEBA's generation capability.","marker":"[26]"},{"why":"The System Usability Scale questionnaire is the instrument used to measure usability in the user study.","marker":"[17]"},{"why":"Validates the SUS benchmark of 68 for digital health apps, the comparison point for MedSEBA's 81.7 score.","marker":"[12]"},{"why":"Gives the commonly used SUS benchmark score of 68 for satisfactory performance.","marker":"[23]"}],"fun_headline_variants":["MedSEBA: AI answers that cite real studies, show consensus","Grounding AI medical answers in evolving PubMed evidence","Medical AI answers with study citations and stance labels","New tool synthesizes evidence from PubMed, tracks consensus","AI health answers you can trace back to the research"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The reliability of the whole answer rests on GPT-4o correctly judging each study's stance toward the question and accurately summarizing it; the paper does not measure that judgment against expert ground truth.","fun_headline_variants_meta":{"raw":{"variants":["MedSEBA: AI answers that cite real studies, show consensus","Grounding AI medical answers in evolving PubMed evidence","Medical AI answers with study citations and stance labels","New tool synthesizes evidence from PubMed, tracks consensus","AI health answers you can trace back to the research"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00011,"raw_usage":{"total_tokens":883,"prompt_tokens":731,"completion_tokens":152,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":475,"completion_tokens_details":{"reasoning_tokens":85}},"tokens_in":475,"tokens_out":152,"duration_ms":3247,"temperature":1.0,"reasoning_tokens":85,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:36:06.817951+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take 50 medical questions, have two medical experts independently label the stance of each of the top-20 retrieved PubMed studies, run MedSEBA's stance classifier on the same studies, and measure agreement; if the system's labels agree with the experts no better than chance, the 'evidence-based' claim is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the biomedical named-entity recognition and synonym expansion that turns the user question into a Boolean PubMed query."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that recency and citation count help predict claim veracity, motivating the trustworthiness metadata displayed for each study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes verifiability as a goal for generative search, motivating clickable references and direct evidence sentences."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates using large language models for medical question answering and positions MedSEBA's generation capability."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The System Usability Scale questionnaire is the instrument used to measure usability in the user study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Validates the SUS benchmark of 68 for digital health apps, the comparison point for MedSEBA's 81.7 score."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the commonly used SUS benchmark score of 68 for satisfactory performance."}],"review_version":1}