{"id":"3288c16a-989b-44f6-8523-de648db4d328","arxiv_id":"2606.23301","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"EHR-Complex is a new interactive benchmark on MIMIC-IV with 52K tasks averaging 31.93 SQL components, where top LLMs achieve 62.3% accuracy and exhibit SQL logic, medical-code, and semantic failures.","lead":"This paper introduces EHR-Complex, a benchmark of roughly 52,000 interactive tasks on the real MIMIC-IV EHR database that require agents to execute SQL or Python code for patient- and population-level clinical queries. A smart generalist should read it because current top models reach only 62.3% exact-match accuracy and show high inconsistency, revealing practical limits in medical AI agents.","discovery_kind":"unclear","skeptic_critique":{"model":"grok-4.3","headline":"Representativeness of the 52K constructed tasks for real clinical EHR workflows remains unverified","rationale":"The reader's weakest_assumption directly identifies the load-bearing precondition for interpreting the reported accuracies and consistency drops as evidence of clinical difficulty. No more internal inconsistency or measurement flaw was apparent from the supplied abstract and claim text; the full-text placeholder does not alter this assessment.","tokens_in":1768,"tokens_out":285,"duration_ms":18102,"concrete_test":"Sample 200 tasks uniformly across the six intents; have two board-certified clinicians independently rate each on realism (1-5 scale) and typicality for their practice; compute mean score and Fleiss' kappa. If mean <4.0 or kappa <0.6, the representativeness claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim that the benchmark 'reveals the clinical difficulty' of EHR reasoning rests on the assumption that the 52K tasks (spanning six intents, avg. 31.93 SQL structural components) accurately reflect the distribution and complexity of tasks clinicians actually perform. The paper describes construction on MIMIC-IV but provides no clinician validation, inter-rater agreement, or comparison against logged real-world query distributions; without this, low model performance (62.3% exact match) could reflect benchmark artifacts rather than genuine clinical gaps.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces EHR-Complex, a benchmark of ~52K interactive tasks built on MIMIC-IV (365K patients, 31 tables) spanning six clinical intents. Each task requires agents to execute SQL or Python in a sandboxed environment, with average query complexity of 31.93 structural components. Evaluation of LLMs shows a top exact-match accuracy of 62.3%, Pass^k consistency below 50% at k=4, and three dominant failure modes (SQL logic errors, medical-code lookup failures, semantic misunderstandings) identified from >3,800 trajectories.","tokens_in":1893,"tokens_out":339,"duration_ms":7959,"significance":"If the tasks are representative of real clinical workflows, the benchmark would provide a valuable large-scale testbed exposing concrete gaps in robust EHR reasoning for agents. The scale, concrete performance numbers, and failure-mode breakdown from thousands of trajectories are strengths; the work ships direct measurements against held-out tasks rather than fitted parameters.","major_comments":[{"comment":"Abstract (paragraph on benchmark design): the central claim that results 'reveal the clinical difficulty' of EHR reasoning scenarios rests on the assumption that the 52K tasks accurately capture the complexity and distribution of practical EHR analysis. The manuscript provides no clinician validation, inter-annotator agreement metrics, or comparison against logged real-world query distributions from MIMIC-IV or clinical logs; without this, the 62.3% exact-match figure and failure-mode analysis could reflect benchmark construction artifacts rather than genuine clinical gaps.","section":"benchmark design"}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback on benchmark design and validation. We address the single major comment point-by-point below and will revise the manuscript to improve transparency.","responses":[{"response":"We agree that the absence of clinician validation, inter-annotator agreement, or direct comparison to real-world query logs is a limitation that could affect claims about representativeness. The 52K tasks were generated programmatically from six clinical intents using templates over the MIMIC-IV schema (365K patients, 31 tables) to produce queries with measured structural complexity averaging 31.93 components; the intents target common patient- and population-level analyses. However, the manuscript does not describe clinician review or log-based calibration. In revision we will (1) expand the methods section with a detailed account of intent selection and template construction, (2) report objective complexity statistics as the primary evidence of difficulty, and (3) add an explicit limitations paragraph acknowledging the lack of clinician validation and real-log comparison. These changes will qualify the central claim without altering the reported performance numbers or failure-mode analysis.","revision_made":"partial","referee_comment":"[benchmark design] Abstract (paragraph on benchmark design): the central claim that results 'reveal the clinical difficulty' of EHR reasoning scenarios rests on the assumption that the 52K tasks accurately capture the complexity and distribution of practical EHR analysis. The manuscript provides no clinician validation, inter-annotator agreement metrics, or comparison against logged real-world query distributions from MIMIC-IV or clinical logs; without this, the 62.3% exact-match figure and failure-mode analysis could reflect benchmark construction artifacts rather than genuine clinical gaps."}],"tokens_in":1399,"tokens_out":358,"duration_ms":29479,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's main contribution is a new benchmark of roughly 52K tasks built on the full MIMIC-IV database. It forces agents to run interactive SQL or Python against real tables rather than static idealized queries, with average structural complexity around 32 components per task and both patient-level and population-level intents. They report exact-match numbers from thousands of trajectories and break out three clear failure modes across the models they tested.\n\nThat scale and the shift to interactive execution are the parts that stand out as new. The failure-mode counts from over 3,800 trajectories are also useful raw data for anyone trying to improve medical agents.\n\nThe soft spot is exactly what the stress-test note flags: nothing in the abstract or description shows clinician review, inter-rater checks, or comparison against actual logged clinical queries. Without that, the 62.3% top score and the consistency drop could just reflect how the tasks were written rather than how hard real EHR work is. The paper treats the 52K tasks as representative, but the evidence for that step is missing.\n\nThis is for groups already working on clinical agents or EHR tooling. A reader who needs a harder testbed than existing static SQL benchmarks will get concrete numbers and error categories to work with. It is worth sending to peer review because the interactive scale is new and the evaluation volume is solid, even though the validation gap will need fixing before the difficulty claims land.","headline":"EHR-Complex gives a large-scale interactive benchmark on MIMIC-IV with concrete failure breakdowns, but the tasks lack any reported clinician validation so the difficulty claims are hard to trust.","tokens_in":2381,"tokens_out":370,"would_cite":false,"duration_ms":14372,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Top model reaches only 62.3 percent exact-match accuracy on interactive EHR reasoning benchmark.","keywords":["EHR reasoning","clinical agents","benchmark","interactive SQL","MIMIC-IV","medical AI","failure modes","stochastic consistency"],"falsifier":"A model achieving above 80 percent exact-match accuracy with pass^4 consistency above 70 percent across the full set of 52K tasks would indicate the benchmark scenarios are less difficult than claimed.","tokens_in":2681,"feed_emoji":"🩺","tokens_out":687,"duration_ms":25448,"temperature":0.7,"pith_summary":"The paper introduces EHR-Complex, a benchmark of roughly 52,000 tasks built on the full MIMIC-IV database of 365,000 patients and more than 500 million records. Each task requires an agent to interact with a sandbox by issuing SQL queries or Python code to answer patient-level or population-level clinical questions across six intents. The tasks reflect real SQL complexity, averaging nearly 32 structural components per query for longitudinal multi-table work. Evaluation of current models shows a ceiling of 62.3 percent exact-match accuracy and pass-to-the-fourth consistency below 50 percent for nearly all systems. Analysis of thousands of failed trajectories identifies three recurring error types: incorrect SQL logic, missed medical-code lookups, and semantic misreads of the clinical request.","feed_headline":"Top model hits 62.3% on complex EHR tasks","feed_subtitle":"52K-task benchmark on real MIMIC-IV records shows accuracy ceiling and consistency collapse for medical agents.","key_machinery":"Interactive clinical database reasoning benchmark of 52K tasks on MIMIC-IV, each requiring multi-turn SQL or Python execution against 31 tables with average structural complexity of 31.93 components per query.","core_discovery":"EHR-Complex reveals the clinical difficulty of these EHR reasoning scenarios, with the top-performing model achieving only 62.3% exact-match accuracy. Pass^k consistency drops below 50% for nearly all evaluated models at k=4, exposing broad stochastic fragility. A fine-grained analysis of more than 3,800 failed trajectories reveals three dominant failure modes: SQL logic errors, medical-code lookup failures, and semantic misunderstandings.","pith_inferences":["Targeted improvements in medical terminology handling could reduce one of the three main failure modes.","Training regimens that emphasize longitudinal multi-table joins might lower the rate of SQL logic errors.","Adding explicit verification steps inside the agent loop could improve consistency without changing base model accuracy."],"forward_implications":["Top-performing model achieves only 62.3% exact-match accuracy on the benchmark.","Pass^k consistency falls below 50% for nearly all models once k reaches 4.","Three dominant failure modes account for most errors: SQL logic mistakes, medical-code lookup failures, and semantic misunderstandings of the query.","Existing benchmarks that use static SQL on idealized data do not match the interactive, compositional demands of real EHR work."],"fun_headline_variants":["EHR-Complex caps top models at 62.3% accuracy","Pass^k drops under 50% at k=4 for EHR agents","SQL logic errors dominate EHR agent failures","Medical code lookups fail clinical agents on MIMIC","Semantic gaps limit agent performance on 52K EHR tasks"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The 52K tasks and their six clinical intents accurately capture the complexity and distribution of practical EHR analysis that clinicians and researchers actually perform.","fun_headline_variants_meta":{"raw":{"variants":["EHR-Complex caps top models at 62.3% accuracy","Pass^k drops under 50% at k=4 for EHR agents","SQL logic errors dominate EHR agent failures","Medical code lookups fail clinical agents on MIMIC","Semantic gaps limit agent performance on 52K EHR tasks"]},"model":"grok-4.3","cost_usd":0.004367,"raw_usage":{"total_tokens":2214,"prompt_tokens":719,"num_sources_used":0,"completion_tokens":79,"cost_in_usd_ticks":43674500,"prompt_tokens_details":{"text_tokens":719,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1416,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":719,"tokens_out":79,"duration_ms":10566,"temperature":1.0,"reasoning_tokens":1416,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T08:20:26.967654+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A model achieving above 80 percent exact-match accuracy with pass^4 consistency above 70 percent across the full set of 52K tasks would indicate the benchmark scenarios are less difficult than claimed.","supporting_citations":[],"review_version":1}