{"id":"2bb06c95-7ff9-4972-932d-6f68f2075d03","arxiv_id":"2608.13476","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MARC is an open-source, YAML-configurable multi-agent framework for clinical AI that decomposes tasks into extraction, reasoning, and answer agents, with an automatic prompt-generation Decomposer module.","lead":"This paper introduces MARC, an open-source framework that splits clinical AI tasks into a fixed sequence of specialized LLM agents (extract, reason, answer) with logged intermediate outputs. It also includes a Decomposer module that auto-generates agent prompts from a plain-language description, removing the need for manual prompt engineering.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Determinism at temperature=0 is asserted but not demonstrated; reproducibility of the orchestration is the load-bearing unverified claim.","rationale":"The reader's weakest_assumption correctly identifies the determinism claim as the most load-bearing part of the central argument. The paper's central contribution is 'deterministic multi-agent orchestration,' and the entire reproducibility and stage-wise failure attribution benefits rest on it. The assumption is fragile because temperature = 0 does not automatically entail determinism across the heterogeneous backends the framework supports, and the paper provides no empirical evidence of identical outputs across runs. The additional ambiguity about the Decomposer's sampling parameters further compounds the issue: the Decomposer is an LLM call that changes the pipeline itself, so its stochasticity would undermine reproducibility even if execution agents were perfectly deterministic. This is not merely an 'outside current consensus' concern; it is an internally unverified empirical claim that the paper asserts as a guarantee. The proposed concrete test—repeated runs with identical settings and explicit control of seed and sampling parameters—would decisively settle whether the determinism claim holds. Therefore the reader's CONDITIONAL verdict remains appropriate: the framework is plausible and the code is available, but the deterministic-orchestration claim must be validated before the paper's headline assertion is accepted. The paper's own Section 4 limitation, acknowledging the lack of a comprehensive empirical benchmark, corroborates that this validation is outstanding.","tokens_in":9734,"tokens_out":8721,"duration_ms":77997,"concrete_test":"Run the full MARC pipeline, including the Decomposer if used, 20 times on a fixed set of clinical inputs (e.g., 5 questions from the paper's use cases) with identical configuration, environment, model versions, and a reported random seed. Record every intermediate output and every final verdict. Then repeat with the Decomposer's temperature explicitly set to 0 and, separately, with a documented seed and top_p setting. If any output differs across runs under identical settings, the 'fully deterministic' claim is refuted; if reproducibility requires a specific seed or backend flag, the paper must state and implement it before the claim is accepted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.3 states that all agents operate zero-shot at temperature = 0, 'ensuring fully deterministic and reproducible outputs across runs.' This is the foundation of the central claim of deterministic multi-agent orchestration. The assertion is not demonstrated and is not guaranteed by temperature = 0 alone: API-based and local inference backends can introduce run-to-run variation through non-deterministic kernels, batching, floating-point order, or default top_p sampling unless greedy decoding with a fixed seed is explicitly implemented and exposed. The paper provides no reproducibility experiment for either the Gemini API or the Ollama/MedGemma deployments. Additionally, the Decomposer module in Section 2.2 is itself an LLM call whose temperature and sampling parameters are unspecified; if the Decomposer is not temperature = 0, the generated prompts (and therefore the whole pipeline) may vary across runs, breaking the reproducibility guarantee even if execution agents are deterministic. The paper's own Section 4 limitation acknowledges the absence of a comprehensive empirical benchmark, but it still presents determinism as an established property. If run-to-run variation occurs, the claimed value as a deterministic, interpretable alternative to monolithic prompting is materially weakened.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MARC v1, an open-source multi-agent framework for clinical AI reasoning. MARC replaces monolithic LLM prompting with a sequential pipeline of role-specialized agents (extraction, reasoning, answer generation, and optional evaluation), with explicit context passing, YAML-based configuration, and logged intermediate outputs. The authors introduce a Decomposer module that generates task-specific agent prompts from a plain-language description, and support both API-based (Gemini) and local CPU-compatible (Ollama/MedGemma) deployment. The paper claims that the framework provides deterministic, reproducible, interpretable, and configurable orchestration, with stage-wise failure attribution. Three representative use cases (biomedical QA, radiology report generation, and task-adaptive pipeline construction) are described, but no experimental results are reported.","tokens_in":9930,"tokens_out":2680,"duration_ms":25042,"significance":"If the framework's design functions as claimed, MARC would be a useful open-source contribution to clinical AI infrastructure, particularly for teams needing interpretability, local deployment, and flexible agent orchestration without code modifications. The paper's strengths include a clean architecture with explicit context passing, structured verdict formatting, YAML-based configurability, and support for multiple deployment backends. However, the central functional claims—fully deterministic outputs, stage-wise failure attribution, and Decomposer effectiveness—are asserted without empirical evidence. The paper itself acknowledges the absence of a comprehensive benchmark in Section 4. As a result, the contribution is presently architectural rather than validated, and the significance depends on future evaluation.","major_comments":[{"comment":"The claim that temperature = 0 ensures 'fully deterministic and reproducible outputs across runs' is not supported. Determinism is not guaranteed by temperature alone for either the Gemini API or Ollama; run-to-run variation can arise from non-deterministic kernels, batching, floating-point ordering, or default sampling parameters (e.g., top_p) unless greedy decoding is explicitly enforced and seeds are fixed. The paper provides no reproducibility experiment (e.g., repeated runs of identical inputs) to demonstrate that outputs are actually identical across runs. Without such a test, the deterministic-orchestration claim is unverified.","section":"Section 2.3"},{"comment":"The Decomposer module is itself an LLM call, but its sampling parameters are not specified. If the Decomposer does not run at temperature = 0 with fixed random seed, the generated prompt templates (and therefore the entire downstream pipeline) may vary across runs, undermining the reproducibility guarantee even if execution agents are deterministic. Additionally, the paper does not provide any evidence that the structural validation of Decomposer outputs (variable bindings, VERDICT formatting, length restrictions) is sufficient to ensure that generated prompts are actually valid for the target task. No qualitative examples, robustness tests, or task-performance results involving Decomposer-generated pipelines are reported.","section":"Section 2.2"},{"comment":"The paper explicitly states that it 'primarily presents the framework design and representative use cases rather than a comprehensive empirical benchmark.' However, the central claims—that MARC 'enables stage-wise failure attribution' and that the Decomposer 'eliminates manual prompt engineering'—are functional claims that require empirical support. The use cases in Section 3 are described but no results, error-localization demonstrations, comparisons against single-prompt baselines, or user studies are provided. As a result, the paper does not yet substantiate its claimed benefits, and the lack of any evaluation limits the strength of the contribution.","section":"Section 4"}],"minor_comments":[{"comment":"The phrase 'entirely configurable via YAML, without code modifications' could be clarified: users still need to set up environment variables (e.g., API keys in .env), which is configuration but not code modification. Suggest rewording to avoid ambiguity.","section":"Abstract"},{"comment":"It is unclear what 'three subtasks' corresponds to in the Decomposer output for tasks that naturally require more or fewer than three reasoning stages. The paper should state whether the three-agent structure is fixed or whether the Decomposer can generate variable-length pipelines.","section":"Section 2.2"},{"comment":"The paper says MARC is 'model-agnostic,' but only Gemini and MedGemma models are described. It would be helpful to state explicitly that other models supported by LangChain/Ollama can be used with no code change, or to note any limitations on model compatibility.","section":"Section 2.6"},{"comment":"The bullet list says MARC is 'demonstrated' on three use cases, but no demonstration results are shown. Consider using 'described' or 'illustrated' to match the actual content of the manuscript.","section":"Section 3"},{"comment":"References 5 and 16 are self-citations; this is acceptable, but the authors should ensure that these citations are used for context rather than implying that prior self-authored work establishes the current framework's claims.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a framework-description paper with no empirical evaluation. The design appears coherent and the open-source release is a plus. However, the load-bearing claims (determinism, failure attribution, Decomposer validity) are unverified. I recommend major revision with a request for at least a small-scale demonstration: reproducibility runs, Decomposer-generated pipeline examples, and error-localization case studies. The paper may also need to clarify the scope of its contribution relative to existing multi-agent frameworks. These are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper is that it is a tooling contribution, not a results paper. The authors have built and released an open-source framework (MARC) that lets you define multi-agent clinical reasoning pipelines in YAML, swap models at the agent level, and run locally via Ollama or on Gemini APIs. The code is real, the writing is clear, and the authors are upfront that this is framework design with representative use cases rather than a comprehensive benchmark. That honesty counts for something.\n\nWhat is actually new: the YAML-driven orchestration layer and the Decomposer module, which takes a plain-language task description and emits role-specific prompts. The three-agent extractor-reasoner-answer pattern is already in RadFabric and other systems, so the novelty is incremental, but the configuration abstraction and the Decomposer are genuine conveniences for non-programmer clinical users. The paper also does a good job explaining the design rationale: explicit context passing, standardized verdict lines, and intermediate logging are all sensible engineering choices for interpretability.\n\nThe soft spots are concentrated in the determinism claim, and the stress-test note is right. Section 2.3 says that temperature = 0 'ensuring fully deterministic and reproducible outputs across runs.' That is not guaranteed. API backends can introduce variation through non-deterministic kernels, batching, or default sampling parameters, and the Decomposer module itself is an LLM call whose temperature and sampling settings are not specified. The paper provides no reproducibility experiment to support the central claim that MARC enables stage-wise failure attribution across runs. This is a load-bearing unverified claim, but it is fixable: run the same pipeline multiple times on both backends, report seed and sampling settings, and show that intermediate outputs are stable. The paper also never compares MARC against a well-tuned single-prompt baseline, so we do not yet know whether the added complexity buys anything beyond interpretability.\n\nCitation pattern is fine. The self-citations are to related prior work and not load-bearing. No fabricated entities or suspicious fitting.\n\nWho is this for? Groups building clinical agentic systems who want a configurable starting point. It is not ready to claim working reliability or transparency without added experiments. The authors should be encouraged to run a small reproducibility study and a baseline comparison; that would materially strengthen the paper.\n\nVerdict: deserve a serious referee. I would send it to peer review with expectation of major revision, focused on making the determinism and attribution claims credible and comparing against single-prompt prompting.","headline":"A modest but genuinely useful open-source multi-agent framework for clinical AI; the architecture is sane and honestly described, but the determinism and failure-attribution claims outrun the evidence.","tokens_in":10430,"tokens_out":1313,"would_cite":true,"duration_ms":14304,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MARC turns one opaque clinical LLM call into a traceable agent pipeline.","keywords":["multi-agent reasoning","clinical AI","LLM orchestration","prompt decomposition","YAML configuration","deterministic inference","radiology workflow","question answering"],"falsifier":"Run the identical clinical input through the full MARC pipeline several times on the same configured model backend at temperature 0 and compare the logged intermediate outputs; if any run differs, the claimed fully deterministic and reproducible orchestration does not hold for that backend.","tokens_in":9549,"feed_emoji":"🩺","tokens_out":4489,"duration_ms":38529,"temperature":0.7,"pith_summary":"The paper presents MARC, an open-source framework that replaces monolithic large-language-model prompting with a deterministic sequence of role-specialized agents for clinical reasoning tasks. It claims that splitting the work into extraction, reasoning, answer generation, and evaluation stages, with explicit context passing between stages and logged intermediate outputs, makes errors attributable to a specific pipeline step. It also introduces a Decomposer that writes agent prompts from a plain-language task description, removing manual prompt engineering. If the claims hold, clinical teams without programming expertise could build, modify, and debug AI reasoning workflows through configuration files alone.","feed_headline":"MARC splits clinical LLM prompts into traceable agent stages","feed_subtitle":"Role-specialized agents pass explicit context between stages, so every failure can be traced to its source","key_machinery":"The load-bearing mechanism is the sequential chain of role-specialized agents connected by explicit context passing: raw input goes only to the extractor, the reasoner receives input plus extracted evidence and emits a verdict line, and the final agent parses and returns that verdict. Each agent is defined declaratively in YAML with a model, prompt template, and optional retrieval-augmented generation context, so orchestration is a configuration layer rather than code. The standardized verdict line and validated Decomposer output are what make final predictions parseable and new pipelines constructible without manual prompt engineering.","core_discovery":"MARC's central claim is that clinical reasoning can be structured as a sequential pipeline of specialized agents rather than a single LLM call, and that doing so improves interpretability without sacrificing adaptability. The default pipeline has three agents: the first extracts task-relevant evidence without answering, the second performs structured reasoning and ends with a standardized verdict line, and the third returns that verdict verbatim as a clean parseable answer. Every agent runs zero-shot at temperature 0, receives the original input plus the previous agent's output through explicit variable bindings, and logs its output, so a wrong final answer can be traced back to whether extraction, reasoning, or formatting failed. The paper also claims the Decomposer can generate valid prompt templates for new tasks in one turn, validated against structural constraints, and that YAML configuration plus CPU-compatible local inference makes the framework deployable in data-restricted clinical settings.","pith_inferences":["The same sequential orchestration pattern could generalize to multimodal clinical inputs if agents are given access to image or signal representations, because roles are prompt-defined rather than architecture-defined.","A natural stress test would compare MARC's error attribution against human review: if stage-wise logs do not let a clinician locate the actual failure cause more often than monolithic outputs do, the interpretability benefit is weaker than claimed.","Because all agents run at temperature 0, MARC could double as an instrument for measuring a model backend's determinism; run-to-run variation would reflect on the backend, not the framework."],"forward_implications":["Errors in a wrong clinical answer can be assigned to a specific stage, letting teams fix the failing agent instead of rewriting the whole prompt.","Different stages of one pipeline can use different models, so teams can trade cost, latency, and domain accuracy per step.","Clinical sites that cannot send protected health information to external APIs can run the full workflow locally on CPU.","New clinical tasks can be brought up by describing them in plain language, since the Decomposer produces the agent prompts automatically."],"supporting_citations":[{"why":"Identifies the pitfalls of consolidated single-prompt LLM systems, the limitation MARC's role separation addresses.","marker":"13"},{"why":"Defines the Level 2 structured-workflow autonomy pattern that MARC's sequential agent design follows.","marker":"20"},{"why":"Provides the fixed-pipeline radiology agent system that MARC generalizes through YAML configuration.","marker":"21"},{"why":"Supplies the retrieval-augmented generation approach that MARC uses for optional per-agent knowledge injection.","marker":"22"},{"why":"Motivates multi-step reasoning and agentic orchestration in radiology, the setting MARC targets.","marker":"14,15"},{"why":"Establishes agentic AI systems in radiology as the broader direction MARC operationalizes.","marker":"16"}],"fun_headline_variants":["MARC turns one LLM prompt into a team of traceable clinical agents","Clinical AI gets a multi-agent framework that traces every mistake","MARC: traceable multi-agent clinical reasoning without prompt engineering","MARC's staged agents make clinical AI failures transparent","MARC generates agent prompts automatically from plain language"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central reproducibility claim rests on the assumption that every model backend is truly deterministic at temperature 0, which is outside the framework's control and not demonstrated in the paper.","fun_headline_variants_meta":{"raw":{"variants":["MARC turns one LLM prompt into a team of traceable clinical agents","Clinical AI gets a multi-agent framework that traces every mistake","MARC: traceable multi-agent clinical reasoning without prompt engineering","MARC's staged agents make clinical AI failures transparent","MARC generates agent prompts automatically from plain language"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000704,"raw_usage":{"total_tokens":3132,"prompt_tokens":858,"completion_tokens":2274,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":474,"completion_tokens_details":{"reasoning_tokens":2191}},"tokens_in":474,"tokens_out":2274,"duration_ms":14376,"temperature":1.0,"reasoning_tokens":2191,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:08:15.514378+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical clinical input through the full MARC pipeline several times on the same configured model backend at temperature 0 and compare the logged intermediate outputs; if any run differs, the claimed fully deterministic and reproducible orchestration does not hold for that backend.","supporting_citations":[{"cited_title":"Weakly Supervised Language Models for Automated Extraction of Critical Findings from Radiology Reports","cited_arxiv_id":null,"evidence_quote":"Identifies the pitfalls of consolidated single-prompt LLM systems, the limitation MARC's role separation addresses."},{"cited_title":"Agentic AI in Radiology","cited_arxiv_id":null,"evidence_quote":"Provides the fixed-pipeline radiology agent system that MARC generalizes through YAML configuration."}],"review_version":1}