{"id":"524b14dd-2d37-4e8c-a023-c3807884a0cc","arxiv_id":"2504.20368","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Agents fine-tuned to follow a SHAP-derived structure template achieved higher average precision for predicting acute kidney injury 48 hours before onset (AP 0.195 vs 0.141) than agents without the structure.","lead":"This paper introduces a multiagent system where several large language models, each acting as a clinician, are given a data-derived list of important risk factors for acute kidney injury and then discuss cases before voting. In a MIMIC-III test set, the structure-following agents achieved higher average precision than agents without the structure, but the study lacks statistical error bars and a control condition.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing permuted-template control leaves the central attribution from SHAP structure to AP gain unsupported.","rationale":"The reader's weakest_assumption identifies exactly the missing random-structure control, and the manual supports that this is the central attribution gap. The paper is a proof-of-concept with a genuine holdout AP measurement, but the specific comparison SF-FT vs NSF-FT conflates the presence/absence of any structured instruction with the content of the learned SHAP structure. A matched-length permuted template is the minimal experiment that would separate these hypotheses. The reader's CONDITIONAL verdict is appropriate; my stress-test does not move it, so I mark UNCHANGED. I agree with the reader that code/data release and repeated seeds would strengthen the empirical basis, but the random-template control is the decisive missing condition for the paper's mechanistic claim.","tokens_in":7778,"tokens_out":5368,"duration_ms":57670,"concrete_test":"Run a matched-length permuted-template ablation: take the exact template from Table 1, keep the ten feature names and sentence format, but randomly shuffle the feature order and randomly flip the risk-direction phrases (e.g., 'lowest eGFR indicates decreased risk'), then fine-tune and evaluate all three agents under otherwise identical settings (same data split, QLoRA, RAG variant, BPRV voting) with at least 5 independent seeds. Compare BPRV AP against SF-FT and NSF-FT; if the permuted-template condition is not statistically distinguishable from SF-FT (e.g., overlapping bootstrap 95% CIs), the SHAP-content explanation collapses.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the content of the SHAP-derived structure template, not merely extra prompt text, drives the AP improvement (SF-FT AP=0.195 vs NSF-FT AP=0.141). As described in Section 2.1, the structure-following condition injects the natural-language template from Table 1 into the agent's note, while the non-structure-following baseline has no such template. There is no ablation with a permuted, reversed, or otherwise meaningless structure matched for length and format. Consequently, the observed gain is equally consistent with the hypothesis that any additional instruction text improves LLM calibration or output formatting, and with the hypothesis that the learned SHAP structure is the active ingredient. Because the paper concludes that 'global structure is necessary' (Section 5), the missing control is load-bearing: without it, the data do not adjudicate between the mechanism claimed and a trivial prompt-length confound. The absence of multiple runs and confidence intervals further prevents assessing whether the AP gap is stable, but the design-level gap is the more fundamental problem.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces STRUC-MAS, a framework in which a global structure learned from data (here, a SHAP-based feature-importance template) is injected as natural-language prior instructions into the prompts of multiple LLM agents. The agents, instantiated as three open-source LLMs (Qwen 2.5, Phi-4, Llama 3.1), are fine-tuned on a MIMIC-III-derived AKI prediction task and optionally augmented with retrieval. The agents vote in rounds, with an early-stopping rule, and are evaluated on a held-out test set. The headline result is that structure-following agents outperform non-structure-following agents under balanced precision-weighted-recall-weighted voting: SF-FT AP=0.195 and SF-FT-RAG AP=0.194 versus NSF-FT AP=0.141 and NSF-FT-RAG AP=0.180. The authors interpret this as evidence that learning and following a global structure is necessary for competitive multiagent diagnostic classification and reasoning.","tokens_in":7975,"tokens_out":3279,"duration_ms":37324,"significance":"If the causal attribution holds, the paper would contribute a reusable recipe for injecting data-derived priors into multiagent LLM systems, with a concrete health-informatics demonstration. The strengths of the work include: the use of an established critical-care dataset (MIMIC-III) with a held-out test split; the employment of three distinct open-weight LLMs with QLoRA fine-tuning; the provision of software and multiagent records on Zenodo; and the definition of a joint classification-and-reasoning score (BCRScore). These are concrete, partially reusable assets. However, the central empirical claim — that the SHAP-derived structure content, rather than the mere presence of additional prompt text, drives the AP gain — is not supported by the experimental design as reported, because the structure-following condition is contrasted only against a no-template baseline. The absence of a permuted or placebo template control and the absence of any uncertainty quantification make the headline AP differences, several of which are small (e.g., 0.194 vs. 0.180), difficult to interpret as evidence for the framework's core hypothesis.","major_comments":[{"comment":"The central attribution of the AP improvement to the SHAP-derived structure template is not supported by the design. The comparison SF-FT vs. NSF-FT contrasts a condition with a full natural-language template (Table 1) against a condition with no template at all. This confounds the content of the structure with the mere presence of extra instructional text, which could improve LLM calibration, output formatting, or voting behavior for reasons unrelated to the learned global structure. The conclusion in §5 that 'global structure is necessary' is load-bearing and requires a control condition with a permuted, reversed, or otherwise meaningless structure matched for length and format. Without such a control, the observed AP gap is equally consistent with the mechanism claimed and with a trivial prompt-length or instruction-presence confound.","section":"§2.1, Table 1 and Table 4"},{"comment":"The headline AP differences are reported without confidence intervals, significance tests, or multiple seeds. For example, the SF-FT-RAG BPRV AP of 0.194 versus the NSF-FT-RAG AP of 0.180 is a small gap, and the SF-FT Agent 1 round-0 AP of 0.133 with recall 0.01 suggests substantial run-to-run and threshold sensitivity. Since the test set has n=1,967 patients and the evaluation is performed once, the stability of the BPRV AP under different random seeds, QLoRA runs, or threshold choices is unknown. I recommend reporting bootstrap confidence intervals or repeated fine-tuning runs and, at minimum, a sensitivity analysis over k (the number of SHAP features), Q (the early-stopping threshold), and the precision/recall threshold settings (0.75 and 0.25 referenced in Table 4).","section":"Table 4 and §3"},{"comment":"The structure template is learned with SHAP using k=10 features, with categorical binning and interaction values, but the paper does not report how sensitive the downstream AP is to the choice of k, to the SHAP version, or to the binning. Because the template is fitted on the validation set and the test evaluation is out-of-sample, this is not circular, but the free parameter k and the early-stopping threshold Q appear to be chosen without a reported sensitivity analysis. If the performance advantage of SF-FT over NSF-FT disappears for other reasonable k values, the conclusion that the structure content matters would be substantially weakened. Please report the results for at least a few values of k and Q, or justify these choices by validation-based selection.","section":"§2.1 and Appendix 6.1"},{"comment":"The claim that 'not all models can leverage the global structure in a meaningful way' is not directly supported by the metrics reported. The paper states that SF-FT Agent 1 performed poorly in round 0 (AP 0.133, recall 0.01) but then improved after explicit interactions. This is interpreted as knowledge distillation, but no baseline shows whether an identical interaction protocol without the structure template would produce a similar improvement. A matched interaction-only control is needed to separate the effect of multiagent consultation from the effect of structure-following, especially because the early-stopping rule P-O < Q is applied to the structure-following condition and may lead to different convergence behavior across conditions.","section":"§3, Appendix 6.4"}],"minor_comments":[{"comment":"The manuscript contains numerous typographical errors and OCR artifacts, including 'es:mated', 'fiLh', 'Mul’-Agent', and 'struc-ture'. A thorough proofreading pass is needed before publication.","section":"Throughout"},{"comment":"The BCRScore formula 'BCRScore = Aα + Bβ' is ambiguous in notation; A and B are metrics, while α and β are weights. Please rewrite with explicit metric names and state the allowed ranges (e.g., A, B ∈ [0,1]).","section":"Eq. (2)"},{"comment":"The retrieval-augmented generation path is described only briefly. Please specify the retriever, the number of retrieved notes, the chunking/formatting of retrieved content, and how retrieved notes are inserted into the prompt.","section":"§2.1, RAG"},{"comment":"The confidence analysis refers to 'reported confidence levels' but does not define what value is extracted from the LLM output (e.g., a numeric score, a token probability, a verbal statement). Please define the extraction and normalization procedure.","section":"Appendix 6.6, Tables 6–7"},{"comment":"The p-value annotations in the figures are not explained; the number of cases per cell (TP, FP, FN, TN) is small for some groups, and the correction for multiple comparisons is not stated. Please report the statistical test used and the sample sizes for each comparison.","section":"Appendix 6.7, Figures 5–8"}],"recommendation":"major_revision","confidential_remarks":"The paper is a modest proof-of-concept with a clear central claim, but the missing permuted-template control is a substantive gap, not a presentational issue. The authors appear to have the infrastructure to run the needed ablation, so major revision rather than rejection seems appropriate. I would ask for the controlled ablation and uncertainty quantification before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a plausible proof of concept that a data-derived structure template can be injected into a multiagent LLM system for a clinical prediction task, but the headline claim—that the SHAP structure is the active ingredient—is not yet supported by the experiments.\n\nWhat's new: combining SHAP feature importance with a natural-language template as priors for fine-tuned LLM agents in an AutoGen-style debate, applied to AKI prediction 48 hours before onset. The evaluation on the held-out MIMIC-III test set is a genuine out-of-sample measurement, and the setup with three locally hosted LLMs (8B-32B), QLoRA fine-tuning, and an optional RAG path is sensible. The prosocial layer and \"smart rounds\" early stopping are thoughtful touches, and the confidence dynamics (Agent 1 high confidence but low recall initially, then adapting) are interesting.\n\nThe soft spot is real and load-bearing: there is no control condition with a permuted, reversed, or otherwise meaningless structure template matched for length and format. Without that, the observed AP gain (SF-FT 0.195 vs NSF-FT 0.141; SF-FT-RAG 0.194 vs NSF-FT-RAG 0.180) is equally consistent with any additional prompt text improving LLM calibration or formatting. The paper's conclusion that \"global structure is necessary\" (Section 5) overreaches. In addition, all AP comparisons are single-run, with no confidence intervals, significance tests, or multiple seeds; the 0.194 vs 0.180 gap could easily be noise. The SHAP model itself is not specified (TreeSHAP? Linear?), and the code/data are not released, which limits reproducibility.\n\nTo be fair: the structure is fit on validation and evaluated on test, so there is no circularity in the performance measurement. The self-citations to prior software releases (RAUS, STRUC-MAS, Multiagent Records) are resources, not a flaw.\n\nWho this is for: researchers working on LLM-based clinical decision support and multiagent reasoning. They will find the architecture and the confidence analysis worth reading, but they should treat the AP numbers as preliminary.\n\nRecommendation: send to peer review, but require the permuted-template control, multiple runs with intervals, and artifact release. With those changes, this could be a solid contribution. As it stands, it is a promising proof of concept with an over-claimed mechanism.","headline":"Plausible proof-of-concept that SHAP-derived structure templates can guide multiagent LLMs for AKI prediction, but the attribution claim lacks a matched random-structure control and statistical grounding.","tokens_in":8518,"tokens_out":2190,"would_cite":false,"duration_ms":21154,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a multiagent system of large language models predicts acute kidney injury (AKI) better when the agents share a learned 'global structure' of the clinical data as a prior belief, reaching average precision of 0.195…","keywords":["multiagent systems","acute kidney injury","structure learning","large language models","clinical decision support","average precision","retrieval-augmented generation","confidence calibration"],"falsifier":"Run the same multiagent pipeline with a structure template whose feature rankings are randomly permuted, or replaced by an unrelated but equally long clinical note, keeping all other prompt text, model, and voting identical; if the permuted-template condition matches the structure-following average precision of about 0.195, the paper's claim that learned structure content drives the improvement would be refuted.","tokens_in":7584,"feed_emoji":"🏥","tokens_out":5304,"duration_ms":49441,"temperature":0.7,"pith_summary":"The paper claims that a multiagent team of large language models predicts acute kidney injury (AKI) better when the agents share a learned 'global structure' of the clinical data as a prior belief. In concrete numbers, structure-following agents reach an average precision of 0.195 versus 0.141 for fine-tuned agents without the structure, and the gap persists when retrieval-augmented generation is added (0.194 versus 0.180). The authors interpret this as evidence that learning and following a global structure, not simply model scale or retrieval, is the ingredient that lifts multiagent diagnostic performance. A sympathetic reader would care because the result suggests a cheap, interpretable route to better clinical prediction: extract a feature-ranking template from tabular data and let LLM agents reason from it.","feed_headline":"Learned structure lifts multiagent AKI prediction before onset","feed_subtitle":"Sharing a data-derived structure template lifts team average precision from 0.141 to 0.195.","key_machinery":"The central object is the autogenerated structure template: a ranked list of feature-value rules (for example, the lowest bin of estimated glomerular filtration rate is the strongest AKI risk indicator) produced by a feature-attribution ranking step on the validation set and serialized into a standardized clinical note. This template is the 'global structure' the agents follow as a prior belief; it carries the argument because the only deliberate difference between the structure-following and baseline conditions is its presence in the prompt. A second mechanism, smart rounds, orchestrates the agents through an implicit round and an explicit interaction round, with early stopping when the average precision gain falls below a threshold.","core_discovery":"On the paper's own terms, the central claim is that STRUC-MAS, a framework that learns a global structure from clinical data and gives it to multiple LLM agents as a textual prior, improves both classification and diagnostic reasoning in AKI prediction. Using a public critical care dataset, the authors learn a ranked structure template from feature-attribution values, serialize it into a standardized note, and have three fine-tuned agents either follow that template or follow it while also retrieving similar past notes. With balanced precision-weighted-recall-weighted voting, structure-following agents outperform their non-structure-following counterparts in average precision (0.195 vs 0.141; with retrieval, 0.194 vs 0.180). The paper also reports that explicit interaction between agents changed confidence in a pattern consistent with knowledge distillation: low-recall agents lowered overconfident misdiagnoses, while high-recall agents became more confident in true positives and false negatives.","pith_inferences":["Editorial inference: the paper has no control condition with a random or permuted structure template, so the causal claim about structure content remains untested; a permuted-template condition would settle whether any extra instruction text produces the gain.","Editorial inference: if the effect is about prior structure rather than prompt length, the same template should transfer across different LLM families and different clinical endpoints, which is a testable extension.","Editorial inference: the confidence dynamics suggest a calibration story, overconfident weak agents become less confident after interaction, but the paper does not measure calibration directly; a calibration analysis would connect the result to known LLM confidence issues.","Editorial inference: the approach may apply beyond AKI to other team-based diagnoses where organ overlaps make the underlying structure unknown, as the paper hints for oncology and cardiology."],"forward_implications":["Structure-following agents beat fine-tuned baselines on average precision, so the template can be added to existing fine-tuned LLM pipelines without architectural changes.","Adding retrieval-augmented generation to structure-following does not beat structure-following alone (0.194 vs 0.195), suggesting the structure, not retrieval, drives the gain.","Explicit agent interaction improves recall of weaker agents and increases confidence of stronger agents, implying multiagent rounds act as knowledge distillation for LLM teams.","The early-stopping rule keeps teams at two rounds when the average precision gain is below 0.040, so the coordination cost stays low.","The paper's conclusion generalizes as a recipe: learn a structure with traditional machine learning, then inject it as text into LLM agents."],"supporting_citations":[{"why":"Supplies the STRUC-MAS architecture that AKIBoards instantiates.","marker":"[8]"},{"why":"Provides the critical care database from which the AKI dataset is extracted.","marker":"[11]"},{"why":"Defines the AKI algorithm and the data split used for training, validation, and testing.","marker":"[12]"},{"why":"Supplies the rank-learning module that produces the structure template.","marker":"[14]"},{"why":"Is the feature-attribution method whose ranked output becomes the structure template.","marker":"[15]"},{"why":"Enables retrieval-augmented generation in the semi-structure-following path.","marker":"[18]"},{"why":"Is the quantization method used to fine-tune the language models.","marker":"[22]"},{"why":"Supplies the semantic-similarity metric used to score agent reasoning.","marker":"[27]"}],"fun_headline_variants":["Structure-following agents lift AKI prediction from 0.141 to 0.195","Multiagent system with shared structure boosts AKI forecast","Learned global structure improves team AKI detection","Shared structure helps agents predict AKI 48h earlier","Structural priors sharpen multiagent AKI diagnosis"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the average-precision gain comes from the content of the learned structure template and not from the mere addition of extra structured text to the prompt; because no random or permuted structure control is run, any added template text could in principle produce the same effect.","fun_headline_variants_meta":{"raw":{"variants":["Structure-following agents lift AKI prediction from 0.141 to 0.195","Multiagent system with shared structure boosts AKI forecast","Learned global structure improves team AKI detection","Shared structure helps agents predict AKI 48h earlier","Structural priors sharpen multiagent AKI diagnosis"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1372,"prompt_tokens":1077,"completion_tokens":295,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":693,"completion_tokens_details":{"reasoning_tokens":211}},"tokens_in":693,"tokens_out":295,"duration_ms":2949,"temperature":1.0,"reasoning_tokens":211,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:30:48.179857+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same multiagent pipeline with a structure template whose feature rankings are randomly permuted, or replaced by an unrelated but equally long clinical note, keeping all other prompt text, model, and voting identical; if the permuted-template condition matches the structure-following average precision of about 0.195, the paper's claim that learned structure content drives the improvement would be refuted.","supporting_citations":[{"cited_title":"Automated Dynamic Bayesian Networks for Predicting Acute Kidney Injury Before Onset","cited_arxiv_id":"2304.10175","evidence_quote":"Defines the AKI algorithm and the data split used for training, validation, and testing."}],"review_version":1}