REVIEW 4 major objections 5 minor 11 references
AKIBoards: A Structure-Following Multiagent System for Predicting Acute Kidney Injury
T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read The paper claims that a multiagent system of large language models predicts acute kidney injury (AKI) better when the agents share a learned 'global structure' of the clinical data as a prior belief, reaching average precision of 0.195…
desk verdict Plausible proof-of-concept that SHAP-derived structure templates can guide multiagent LLMs for AKI prediction, but the attribution claim lacks a matched random-structure control and statistical grounding. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the autogenerated structure template: a ranked list of feature-value rules (for example, the lowest bin of estimated glomerular filtration rate is the strongest AKI risk indicator) produced by a feature-attribution ranking step on the validation set and serialized into a standardized clinical note. This template is the 'global structure' the agents follow as a prior belief; it carries the argument because the only deliberate difference between the structure-following and baseline conditions is its presence in the prompt. A second mechanism, smart rounds, orchestrates the agents through an implicit round and an explicit interaction round, with early stopping when the average precision gain falls below a threshold.
What would settle it
Run the same multiagent pipeline with a structure template whose feature rankings are randomly permuted, or replaced by an unrelated but equally long clinical note, keeping all other prompt text, model, and voting identical; if the permuted-template condition matches the structure-following average precision of about 0.195, the paper's claim that learned structure content drives the improvement would be refuted.
Extended reading notes
Core claim
On the paper's own terms, the central claim is that STRUC-MAS, a framework that learns a global structure from clinical data and gives it to multiple LLM agents as a textual prior, improves both classification and diagnostic reasoning in AKI prediction. Using a public critical care dataset, the authors learn a ranked structure template from feature-attribution values, serialize it into a standardized note, and have three fine-tuned agents either follow that template or follow it while also retrieving similar past notes. With balanced precision-weighted-recall-weighted voting, structure-following agents outperform their non-structure-following counterparts in average precision (0.195 vs 0.141; with retrieval, 0.194 vs 0.180). The paper also reports that explicit interaction between agents changed confidence in a pattern consistent with knowledge distillation: low-recall agents lowered overconfident misdiagnoses, while high-recall agents became more confident in true positives and false negatives.
Load-bearing premise
The load-bearing assumption is that the average-precision gain comes from the content of the learned structure template and not from the mere addition of extra structured text to the prompt; because no random or permuted structure control is run, any added template text could in principle produce the same effect.
Editorial extensions
If this is right
- Structure-following agents beat fine-tuned baselines on average precision, so the template can be added to existing fine-tuned LLM pipelines without architectural changes.
- Adding retrieval-augmented generation to structure-following does not beat structure-following alone (0.194 vs 0.195), suggesting the structure, not retrieval, drives the gain.
- Explicit agent interaction improves recall of weaker agents and increases confidence of stronger agents, implying multiagent rounds act as knowledge distillation for LLM teams.
- The early-stopping rule keeps teams at two rounds when the average precision gain is below 0.040, so the coordination cost stays low.
- The paper's conclusion generalizes as a recipe: learn a structure with traditional machine learning, then inject it as text into LLM agents.
Reading between the lines
- Editorial inference: the paper has no control condition with a random or permuted structure template, so the causal claim about structure content remains untested; a permuted-template condition would settle whether any extra instruction text produces the gain.
- Editorial inference: if the effect is about prior structure rather than prompt length, the same template should transfer across different LLM families and different clinical endpoints, which is a testable extension.
- Editorial inference: the confidence dynamics suggest a calibration story, overconfident weak agents become less confident after interaction, but the paper does not measure calibration directly; a calibration analysis would connect the result to known LLM confidence issues.
- Editorial inference: the approach may apply beyond AKI to other team-based diagnoses where organ overlaps make the underlying structure unknown, as the paper hints for oncology and cardiology.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces STRUC-MAS, a framework in which a global structure learned from data (here, a SHAP-based feature-importance template) is injected as natural-language prior instructions into the prompts of multiple LLM agents. The agents, instantiated as three open-source LLMs (Qwen 2.5, Phi-4, Llama 3.1), are fine-tuned on a MIMIC-III-derived AKI prediction task and optionally augmented with retrieval. The agents vote in rounds, with an early-stopping rule, and are evaluated on a held-out test set. The headline result is that structure-following agents outperform non-structure-following agents under balanced precision-weighted-recall-weighted voting: SF-FT AP=0.195 and SF-FT-RAG AP=0.194 versus NSF-FT AP=0.141 and NSF-FT-RAG AP=0.180. The authors interpret this as evidence that learning and following a global structure is necessary for competitive multiagent diagnostic classification and reasoning.
Significance. If the causal attribution holds, the paper would contribute a reusable recipe for injecting data-derived priors into multiagent LLM systems, with a concrete health-informatics demonstration. The strengths of the work include: the use of an established critical-care dataset (MIMIC-III) with a held-out test split; the employment of three distinct open-weight LLMs with QLoRA fine-tuning; the provision of software and multiagent records on Zenodo; and the definition of a joint classification-and-reasoning score (BCRScore). These are concrete, partially reusable assets. However, the central empirical claim — that the SHAP-derived structure content, rather than the mere presence of additional prompt text, drives the AP gain — is not supported by the experimental design as reported, because the structure-following condition is contrasted only against a no-template baseline. The absence of a permuted or placebo template control and the absence of any uncertainty quantification make the headline AP differences, several of which are small (e.g., 0.194 vs. 0.180), difficult to interpret as evidence for the framework's core hypothesis.
major comments (4)
- [§2.1, Table 1 and Table 4] The central attribution of the AP improvement to the SHAP-derived structure template is not supported by the design. The comparison SF-FT vs. NSF-FT contrasts a condition with a full natural-language template (Table 1) against a condition with no template at all. This confounds the content of the structure with the mere presence of extra instructional text, which could improve LLM calibration, output formatting, or voting behavior for reasons unrelated to the learned global structure. The conclusion in §5 that 'global structure is necessary' is load-bearing and requires a control condition with a permuted, reversed, or otherwise meaningless structure matched for length and format. Without such a control, the observed AP gap is equally consistent with the mechanism claimed and with a trivial prompt-length or instruction-presence confound.
- [Table 4 and §3] The headline AP differences are reported without confidence intervals, significance tests, or multiple seeds. For example, the SF-FT-RAG BPRV AP of 0.194 versus the NSF-FT-RAG AP of 0.180 is a small gap, and the SF-FT Agent 1 round-0 AP of 0.133 with recall 0.01 suggests substantial run-to-run and threshold sensitivity. Since the test set has n=1,967 patients and the evaluation is performed once, the stability of the BPRV AP under different random seeds, QLoRA runs, or threshold choices is unknown. I recommend reporting bootstrap confidence intervals or repeated fine-tuning runs and, at minimum, a sensitivity analysis over k (the number of SHAP features), Q (the early-stopping threshold), and the precision/recall threshold settings (0.75 and 0.25 referenced in Table 4).
- [§2.1 and Appendix 6.1] The structure template is learned with SHAP using k=10 features, with categorical binning and interaction values, but the paper does not report how sensitive the downstream AP is to the choice of k, to the SHAP version, or to the binning. Because the template is fitted on the validation set and the test evaluation is out-of-sample, this is not circular, but the free parameter k and the early-stopping threshold Q appear to be chosen without a reported sensitivity analysis. If the performance advantage of SF-FT over NSF-FT disappears for other reasonable k values, the conclusion that the structure content matters would be substantially weakened. Please report the results for at least a few values of k and Q, or justify these choices by validation-based selection.
- [§3, Appendix 6.4] The claim that 'not all models can leverage the global structure in a meaningful way' is not directly supported by the metrics reported. The paper states that SF-FT Agent 1 performed poorly in round 0 (AP 0.133, recall 0.01) but then improved after explicit interactions. This is interpreted as knowledge distillation, but no baseline shows whether an identical interaction protocol without the structure template would produce a similar improvement. A matched interaction-only control is needed to separate the effect of multiagent consultation from the effect of structure-following, especially because the early-stopping rule P-O < Q is applied to the structure-following condition and may lead to different convergence behavior across conditions.
minor comments (5)
- [Throughout] The manuscript contains numerous typographical errors and OCR artifacts, including 'es:mated', 'fiLh', 'Mul’-Agent', and 'struc-ture'. A thorough proofreading pass is needed before publication.
- [Eq. (2)] The BCRScore formula 'BCRScore = Aα + Bβ' is ambiguous in notation; A and B are metrics, while α and β are weights. Please rewrite with explicit metric names and state the allowed ranges (e.g., A, B ∈ [0,1]).
- [§2.1, RAG] The retrieval-augmented generation path is described only briefly. Please specify the retriever, the number of retrieved notes, the chunking/formatting of retrieved content, and how retrieved notes are inserted into the prompt.
- [Appendix 6.6, Tables 6–7] The confidence analysis refers to 'reported confidence levels' but does not define what value is extracted from the LLM output (e.g., a numeric score, a token probability, a verbal statement). Please define the extraction and normalization procedure.
- [Appendix 6.7, Figures 5–8] The p-value annotations in the figures are not explained; the number of cases per cell (TP, FP, FN, TN) is small for some groups, and the correction for multiple comparisons is not stated. Please report the statistical test used and the sample sizes for each comparison.
Circularity Check
No circularity: structure template is fit on validation and AP is measured on the holdout test set.
full rationale
The paper's claimed derivation chain is not circular. The SHAP-derived structure template is fit on the validation set, and the agents are evaluated on the held-out test set: 'the structures fit on the validation set, and the agents evaluated on the holdout test set.' The reported AP improvement (SF-FT AP=0.195 vs NSF-FT AP=0.141) is therefore an out-of-sample measurement that is not forced by construction; adding the template could plausibly have hurt performance. The structure template itself is a prompt-level summary of validation-set feature associations and does not encode test labels. The paper lacks a permuted-template or extra-text control, which is a design/confound concern rather than circularity, because the comparison is not equivalent to its inputs by definition. Self-citations to prior work ([8], [12], [14], [26]) supply the dataset, AKI algorithm, and software resources; none is used to define the test-set prediction or to forbid alternative explanations. The central AP result is an empirical out-of-sample comparison, so no prediction reduces to a fitted input or to a self-citation chain.
Assumptions & free parameters
free parameters (5)
- Number of top SHAP features (k) =
10
- Early stopping threshold (Q) =
0.040
- BCRScore weights (alpha, beta) =
0.50, 0.50
- Prosocial score issue weights =
0.333 each
- Precision/recall thresholds =
0.75 and 0.25
assumptions (5)
- domain assumption The MIMIC-III extraction and AKI definition from the authors' prior study are valid.
- domain assumption SHAP feature importance computed on the validation set faithfully represents the global structure of AKI risk.
- domain assumption BERTScore semantic similarity is an adequate measure of diagnostic reasoning quality.
- domain assumption Self-reported LLM confidence levels are meaningful measures of belief.
- domain assumption QLoRA fine-tuning on serialized tabular notes preserves enough clinical knowledge for the agents to reason about AKI.
Cite this review
Pith. "Pith review of AKIBoards: A Structure-Following Multiagent System for Predicting Acute Kidney Injury." pith.science (2026). https://pith.science/paper/DAZR7O2C
@misc{pith2026250420368,
author = {Pith},
title = {Pith review of: AKIBoards: A Structure-Following Multiagent System for Predicting Acute Kidney Injury},
year = {2026},
howpublished = {\url{https://pith.science/paper/DAZR7O2C}},
note = {Machine review of arXiv:2504.20368}
}
read the original abstract
Diagnostic reasoning entails a physician's local (mental) model based on an assumed or known shared perspective (global model) to explain patient observations with evidence assigned towards a clinical assessment. But in several (complex) medical situations, multiple experts work together as a team to optimize health evaluation and decision-making by leveraging different perspectives. Such consensus-driven reasoning reflects individual knowledge contributing toward a broader perspective on the patient. In this light, we introduce STRUCture-following for Multiagent Systems (STRUC-MAS), a framework automating the learning of these global models and their incorporation as prior beliefs for agents in multiagent systems (MAS) to follow. We demonstrate proof of concept with a prosocial MAS application for predicting acute kidney injuries (AKIs). In this case, we found that incorporating a global structure enabled multiple agents to achieve better performance (average precision, AP) in predicting AKI 48 hours before onset (structure-following-fine-tuned, SF-FT, AP=0.195; SF-FT-retrieval-augmented generation, SF-FT-RAG, AP=0.194) vs. baseline (non-structure-following-FT, NSF-FT, AP=0.141; NSF-FT-RAG, AP=0.180) for balanced precision-weighted-recall-weighted voting. Markedly, SF-FT agents with higher recall scores reported lower confidence levels in the initial round on true positive and false negative cases. But after explicit interactions, their confidence in their decisions increased (suggesting reinforced belief). In contrast, the SF-FT agent with the lowest recall decreased its confidence in true positive and false negative cases (suggesting a new belief). This approach suggests that learning and leveraging global structures in MAS is necessary prior to achieving competitive classification and diagnostic reasoning performance.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[11]
MIMIC-III, a freely accessible critical care database,
A. E. Johnson et al., "MIMIC-III, a freely accessible critical care database," (in eng), Sci Data, vol. 3, p. 160035, May 2016, doi: 10.1038/sdata.2016.35
-
[12]
Automated Dynamic Bayesian Networks for Predicting Acute Kidney Injury Before Onset
D. Gordon et al., "Automated Dynamic Bayesian Networks for Predicting Acute Kidney Injury Before Onset," arXiv preprint arXiv:2304.10175, 2023, doi: 10.48550/arXiv.2304.10175
work page Pith review arXiv doi:10.48550/arxiv.2304.10175 2023
-
[13]
Tabllm: Few-shot classification of tabular data with large language models,
S. Hegselmann, A. Buendia, H. Lang, M. Agrawal, X. Jiang, and D. Sontag, "Tabllm: Few-shot classification of tabular data with large language models," in International Conference on Artificial Intelligence and Statistics, 2023: PMLR, pp. 5549-5581
work page 2023
-
[15]
A unified approach to interpreting model predictions,
S. Lundberg, "A unified approach to interpreting model predictions," arXiv preprint arXiv:1705.07874,
-
[17]
A simple and effective model-based variable importance measure,
B. M. Greenwell, B. C. Boehmke, and A. J. McCarthy, "A simple and effective model-based variable importance measure," arXiv preprint arXiv:1805.04755,
-
[19]
A. Yang et al., "Qwen2. 5 technical report," arXiv preprint arXiv:2412.15115,
-
[20]
M. Abdin et al., "Phi-4 technical report," arXiv preprint arXiv:2412.08905,
-
[21]
A. Dubey et al., "The llama 3 herd of models," arXiv preprint arXiv:2407.21783,
Show all 11 references
-
[23]
Autogen: Enabling next-gen llm applications via multi-agent conversation framework,
Q. Wu et al., "Autogen: Enabling next-gen llm applications via multi-agent conversation framework," arXiv preprint arXiv:2308.08155,
-
[25]
Distilling the Knowledge in a Neural Network,
G. Hinton, "Distilling the Knowledge in a Neural Network," arXiv preprint arXiv:1503.02531,
-
[27]
Bertscore: Evaluating text generation with bert,
T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, "Bertscore: Evaluating text generation with bert," arXiv preprint arXiv:1904.09675,
1904 arXiv
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.