{"id":"bb73dc13-1417-400e-813a-e1ab5cc2d7de","arxiv_id":"2608.07688","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A retrieval-grounded multi-agent LLM system for ISO 27001 evidence review receives mostly positive but mixed ratings from practicing auditors on simulated organizations.","lead":"IntelliAudit is a multi-agent system that retrieves evidence from a company's documents and spreadsheets and produces recommendations on whether ISO 27001 audit controls are met. It was tested with practicing auditors on four simulated organizations, and the results suggest such systems can help prepare for audits but should not make final certification decisions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark assigns each control case a seeded compliant/semi/non-compliant condition, but the paper never reports agreement between IntelliAudit's status labels and those seeds, leaving the RQ2 evidence-sufficiency claim supported only by small-n subjective ratings.","rationale":"The central claim is that a retrieval-grounded multi-agent system can support evidence review, not replace auditors. The paper is honest, well-structured, and the human-feedback results are valuable evidence of calibration. However, the strongest claim's word 'evidence-grounded reasoning' is exactly what the evaluation never directly tests: the benchmark already contains labels the authors created for each control instance, and those labels are a far more objective reference than Likert ratings. Reporting agreement with seeds is cheap, requires no new data collection, and would settle whether the system's status labels track the intended evidence condition. The current reliance on small panels of auditors reviewing synthetic corpora leaves the possibility that the system produces fluent, plausible-sounding explanations while often reaching the wrong conclusion; the seed labels would disambiguate this. The reader's weakest assumption is closely related but slightly less precise: the issue is not only the absence of a public benchmark, but the non-use of the benchmark's own internal ground truth. This does not require rejecting the paper; it requires conditional acceptance with the seed-agreement analysis added.","tokens_in":12579,"tokens_out":5061,"duration_ms":51271,"concrete_test":"Add a confusion-matrix analysis for all 48 control cases comparing IntelliAudit's final status and Auditor-only status against the seeded evidence condition, mapping compliant to COMPLIANT, semi-compliant to PARTIAL, non-compliant to NON_COMPLIANT, and treating INSUFFICIENT_EVIDENCE as abstention; report overall agreement and per-condition recall. Also report the same agreement restricted to cases where Group A rated Fulfill >= 4. If final-status agreement with seeds is below roughly 75% or if any seeded non-compliant case receives COMPLIANT in the final output, the RQ2 conclusion needs substantial qualification; if agreement is high, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing condition for the central claim is that IntelliAudit's status labels are evidence-grounded. The paper's own benchmark contains an internal ground truth: 'Each control instance is assigned one of three seeded evidence conditions—compliant, semi-compliant, or non-compliant' (Benchmark and Evaluation). Across 48 control cases (12 assessment units x 4 organizations) these seeds fix what a correct system should conclude, yet no table reports precision, recall, or confusion between final (or Auditor-only) statuses and the seed condition. The stated rationale for human evaluation—'Because no public ground-truth dataset exists for ISO 27001 evidence sufficiency'—addresses external reproducibility, not the internal labels the authors already built. This omission matters because RQ2 is explicitly 'the factual accuracy and sufficiency of the cited evidence in fulfilling the assigned status,' and its only evidence is Group A Likert means: Factual 3.75 (SD 1.04), Fulfill 3.20 (SD 1.32), from n=12-17 raters per control. LLM explanations can be fluent and moderately rated even when the conclusion is wrong. The asymmetry of the workflow (Defender only challenges adverse findings; Judge only on disagreement/feedback) makes false-positive COMPLIANT labels especially likely to go uncorrected in the automated path, and indeed human feedback moved 9 of 15 COMPLIANT initial findings to stricter labels—but the paper never quantifies how often initial or final labels match the seeded conditions. Without that agreement measure, the central 'evidence-grounded reasoning' claim is not actually demonstrated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents IntelliAudit, a retrieval-grounded multi-agent system for IT audit evidence evaluation. Given an ISO/IEC 27001 control and an organizational evidence corpus, the Auditor produces an initial evidence-grounded recommendation, a Defender searches for mitigating evidence when the finding is adverse, and a Judge adjudicates disagreements or human feedback; the final output is an auditor-facing recommendation with cited evidence, rationale, missing-evidence analysis, and remediation guidance. The system is evaluated on an author-constructed benchmark covering 14 controls across four simulated organizations, using Likert-scale ratings from practicing auditors (Group A) and cybersecurity-informed audit-readiness users (Group B), plus a small blinded forced-choice comparison and an analysis of human-feedback re-adjudications. The paper reports that the system supports control interpretation and artifact-level evidence review, that sufficiency judgments are harder than interpretation, that spreadsheet-heavy cases expose completeness errors, and that human feedback often corrects overly permissive recommendations. The claims are modest in scope, and the paper explicitly frames the system as a decision-support tool rather than an autonomous certification system.","tokens_in":12857,"tokens_out":4217,"duration_ms":42431,"significance":"If the evaluation supported the claims, IntelliAudit would be a useful contribution to an under-studied area: LLM support for heterogeneous-evidence IT audit review. The paper's strengths include a clear task formulation with traceability constraints, a sensible selective-escalation workflow, a substantively realistic simulated benchmark, and unusual honesty about failures such as Control 5.9's population-level spreadsheet error and the calibration problems in Controls 5.24 and 6.4. The authors also ship prompts and configuration files, which aids reproducibility. However, the evidence quality is limited: the panels are small (n=12-17 per control for Group A), the benchmark is author-constructed with no external ground truth, the multi-agent and feedback analyses are based on small, purposively selected subsets, and no statistical tests or inter-rater reliability are reported. The central claims are modest enough that these limitations do not require rejection, but they do require either additional analysis or explicit re-scoping of what the evaluation can establish.","major_comments":[{"comment":"The benchmark assigns each control instance a seeded evidence condition ('compliant,' 'semi-compliant,' or 'non-compliant'), which is an internal ground truth for the substantive status, yet the paper never reports agreement between IntelliAudit's final (or Auditor-only) status labels and these seeds. The statement 'Because no public ground-truth dataset exists for ISO 27001 evidence sufficiency, we use human evaluation as the primary reference' addresses external reproducibility, not the internal labels the authors already built. As a result, RQ2 ('the factual accuracy and sufficiency of the cited evidence in fulfilling the assigned status') is supported only by Group A Likert means (Factual 3.75, SD 1.04; Fulfill 3.20, SD 1.32) from small panels, and no precision, recall, or confusion analysis is reported. A simple table of final status labels by seeded condition, including INSUFFICIENT_EVIDENCE abstentions, would directly test the load-bearing claim that final recommendations are evidence-grounded. I request this analysis, or an explicit statement that the seeds are not intended as ground truth, with justification.","section":"Benchmark and Evaluation, Benchmark"},{"comment":"The evidence for the multi-agent workflow's benefit is thin and selected by the activation condition. The Judge was activated in 5 of 48 cases; among these, the blinded forced choice (3 reviewers, 15 judgments) yielded 9 preferences for Judge-final output, concentrated in status-changing cases (7/9), while the two status-preserving cases favored the Auditor-only output (4/6). With five purposively selected cases and no statistical test, this cannot support a general claim that multi-agent review improves quality. The paper's own caveat ('not evidence of general multi-agent superiority') is appropriate, but the Introduction still presents the 'selectively escalated multi-agent workflow' as a contribution. Please either report all 48 cases in the ablation (e.g., forcing the Judge to run on all cases) or explicitly re-scope the claim to a feasibility demonstration.","section":"Results, Effect of Multi-Agent Review"},{"comment":"The claim that human feedback 'corrects overly permissive recommendations' is partly a design property rather than a measured system capability. Because the workflow routes any human feedback h_j to the Judge for re-evaluation (Human Feedback section), it is expected that supplying a correction often changes the label; the observed 15/27 status changes and 9/15 moves away from COMPLIANT do not distinguish between genuine model correction and the mechanical effect of conditioning the Judge on the human's stated conclusion. What is missing is a control condition (e.g., re-running the Judge without feedback, or with feedback phrased as a no-change instruction) or an analysis of how often the Judge resists incorrect feedback. I recommend tempering the calibration claim or adding such a comparison.","section":"Results, Human Feedback as Audit Calibration"}],"minor_comments":[{"comment":"The sentence reporting the mean usefulness rating contains a doubled closing parenthesis: 'across four organizations))'; please fix.","section":"Results, RQ4"},{"comment":"The benchmark is described as covering 14 ISO 27001 controls but also as having 12 assessment units (with 5.25–5.27 consolidated); the relationship between controls and assessment units should be stated explicitly in the main text, not only implied by Table 3.","section":"Benchmark and Evaluation, Benchmark"},{"comment":"The report says Group A ratings have n=12–17 per control, but it is unclear whether n counts distinct auditors, distinct rating instances, or both; please clarify the sampling unit and report inter-rater agreement (e.g., Krippendorff's alpha) if multiple ratings per control are available.","section":"Benchmark and Evaluation, Human Evaluation"},{"comment":"The implementation names 'Claude Sonnet 4.6' and 'LangGraph' without version details or citations; since these are central to reproducibility, please add references or state that they are proprietary/commercial components.","section":"Methodology, Implementation"},{"comment":"The text says 44 of 48 final statuses (91.7%) matched the Auditor-agent-only label and then notes that one case produced no output; please state the denominator explicitly for the match rate (e.g., 44/47 output-producing cases) so the reader can interpret the 91.7% figure.","section":"Results, Effect of Multi-Agent Review"}],"recommendation":"major_revision","confidential_remarks":"The paper's honest reporting of failures is a real strength, and the benchmark plus configuration archive is a useful community resource. The essential blocking issue is that the benchmark's internal seeded conditions are not used as a correctness check; reporting a confusion-style analysis of status labels against seeds would materially raise the evidentiary value of RQ2. The multi-agent and feedback analyses are also small and selected, so the contribution claims should either be supported by fuller ablations or scoped down. The paper is within scope for cs.AI and I see no citation or novelty disclosure concern beyond the general difficulty of assessing a system built on proprietary components."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one. It is a serious systems paper: IntelliAudit is a retrieval-grounded multi-agent pipeline for ISO 27001 evidence review, and it ships a new benchmark of four simulated organizations, 14 controls, with seeded compliant/semi/non-compliant conditions. The claim is appropriately modest — decision support, not certification — and the workflow (Auditor always, Defender only on adverse findings, Judge only on disagreement or human feedback) is described cleanly. The authors also report their own failures honestly: spreadsheet completeness errors, retrieval scoping issues, calibration problems. That is a good sign.\n\nWhat is genuinely new is the selective escalation design and the benchmark. Prior LLM audit work is mostly financial or narrow; this is the first attempt I know at heterogeneous-evidence sufficiency judgment for IT controls, with human auditor evaluation. The related work is fine, citations look appropriate, and there is no overclaiming.\n\nThe soft spots are mostly empirical. The biggest one: the benchmark has an internal ground truth — every control instance is seeded compliant, semi-compliant, or non-compliant — and the paper never reports how often IntelliAudit's final status matches that seed. That is not a fatal flaw, but it is a wasted check. RQ2 claims evidence-grounded reasoning, and the evidence given is Group A Likert means from 12–17 raters per control. Fluent explanations can get decent ratings even when the conclusion is wrong. A confusion table against the seeds would have been cheap and would have made the RQ2 claim much stronger. Without it, the central claim rests entirely on small-n subjective ratings.\n\nOther soft spots in proportion: the multi-agent ablation covers just five Judge-activation cases, selected precisely because they are contested, and the paper itself reads that correctly. The feedback calibration result — 15 revisions, 13 stricter — is interesting but again small. No statistical tests anywhere; with n in the teens that is probably acceptable, but should be disclosed clearly. Group B measures usefulness, not correctness, which the paper states plainly. That is good discipline.\n\nBottom line: the architecture is sensible, the benchmark is a real contribution, and the limitations are stated honestly. The missing seed-agreement table is the one thing I would push the authors to add, plus a public repo for the benchmark. It deserves a serious referee.","headline":"Honest systems paper with a genuinely new benchmark, but the evaluation misses an easy internal check that would have strengthened RQ2.","tokens_in":13439,"tokens_out":1688,"would_cite":true,"duration_ms":17695,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"IntelliAudit claims that a retrieval-grounded, selectively escalated multi-agent LLM system can support IT audit evidence evaluation, with sufficiency judgments remaining the point where human oversight is required.","keywords":["IT audit","ISO/IEC 27001","large language models","multi-agent systems","retrieval-augmented generation","evidence sufficiency","human-in-the-loop","audit readiness"],"falsifier":"Run IntelliAudit on a benchmark of real or realistically constructed ISO 27001 evidence bundles whose sufficiency status has been independently determined by completed professional audits, then count, over all control instances, the fraction of COMPLIANT recommendations that later receive an adverse audit finding; if that false-compliance rate approaches the base rate of non-compliance, the system's sufficiency judgments add no information.","tokens_in":12366,"feed_emoji":"🧾","tokens_out":9909,"duration_ms":83286,"temperature":0.7,"pith_summary":"IntelliAudit sets out to show that a retrieval-grounded, selectively escalated multi-agent LLM system can assist IT auditors in judging whether organizational evidence satisfies semantic security and compliance controls, using ISO/IEC 27001 as the test case. The paper argues that the task should be framed as an explainable evidence-grounded recommendation, not autonomous certification, and that the system's output should be an auditor-facing package with cited evidence, rationale, missing-evidence analysis, and remediation guidance. Across four simulated organizations and 14 controls, practicing auditors rated the system positively for control interpretation (mean 3.92) and factual evidence evaluation (mean 3.75), while evidence fulfillment scored lower (mean 3.20); audit-preparation users rated usefulness at 3.84. If the central claim is right, AI can take on labor-intensive evidence review and audit preparation, but the final sufficiency judgment must remain a human decision.","feed_headline":"LLM audit assistant: strong on evidence, needs human on sufficiency","feed_subtitle":"Retrieval-grounded LLM agents pass auditor review on ISO 27001, but sufficiency judgments need humans.","key_machinery":"The central mechanism is a selectively escalated multi-agent workflow. The Auditor always produces the first evidence-grounded assessment; the Defender is invoked only for adverse findings (PARTIAL or NON_COMPLIANT) to search for overlooked mitigating evidence; the Judge is invoked only when the agents disagree or when a human auditor supplies feedback, and it is the only agent allowed to return INSUFFICIENT_EVIDENCE, an epistemic abstention distinct from PARTIAL. Retrieval grounding is enforced by a hybrid search over semantic, document-level, and spreadsheet-row indexes, and synthesis is restricted to the retrieved and cited evidence bundle, with every substantive claim required to be supported by a cited artifact. The output is a recommendation package (status, cited evidence, rationale, missing-evidence analysis, follow-up actions) that remains provisional until a human auditor accepts or challenges it.","core_discovery":"The paper's central claim is that IT audit evidence evaluation can be operationalized as a retrieval-grounded recommendation task in which an Auditor agent produces a provisional status from cited evidence, a Defender searches for overlooked mitigating evidence when the initial finding is adverse, and a Judge resolves disagreements and may issue INSUFFICIENT_EVIDENCE when the record cannot support a reliable substantive status. The authors report that this workflow supports control interpretation beyond surface text and produces factually accurate evidence evaluation, but that converting evidence into calibrated sufficiency judgments remains harder, with human feedback acting as a conservative calibration layer: across 27 case reviews, 15 re-adjudications changed status, 13 moved away from a more permissive label, and none ended as COMPLIANT. The authors therefore present IntelliAudit as decision support for audit preparation and evidence review rather than as an autonomous certification system.","pith_inferences":["A testable extension is to build a public ground-truth benchmark of ISO 27001 evidence sufficiency from completed real audits; the paper explicitly notes that no such dataset exists, so without it the sufficiency results rest on subjective ratings.","The selective-escalation pattern likely transfers to other evidence-sufficiency tasks such as regulatory filing review or due-diligence documentation, but the paper evaluates only ISO 27001, so this is an extrapolation.","The Control 5.9 spreadsheet failure suggests a concrete improvement the paper does not implement: verify population coverage of structured evidence (for example, all asset categories in an inventory) before allowing LLM synthesis to generalize.","If sufficiency calibration improves, the human role in audits may shift from reviewing evidence to adjudicating system-recommended gaps; the paper's traceable human-feedback loop is a step toward that division of labor."],"forward_implications":["Multi-agent escalation can be selective: the Judge activated in only 5 of 48 control cases, changing 3 statuses, all in a stricter direction, so adversarial review can be reserved for contested cases without a universal debate tax.","Human feedback routed through the Judge functions as a conservative calibration layer: 15 of 27 feedback interactions changed status, 13 moved away from a more permissive label, and no revision ended as COMPLIANT.","The system is more reliable at factual evidence interpretation than at sufficiency calibration, so deployment should pair LLM evidence review with deterministic completeness checks, especially over spreadsheet evidence where Control 5.9 showed a population-level reasoning error.","Audit-preparation users found the outputs actionable for identifying gaps and next steps (mean 3.84), which supports using such systems to help organizations prepare evidence before a formal audit."],"supporting_citations":[{"why":"Supplies the ISO 27001 control set and requirements that define what evidence must satisfy.","marker":"ISO/IEC 2022"},{"why":"Establishes retrieval-augmented generation, the grounding approach that keeps IntelliAudit's synthesis tied to cited evidence.","marker":"Lewis et al. 2020"},{"why":"Prior multi-agent retrieval for regulatory compliance that IntelliAudit extends with adversarial control assessment.","marker":"Agarwal et al. 2025"},{"why":"Earlier feasibility study of automating ISO 27001 auditing that IntelliAudit moves from proposal to implemented, evaluated system.","marker":"Riasat and Brachten 2024"},{"why":"Recent financial-audit agentic system that motivates the contrast: semantic IT controls resist deterministic verification.","marker":"Wang et al. 2026"},{"why":"Shows multiagent debate improves reasoning, grounding the Defender-and-Judge escalation design.","marker":"Du et al. 2023"},{"why":"Supports human-in-the-loop auditing of LLM outputs, informing the human-feedback channel through the Judge.","marker":"Rastogi et al. 2023"}],"fun_headline_variants":["Multi-agent LLM audit tool: good on facts, weak on sufficiency","LLM audit agents ace evidence, stumble on sufficiency calls","Audit LLMs: cited evidence solid, sufficiency needs humans","IntelliAudit: LLMs help audit prep, but not final sufficiency","Retrieval-grounded LLMs: audit support, not certifier"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that Likert ratings from a small panel of practicing auditors (12 to 17 per control) on author-constructed simulated evidence corpora are a reliable measure of audit-evidence sufficiency, because the paper states that no public ground-truth dataset exists for ISO 27001 evidence sufficiency and uses human evaluation as the primary reference.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent LLM audit tool: good on facts, weak on sufficiency","LLM audit agents ace evidence, stumble on sufficiency calls","Audit LLMs: cited evidence solid, sufficiency needs humans","IntelliAudit: LLMs help audit prep, but not final sufficiency","Retrieval-grounded LLMs: audit support, not certifier"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000172,"raw_usage":{"total_tokens":1264,"prompt_tokens":924,"completion_tokens":340,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":244}},"tokens_in":540,"tokens_out":340,"duration_ms":3937,"temperature":1.0,"reasoning_tokens":244,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:23:35.634540+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run IntelliAudit on a benchmark of real or realistically constructed ISO 27001 evidence bundles whose sufficiency status has been independently determined by completed professional audits, then count, over all control instances, the fraction of COMPLIANT recommendations that later receive an adverse audit finding; if that false-compliance rate approaches the base rate of non-compliance, the system's sufficiency judgments add no information.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ISO 27001 control set and requirements that define what evidence must satisfy."},{"cited_title":"I.; and Brachten, C","cited_arxiv_id":null,"evidence_quote":"Earlier feasibility study of automating ISO 27001 auditing that IntelliAudit moves from proposal to implemented, evaluated system."},{"cited_title":"AUDITFLOW: Executable Symbolic Environments for Structured Financial Reporting Verification","cited_arxiv_id":"2606.03031","evidence_quote":"Recent financial-audit agentic system that motivates the contrast: semantic IT controls resist deterministic verification."},{"cited_title":"Supporting Human-AI Collaboration in Auditing LLMs with LLMs","cited_arxiv_id":"2304.09991","evidence_quote":"Supports human-in-the-loop auditing of LLM outputs, informing the human-feedback channel through the Judge."}],"review_version":1}