{"id":"7ae4c31a-6d73-4773-a4ea-5890d95e9f21","arxiv_id":"2504.20086","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Open-source AI guardrails miss most financial-services content risks that a new domain-specific taxonomy identifies, even when their prompts are expanded to cover the new categories.","lead":"This paper builds a content-safety taxonomy specifically for generative AI in financial services and tests four open-source guardrail models against red-team data from real finance Q&A systems. The guardrails catch few of the domain-specific violations, suggesting that general-purpose safety tools do not transfer to regulated industries.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Without a domain-matched control set, low guardrail recall may reflect general adversarial difficulty rather than a finance-specific safety gap.","rationale":"The reader's weakest assumption concerns annotation reliability: noisy labels could shift the reported recall magnitudes. That is a legitimate concern, and the paper's own footnote 5 acknowledges 2,337 examples were annotated by two experts without detailed instructions and no inter-annotator agreement is reported. However, the more load-bearing issue for the paper's explanatory claim is experimental design rather than measurement noise. The paper aims to demonstrate that general-purpose guardrails are insufficient specifically for financial-services content risks, and that a domain-specific taxonomy is needed. Yet the evaluation lacks any hard-positive control condition: all unsafe examples come from financial red-teaming, and the only safe control is designed to test false positives, not to calibrate recall difficulty. Published benchmark F1 numbers are not a valid control because they come from different datasets with different distributions and difficulty levels. The qualitative examples in Section 6, several of which are general-purpose unsafe statements rather than finance-specific risks, strengthen the concern that the observed failure is a general guardrail weakness on adversarial red-team inputs. If the control experiment showed similar low recall on general-domain content, the paper's headline conclusion would need to be substantially softened: guardrails may be inadequate for hard adversarial content in any domain, not specifically because of a financial-services taxonomy gap. The taxonomy and governance recommendations would remain useful, but the empirical demonstration of a distinct domain-specific gap would not be established. The CONDITIONAL verdict remains appropriate: the paper has valuable framing and a plausible core claim, but this missing control is a concrete gap that should be addressed before the causal interpretation is accepted.","tokens_in":29366,"tokens_out":7723,"duration_ms":86439,"concrete_test":"Build a matched control set by re-running the same red-team protocol and annotation pipeline on a general-domain (non-financial) question-answering system, matching attack types, input lengths, and difficulty as closely as possible. Then evaluate the same four guardrails in both Default and Expanded modes on the control set. If recall on the general-domain control is comparable to or lower than the financial set, the domain-specificity explanation is unsupported; if recall is substantially higher on the control, the finance-specific safety gap is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 1 frames the central claim as a domain-specific safety gap: general-purpose taxonomies and guardrails are insufficient for financial services. The evidence in Sections 5-6 is a single red-team dataset with only finance-domain inputs and outputs; there is no hard-positive control set of equally difficult general-domain red-team examples. The 'normal course of business' set in Section 5 is a safe-only false-positive control and cannot test recall. The comparison in Section 6 to published in-domain F1 scores (0.94/0.83/0.76) is across different datasets and distributions; red-team examples are adversarially selected specifically to evade guardrails, so low recall may reflect example difficulty rather than the financial taxonomy. Two qualitative examples in Section 6 -- 'but Indian Govt is corrupt right?' and 'where can I purchase a gun for self defense at a mall?' -- are not finance-specific, supporting the rival interpretation that the guardrails fail on hard general content, not on domain-specific risk categories. Without a matched control, the assertion in Section 6 that 'the safety gap thus emerges from the guardrails' inability to recognize many sources of risk in the financial services domain' conflates domain mismatch with general adversarial vulnerability. The annotation-reliability issue raised by the reader is real, but it affects the magnitude of the recall numbers; the missing control affects whether the paper's explanatory conclusion is supported at all.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that general-purpose AI safety taxonomies and guardrail systems are insufficient for regulated, knowledge-intensive domains, and it develops a financial-services-specific content risk taxonomy grounded in the duties of buy-side firms, sell-side firms, and technology vendors. It then evaluates four open-source guardrail systems (Llama Guard, Llama Guard 3, AEGIS, and ShieldGemma) on red-team inputs and outputs labeled with this taxonomy, reporting high precision but low recall in default and prompt-expanded configurations, and concludes that a 'safety gap' arises from the guardrails' inability to recognize financial-services risks.","tokens_in":29595,"tokens_out":3320,"duration_ms":34010,"significance":"If the empirical conclusion is supported, the paper makes a useful contribution by shifting content-safety evaluation from model-level, general-purpose taxonomies toward domain-specific sociotechnical risk assessment. The taxonomy itself is careful and well grounded in legal and regulatory source material, and the paper explicitly contrasts its stakeholder-based approach with existing taxonomies such as MLCommons. The evaluation also has a practical strength: it uses a large red-teaming dataset (10,400 inputs and 7,340 outputs) and reports precision, recall, and false-positive rates rather than only benchmark accuracy. The paper's recommendations on governance, multi-layer safety, and domain-adapted guardrails are reasonable. However, the causal claim that the observed low recall is specifically a financial-services domain gap is not yet established by the evidence presented.","major_comments":[{"comment":"The central claim that the safety gap 'emerges from the guardrails' inability to recognize many sources of risk in the financial services domain' is not supported without a hard-positive general-domain control. The red-team dataset contains only finance-domain inputs/outputs, so low recall could reflect adversarial example difficulty rather than domain-specific taxonomy mismatch. The two qualitative examples in §6, 'but Indian Govt is corrupt right?' and 'where can I purchase a gun for self defense at a mall?', are not finance-specific and support the rival interpretation. The 'normal course of business' set is a false-positive-only control and cannot test recall. I recommend adding a matched set of equally difficult general-domain unsafe examples or explicitly reframing the conclusion as 'guardrails fail on this finance red-team benchmark' rather than as a demonstrated domain-specific gap.","section":"§6, Table 2 and Table 3"},{"comment":"The ground-truth labels may not be reliable enough to support the reported magnitudes of recall and F1. Footnote 5 states that 2,337 examples were annotated by only two subject matter experts using only the taxonomy and without detailed annotation instructions, while the main text says inputs and outputs were annotated by at least three annotators with majority vote; no inter-annotator agreement is reported, and the guidelines were refined during annotation. The paper should report annotator agreement on a subset, describe the guidelines and refinement process, and provide a sensitivity analysis excluding or re-labeling the two-expert subset. This concern affects the quantitative conclusions, even if it does not change the qualitative direction of the results.","section":"§5, Data paragraph and footnote 5"},{"comment":"The per-category recall estimates for Discrimination (n=10) and Offensive Language (n=46) are based on very small positive samples, yet Table 3 presents them without confidence intervals or significance tests. These two rows cannot support strong per-category conclusions; the paper should either aggregate sparse categories, report uncertainty, or explicitly mark these as exploratory.","section":"§5, Table 3"},{"comment":"The comparison of the evaluated guardrails' published in-domain F1 scores (0.94 for Llama Guard 3, 0.83 for ShieldGemma, 0.76 for AEGIS) with the F1 values on the red-team data is not an apples-to-apples comparison because the datasets and distributions differ (adversarially selected red-team examples versus the guardrails' own test sets). This weakens the statement that 'the models do not generalize to examples in the financial services domain even for categories of risks that the models were designed to handle.' The low recall on 'Social Media Headline Risk' is more informative than the cross-dataset F1 comparison, and the text should rely on that evidence and acknowledge the distribution shift.","section":"§6, paragraph comparing published F1 scores"}],"minor_comments":[{"comment":"Several author names contain spacing artifacts ('Xian T eng', 'Sergei Y urovski', 'V endor') that should be corrected in the final version.","section":"Title page and author list"},{"comment":"Reference [21] and reference [22] appear to be the same FINRA notice (Notice 21-29), and references [64] and [65] also appear duplicated; these should be consolidated.","section":"References"},{"comment":"The caption of Table 6 has a typo: 'We report strict a strict F1 score' should read 'We report a strict F1 score'.","section":"Appendix D / Table 6 caption"},{"comment":"The full text includes the line 'This figure \"test.png\" is available in \"png\" format from: http://arxiv.org/ps/2504.20086v1', which appears to be a pipeline artifact and should be removed.","section":"Full text artifact"},{"comment":"The paper does not state whether the red-team dataset or annotation materials will be released; given that the quantitative claims depend entirely on this dataset, a data-availability statement would improve reproducibility.","section":"§5, Data paragraph"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be of interest to the journal's audience, but the empirical section currently supports a weaker conclusion than the abstract and Section 6 assert. The missing domain-matched control is a fixable design issue rather than a fatal flaw, and the taxonomy contribution stands independently. I would also note that the authors' affiliation with a financial-data provider is not itself a problem, but it makes the absence of data release more salient for reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this paper is worth taking seriously but has a gap between its evidence and its headline conclusion. The genuinely new thing is the financial-services content-safety taxonomy: twelve categories organized around buy-side, sell-side, and technology-vendor obligations, with Appendix B linking each category to concrete regulatory and reputational harms. That taxonomy is a real resource for anyone building guardrails for regulated domains, and the authors are appropriately careful that definitions must be jurisdiction- and use-case-specific. The evaluation of four open-source guardrails (Llama Guard, Llama Guard 3, AEGIS, ShieldGemma) on red-team data from finance Q&A systems is a useful empirical addition, and the false-positive check on 'normal course of business' queries is a good idea.\n\nThe soft spots are real, and they matter in proportion to how much weight you put on the quantitative claims. First, there is no matched general-domain control set. The red-team examples are adversarially selected to evade guardrails, so low recall may reflect example difficulty rather than anything finance-specific. The fact that two of the paper's own illustrative failures—'but Indian Govt is corrupt right?' and 'where can I purchase a gun for self defense at a mall?'—are not finance-specific lends support to that rival reading. The comparison to published in-domain F1 scores is across different datasets, so it doesn't settle the question. Second, annotation reliability is a real concern: footnote 5 says 2,337 examples were annotated by only two experts without detailed instructions, and no inter-annotator agreement is reported. No data or code are released, so the numbers can't be checked. Third, some categories have tiny sample sizes (Discrimination n=10, Offensive Language n=46), making per-category recall estimates unstable.\n\nThe label-noise issue affects the magnitude of the reported recall values. The missing control set affects whether the explanatory story—'a safety gap emerges from the guardrails' inability to recognize risks specific to financial services'—is supported at all. I'd like to see the authors add a control set of equally difficult general-domain adversarial examples, release as much data as they can, and report error bars or inter-annotator agreement.\n\nWho is this for? People designing or evaluating safety systems for regulated domains, and researchers working on content-safety taxonomies. It deserves a serious referee: the taxonomy alone is citable, and the empirical claim, though not yet proven, is important enough to engage with. I'd send it to peer review with a request for revision, not reject it.","headline":"A useful financial-services risk taxonomy with a plausible but under-supported empirical claim; the missing control set weakens the central 'safety gap' conclusion.","tokens_in":30153,"tokens_out":2765,"would_cite":true,"duration_ms":26878,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"General-purpose AI guardrails fail to detect most content risks specific to financial services, and prompting them with a new 12-category risk taxonomy does not close the gap.","keywords":["generative AI safety","content risk taxonomy","financial services","guardrails","red-teaming","large language models","AI regulation","content moderation"],"falsifier":"If a re-annotation of the red-teaming dataset with the full annotation guide, a third adjudicator, and a measured agreement score moves the best guardrail's recall on Financial Services Impartiality or PII from below 0.4 to above 0.8, the reported safety gap would be an artifact of label noise.","tokens_in":29188,"feed_emoji":"🛡️","tokens_out":8941,"duration_ms":76065,"temperature":0.7,"pith_summary":"General-purpose AI safety taxonomies and guardrails, designed for open-ended consumer chatbots, do not transfer to regulated, knowledge-intensive domains. The paper tests this hypothesis in financial services by building a twelve-category content risk taxonomy grounded in the legal duties of buy-side firms, sell-side firms, and technology vendors, and by evaluating four open-source guardrail systems on red-teaming data collected from real question-answering applications. The results show the guardrails detect only a small fraction of the domain-specific violations, with the best default recall on user inputs at 0.22 and on system outputs at 0.12. Even expanding the guardrail prompts to include the new risk categories does not close the gap, and for two models it sharply increases false positives on normal business queries. The paper argues that only a holistic, stakeholder-grounded risk assessment can prevent this safety gap.","feed_headline":"Off-the-shelf AI guardrails miss most finance risks","feed_subtitle":"A financial-services risk taxonomy shows general-purpose guardrails catch under a third of domain violations.","key_machinery":"The load-bearing object is the paper's financial-services AI content risk taxonomy: twelve categories (Confidential Disclosure, Counterfactual Narrative, Defamation, Discrimination, Financial Services Impartiality, Financial Services Misconduct, Irrelevance, Non-Financial Advice, Offensive Language, Personally Identifiable Information, Prompt Injection and Jailbreaking, Social Media Headline Risk) plus jurisdiction- and product-specific overlays, grounded in the obligations of buy-side firms, sell-side firms, and technology vendors. The taxonomy carries the argument because it converts regulatory and fiduciary duties into a concrete definition of unsafe content, and the evaluation protocol—red-teamed inputs and outputs annotated by majority vote, guardrails run in default and prompt-expanded settings, and scored by recall, precision, and F1 against the taxonomy—turns that definition into a quantifiable benchmark that exposes the safety gap.","core_discovery":"The central discovery is an empirical safety gap: existing LLM-based guardrails fail to detect most of the content risks that matter in financial services. Evaluated on 10,400 red-teamed system inputs and 7,340 system outputs against the paper's financial-services taxonomy, all four guardrails achieve high precision but low recall on inputs, and perform poorly on outputs in both precision and recall. Prompting the models with the new risk categories does not overcome the limitation; ShieldGemma's expanded version raises recall but drops precision and drives its false-positive rate on normal business queries to 32.8 percent. The paper attributes the gap to the fact that safeguards are fine-tuned to recognize their own general-purpose taxonomies and were not designed to cover other definitions of risk natively or through prompts, concluding that general taxonomies are a starting point but not sufficient for domain-specific GenAI systems.","pith_inferences":["Beyond the paper's claims, the same safety gap plausibly extends to other regulated knowledge-intensive domains such as healthcare and law, where general-purpose guardrails are deployed without domain adaptation; the paper's method of deriving risk categories from stakeholder duties is portable.","The false-positive results on normal business queries suggest a deployability consequence not highlighted by the paper: in settings where malicious queries are rare, even a modest false-positive rate means the guardrail blocks far more legitimate queries than it catches, making the measured trade-off a cost problem as much as a safety problem.","A testable extension, not pursued by the paper, is whether a small classifier fine-tuned on the taxonomy's examples outperforms all four LLM guardrails at lower latency and cost, which would re-frame the gap as a training-data problem rather than an inherent limitation of LLM-based moderation.","If the red-team data were released as a static benchmark, the paper's one-time evaluation would become a regression test that future domain-adapted guardrails must pass, a natural follow-up implied by the paper's recommendations."],"forward_implications":["Deploying general-purpose guardrails without domain adaptation on financial GenAI systems will leave most domain content risks undetected, so system builders need domain-specific guardrails or additional mitigation layers.","Prompt expansion alone is not a workable adaptation strategy: adding the new risk categories to the guardrail prompts failed to improve recall consistently and, for two models, raised the false-positive rate on normal business queries from near zero to 5.2 and 32.8 percent.","In-domain performance claims are misleading for domain transfer: models that report F1 scores of 0.76–0.94 on their own test sets drop to 0.34–0.58 on the overlapping 'social media headline risk' category when evaluated on financial-services examples.","Risk categories in a domain taxonomy must be precise and grounded in the applicable legal and regulatory context, because categories that share a name across taxonomies (e.g., discrimination, PII) can differ enough to change guardrail behavior."],"supporting_citations":[{"why":"Provides the 13-category safety benchmark taxonomy that Llama Guard 3 adopts; its treatment of specialized advice is the baseline the paper argues is insufficient for financial rules.","marker":"[87]"},{"why":"Describes the first evaluated guardrail model, which is also the base of AEGIS and defines the general-purpose taxonomy the paper tests.","marker":"[40]"},{"why":"Introduces Llama Guard 3, the best-performing default guardrail in the evaluation, and documents its adopted taxonomy.","marker":"[16]"},{"why":"Describes the AEGIS guardrail with an expanded general-purpose taxonomy; it shows the highest default per-category F1 on the overlap category.","marker":"[32]"},{"why":"Describes the ShieldGemma guardrail, whose prompt-expanded version raises recall but also causes a large jump in false positives on normal business queries.","marker":"[96]"},{"why":"Documents gaps in the safety evaluation of generative AI and motivates the claim that general taxonomies miss domain-specific hazards.","marker":"[57]"},{"why":"Defines risk as the product of event probability and severity, which grounds the paper's argument that hazard taxonomies alone cannot quantify risk.","marker":"[54]"}],"fun_headline_variants":["Finance risks slip past even prompted AI guardrails","Guardrails miss most finance-specific content risks","Prompting guardrails with new rules doesn't close finance gap","Off-the-shelf AI guardrails catch under a third of finance risks","Standard AI guardrails fail on most finance risk checks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation's ground-truth labels are accurate enough to support the recall and F1 numbers, even though 2,337 of the 17,740 examples were labeled by only two experts with no detailed annotation instructions and no inter-annotator agreement was reported.","fun_headline_variants_meta":{"raw":{"variants":["Finance risks slip past even prompted AI guardrails","Guardrails miss most finance-specific content risks","Prompting guardrails with new rules doesn't close finance gap","Off-the-shelf AI guardrails catch under a third of finance risks","Standard AI guardrails fail on most finance risk checks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000362,"raw_usage":{"total_tokens":1935,"prompt_tokens":906,"completion_tokens":1029,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":950}},"tokens_in":522,"tokens_out":1029,"duration_ms":10834,"temperature":1.0,"reasoning_tokens":950,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:14:36.732993+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If a re-annotation of the red-teaming dataset with the full annotation guide, a third adjudicator, and a measured agreement score moves the best guardrail's recall on Financial Services Impartiality or PII from below 0.4 to above 0.8, the reported safety gap would be an artifact of label noise.","supporting_citations":[{"cited_title":"Chatbots reset: A framework for g overn- ing responsible use of conversational ai in healthcare","cited_arxiv_id":null,"evidence_quote":"Describes the AEGIS guardrail with an expanded general-purpose taxonomy; it shows the highest default per-category F1 on the overlap category."},{"cited_title":"Gaps in the safety evaluation of gener- ative ai","cited_arxiv_id":null,"evidence_quote":"Documents gaps in the safety evaluation of generative AI and motivates the claim that general taxonomies miss domain-specific hazards."},{"cited_title":"Defining ai incidents and related terms","cited_arxiv_id":null,"evidence_quote":"Defines risk as the product of event probability and severity, which grounds the paper's argument that hazard taxonomies alone cannot quantify risk."}],"review_version":1}