Pith. sign in

REVIEW 5 major objections 6 minor 2 cited by

LLM-based HSE Compliance Assessment: Benchmark, Performance, and Advancements

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Large language models appear competent on HSE compliance questions by matching words between scenario and options, not by reasoning from the regulations, and a new multi-expert prompting technique produces more accurate decisions.

desk verdict HSE-Bench is a genuinely useful resource, but the semantic-matching claim rests on an unvalidated metric comparison. read the letter →

arxiv 2505.22959 v1 pith:MTQCKKXX submitted 2025-05-29 cs.CL

Jianwei Wang , Mengqi Wang , Yinsi Zhou , Zhenchang Xing , Qing Liu , Xiwei Xu , Wenjie Zhang , Liming Zhu This is my paper
classification cs.CL
keywords hsecomplianceassessmentllmevaluationbenchmarkdatasetiracframeworksemanticmatchinglegalreasoningpromptengineeringofexperts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish what large language models actually do when they appear to assess health, safety, and environment (HSE) compliance, and the answer is that they mostly match words, not rules. To show this it builds HSE-Bench, over 1,000 curated questions from regulations, court cases, safety exams, and fieldwork videos, organized by the four-step IRAC legal-reasoning pipeline. Across twelve models, accuracy with answer options visible averages about 90 percent, but when options are withheld the same models' free-form answers score about 25 percentage points lower, which the authors read as evidence that the apparent competence rests on semantic matching rather than principled reasoning grounded in the compliance context. The paper's remedy is a prompting strategy, Reasoning of Expert (RoE), that makes the model deliberate as several domain experts before deciding, and it reports accuracy gains up to 25.01 percent over chain-of-thought on DeepSeek-R1. If these findings hold, organizations should treat LLM compliance advice as unverified until a human checks the reasoning.

What carries the argument

The machinery that carries the argument is a pair of matched evaluation protocols whose score gap is treated as evidence of how models think. In the accuracy protocol the model sees the scenario, question, and all four options and must choose one, which the paper argues rewards surface-level semantic matching. In the AUC-ROC protocol only the scenario and question are shown; the model's free-form answer is embedded with BERT and compared by cosine similarity against each option's embedding, so the model must effectively produce the correct content on its own. The benchmark itself is organized by the IRAC framework (Issue spotting, rule Recall, rule Application, rule Conclusion), a four-step legal-reasoning schema borrowed from legal education, which lets the authors localize failures to specific reasoning stages. The proposed improvement, Reasoning of Expert (RoE), is a prompt that instructs the model to assume several domain-expert roles, produce an independent IRAC analysis for each, and synthesize the opinions into a final judgment that balances regulatory compliance with practical safety.

What would settle it

Run the same 1,020 questions under a single metric: ask each model to answer without options and then grade those free-form answers the way the accuracy protocol grades choices, or, conversely, present options but grade by whether the model's reasoning names the correct regulation before picking an option. If a model's option-free accuracy matched its optioned accuracy on questions whose distractors share no surface vocabulary with the scenario, the semantic-matching explanation would collapse. A concrete check is available in the released data: compute the accuracy–AUC-ROC gap separately for distractors with high versus low lexical overlap with the scenario; the gap should largely vanish for low-overlap distractors under the paper's own account.

Watch

Extended reading notes

Core claim

The paper's central claim is that current large language models give a misleading impression of HSE compliance competence. Given a scenario and four answer options they select the correct option at roughly 90 percent accuracy, but the paper argues this success is driven by semantic matching between the scenario and the options rather than by principled reasoning grounded in the underlying HSE compliance context. The evidence is the consistent gap between the two evaluation protocols: when only the scenario and question are presented and the model's free-form answer is scored by how strongly it resembles each option, average performance falls by 24.73 percent across all models and data sources. Reasoning-trace case studies show models engaged in option-by-option elimination rather than structured legal analysis, and the paper claims their native reasoning lacks the systematic legal pipeline—issue spotting, rule recall, rule application, rule conclusion—that rigorous HSE assessment requires. The paper further claims that its Reasoning of Expert (RoE) prompt, which has the model produce independent IRAC-based opinions from multiple expert roles and then synthesize a unified decision, substantially improves accuracy over chain-of-thought prompting.

Load-bearing premise

The headline finding depends on the assumption that the accuracy score (options supplied) and the AUC-ROC score (options withheld, measured by BERT cosine similarity) are comparable measures of the same ability, so that the gap between them reflects semantic matching rather than a difference in task difficulty or a weakness of the similarity-based scoring proxy.

Editorial extensions

If this is right

  • Human oversight remains essential if LLMs are used in HSE compliance assessment, since high multiple-choice accuracy does not indicate command of the underlying regulations.
  • Chain-of-thought prompting does not close the HSE reasoning gap; in the reported experiments CoT-based prompts often failed to beat zero-shot prompting on both accuracy and AUC-ROC.
  • Reasoning-oriented models did not outperform foundation models on HSE-Bench, and their accuracy-to-AUC-ROC drop was larger, so general reasoning strength does not transfer automatically to regulatory compliance reasoning.
  • The Reasoning of Expert (RoE) prompt improved both models tested, with gains up to 25.01 percent accuracy over CoT on DeepSeek-R1 and up to 8.03 percent AUC-ROC, with larger relative gains on the reasoning model.
  • Rule recall is the weakest stage of the IRAC pipeline across models, with an average AUC-ROC of 60.75 percent, indicating that LLMs struggle to name the specific regulation that governs a spotted issue.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The accuracy–AUC-ROC gap would be a stronger argument if the two protocols differed only in whether options are shown; as designed they also differ in scoring method (exact choice versus BERT cosine similarity) and in output format, so some of the 24.73 percent gap may be measurement artifact rather than evidence of semantic matching.
  • A natural deployment implication the authors do not spell out is that option-free, generation-style probes are the more honest test of an LLM's HSE competence, so organizations could re-run the HSE-Bench questions in open-ended form as a spot audit before trusting model output.
  • RoE's expert-role deliberation resembles decision processes in other regulated fields, so the prompt is a plausible candidate for testing in financial compliance, medical safety, or privacy-law assessment tasks.
  • The benchmark could be extended to measure grounding directly by scoring whether a model's answer cites a real rule identifier; that would convert the semantic-matching claim from an interpretation of a score gap into a directly observable behavior.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper introduces HSE-Bench, a benchmark of 1,020 multiple-choice questions for Health, Safety, and Environment compliance assessment, constructed by prompting GPT-4o to convert regulations, court cases, safety exams, and fieldwork-video transcripts into IRAC-structured (Issue, Rule, Application, Conclusion) questions, followed by adversarial augmentation and expert double-checking. The authors evaluate 12 LLMs under six prompting strategies and report two metrics: selection accuracy when options are provided and BERT-cosine-based AUC-ROC when options are omitted. They observe a large gap (~24.73% on average) between the two and conclude that LLMs rely on semantic matching rather than principled HSE reasoning; they also find that reasoning models do not beat foundation models and that their proposed Reasoning of Experts (RoE) prompt improves accuracy, with up to 25.01% over CoT on DeepSeek-R1.

Significance. The benchmark and code are publicly released, the data sources are diverse, and the IRAC-based structure is a useful contribution to evaluating LLMs in a high-stakes regulatory domain. The RoE prompt is a simple, plausible intervention, and the reported average improvements are worth investigating. However, the headline conclusion that 'capabilities largely rely on semantic matching' rests on a comparison of two different tasks with two different scoring functions, and the benchmark labels are themselves largely GPT-4o-generated; both issues need to be resolved before the central claims can be accepted. The paper also omits numerical tables, error bars, and human-verification statistics, which are needed to support several secondary claims.

major comments (5)
  1. [Section 4.1 and 4.2.1, Figure 2] The semantic-matching conclusion is not supported by the reported experiment because accuracy (options provided, direct selection) and AUC-ROC (options omitted, BERT cosine similarity) differ in both input format and scoring mechanism. A drop from ~90% accuracy to ~70% AUC-ROC is therefore compatible with open-ended generation being harder or with BERT similarity being a noisy proxy, rather than with semantic matching. To support the claim, the authors should validate the BERT-similarity score against human judgments on the same free-form outputs (or a second embedding model / log-probability confidence), and ideally compare like with like by scoring both settings with the same mechanism (e.g., forced-choice accuracy on options vs. open-form answers judged by humans or a calibrated model). The case studies in Table 4 are anecdotal and do not quantify the causal role of semantic matching.
  2. [Section 3.3 and Table 1] The benchmark labels are generated by GPT-4o and only briefly described as double-checked by two experts; no expert qualifications, annotation instructions, agreement rates, or counts of corrected labels are reported. Because every model is evaluated against these labels, the reported performances may partly reflect agreement with GPT-4o's labeling style rather than HSE competence. The authors should report inter-annotator agreement, the number of questions changed during review, and ideally provide a human-verified gold subset that is independent of the LLM generation process.
  3. [Section 3.1 and Section 4.1 (Experimental Setup)] The fieldwork-video questions appear to be constructed from GPT-4o-generated textual descriptions of videos, and the setup states that for video LLMs 'we follow the existing method... to extract frames from the video and use GPT-4o to generate textual descriptions.' If this means that both vision-language models and text-only models are evaluated on the same text, then the vision-model results do not test visual input at all. Please state explicitly whether vision models received frames/videos; if they did not, the 'vision models' category should be removed or re-scoped to text-only evaluation.
  4. [Section B.1 and Section 4.2.2] The prompt templates are internally inconsistent: the CoT and 0-CoT templates include 'Choose the best answer. Reply only with the letter A, B, C, or D. **Do NOT explain your choice...**' immediately before 'Let's think step by step:'. If the 'do not explain' instruction was actually sent, the CoT conditions may not have elicited the intended reasoning, which would directly affect the comparison with RoE and the conclusion that CoT does not help. In addition, the abstract's 'up to 25.01%' improvement for RoE over CoT on DeepSeek-R1 cannot be traced in the reported text, which gives average gains of about 5%; the full per-cell numerical results behind Figure 3 should be provided.
  5. [Section 4.2.1 and 4.2.2] The comparative claims (e.g., reasoning models do not outperform foundation models; the drop is larger for reasoning models; later IRAC phases are harder) are made without confidence intervals or repeated runs. The only stochastic element discussed is CoT+SC, which is run five times, and no variance is reported. Because LLM outputs are random, the authors should report at least three runs with standard deviations for the headline comparisons or use significance tests, and include the underlying numerical tables rather than only figures.
minor comments (6)
  1. [Appendix C heading] The heading 'Benchmark deatils' contains a typo; it should read 'Benchmark details'.
  2. [Figure 2(a-d) and Figure 5] Figure 2(a-d) labels the y-axis 'Accuracy' but extends it to 1.4, and some AUC-ROC axes in Figure 5 extend above 1.0; since neither metric can exceed 1.0, the axes should be rescaled or the plotted values should be rechecked.
  3. [Section 4.2.1] The sentence 'AUC-ROC is typically higher than or equal to accuracy' is used to argue that the observed drop is even greater, but that property holds only when the same scoring function underlies both metrics; since the paper's AUC-ROC uses BERT cosine similarity rather than model confidence, the argument is not valid as stated.
  4. [Section 2 (Related Works)] The paper claims to be the 'first comprehensive HSE compliance benchmark' but does not position itself against LabSafety Bench and AIR-Bench in enough detail; a short comparison of task scope and data sources would make the novelty claim clearer.
  5. [Table 3 in Appendix C] The correct options in Table 3 are marked in bold, but the caption does not state this; please add a note explaining the bold formatting.
  6. [Section B.1] The few-shot prompt template says 'k shot of different...', while the text states that 5 examples are used; please make the placeholder and the actual number consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark labels are manually checked, and the semantic-matching claim is an interpretation of a measured gap rather than a definitional reduction.

full rationale

I walked the paper's derivation chain and found no step that reduces to its own inputs by construction. Benchmark construction uses GPT-4o to propose scenario-based QA instances, but the paper states that two experts manually check every question and that regulation, court-case, and safety-exam labels are directly grounded in the source documents; this provides independent grounding. Evaluating GPT-4o-mini against GPT-4o-generated labels is a possible contamination or alignment risk, but it is not a self-definitional loop because the gold labels are not defined in terms of the evaluated models' outputs. The central 'semantic matching' finding is an empirical interpretation of an observed gap between with-options accuracy and no-options BERT-cosine AUC-ROC, supported by qualitative reasoning-trace case studies; the paper does not formally define 'semantic matching' as equivalent to that gap via an equation, so the claim is underdetermined but not circular. RoE prompting is an independent contribution evaluated on the same benchmark, and the self-citations in the reference list are methodological and not load-bearing. No uniqueness theorem, fitted parameter renamed as a prediction, or ansatz smuggled in via self-citation appears. The honest finding is therefore no significant circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No numeric free parameters are fitted. The main assumptions are domain assumptions about label reliability, IRAC as the right decomposition, BERT similarity as a reasoning probe, and source representativeness. No new entities are postulated.

assumptions (5)
  • domain assumption GPT-4o-generated questions and labels are accurate after two experts check each question.
    Used in Section 3.3 to establish ground truth; no inter-annotator agreement, expert qualifications, or error audit is reported.
  • domain assumption IRAC is an appropriate and complete decomposition of HSE compliance assessment.
    Section 3.2 defines all questions through Issue, Rule, Application, and Conclusion; if this schema omits relevant reasoning modes, the benchmark is incomplete.
  • domain assumption BERT cosine similarity between a free-form model answer and each option is a valid proxy for the model's confidence in that option.
    Section 4.1 defines AUC-ROC this way, and the semantic-matching finding depends on this proxy.
  • ad hoc to paper Comparing multiple-choice accuracy with open-ended AUC-ROC can reveal semantic matching versus principled reasoning.
    Section 4.2.1 interprets the accuracy/AUC gap as evidence for semantic matching, but the two measurements use different input formats and different scoring mechanisms.
  • domain assumption The selected regulations, court cases, exams, and videos are representative of real HSE compliance scenarios.
    Appendix C lists manually collected sources; representativeness is asserted rather than demonstrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM-based HSE Compliance Assessment: Benchmark, Performance, and Advancements." pith.science (2026). https://pith.science/paper/MTQCKKXX

@misc{pith2026250522959,
  author       = {Pith},
  title        = {Pith review of: LLM-based HSE Compliance Assessment: Benchmark, Performance, and Advancements},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MTQCKKXX}},
  note         = {Machine review of arXiv:2505.22959}
}
read the original abstract

Health, Safety, and Environment (HSE) compliance assessment demands dynamic real-time decision-making under complicated regulations and complex human-machine-environment interactions. While large language models (LLMs) hold significant potential for decision intelligence and contextual dialogue, their capacity for domain-specific knowledge in HSE and structured legal reasoning remains underexplored. We introduce HSE-Bench, the first benchmark dataset designed to evaluate the HSE compliance assessment capabilities of LLM. HSE-Bench comprises over 1,000 manually curated questions drawn from regulations, court cases, safety exams, and fieldwork videos, and integrates a reasoning flow based on Issue spotting, rule Recall, rule Application, and rule Conclusion (IRAC) to assess the holistic reasoning pipeline. We conduct extensive evaluations on different prompting strategies and more than 10 LLMs, including foundation models, reasoning models and multimodal vision models. The results show that, although current LLMs achieve good performance, their capabilities largely rely on semantic matching rather than principled reasoning grounded in the underlying HSE compliance context. Moreover, their native reasoning trace lacks the systematic legal reasoning required for rigorous HSE compliance assessment. To alleviate these, we propose a new prompting technique, Reasoning of Expert (RoE), which guides LLMs to simulate the reasoning process of different experts for compliance assessment and reach a more accurate unified decision. We hope our study highlights reasoning gaps in LLMs for HSE compliance and inspires further research on related tasks.

Figures

Figures reproduced from arXiv: 2505.22959 by the authors.

Figure 1
Figure 1. Conceptual framework of this project, featuring benchmark construction, comprehensive [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overall results on different sources of data. Without options, LLMs show a sharp perfor [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Prompt strategies evaluation on both DeepSeek-V3 and DeepSeek-R1. Our RoE prompt [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: IRAC reasoning evaluation. LLMs show a significant performance drop in the full IRAC [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Prompt strategies evaluation (AUC-ROC). 64.25%, 65.92%, 65.04%, 65.19%, and 66.10% for DeepSeek-R1. Secondly, for reasoning-centric models, appropriate prompting strategies can lead to more significant performance gains compared to those of foundation models. Notably, …

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. WuYu-EnvLE-Bench: A Benchmark for Evaluating Large Language Models in Environmental Law Enforcement

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Across 14 environmental-law-enforcement tasks, current LLMs score 80-90+ on rule-bounded decisions but only 20-50 on contradiction monitoring and multi-evidence integration, and medium models nearly match frontier mod...

  2. PASTA: A Scalable Framework for Multi-Policy AI Compliance Evaluation

    cs.HC 2026-01 conditional novelty 6.0 of 10

    PASTA is a model-card-based LLM pipeline that evaluates an AI system against five regulations in minutes for about $3, with expert-aligned violation and relevance scores.

Reference graph

Works this paper leans on

72 extracted references · 55 canonical work pages · cited by 2 Pith papers

  1. [1]

    Cooperation, competition, and maliciousness: Llm-stakeholders interactive negotiation.Advances in Neural Information Processing Systems, 37:83548–83599, 2024

    Sahar Abdelnabi, Amr Gomaa, Sarath Sivaprasad, Lea Sch¨onherr, and Mario Fritz. Cooperation, competition, and maliciousness: Llm-stakeholders interactive negotiation.Advances in Neural Information Processing Systems, 37:83548–83599, 2024

    work page 2024

  2. [2]

    Working for the environment and against safety: How compliance affects health and safety on board ships.Safety Science, 87:131–143, 2016

    Ngwatung Akamangwa. Working for the environment and against safety: How compliance affects health and safety on board ships.Safety Science, 87:131–143, 2016

    work page 2016

  3. [3]

    Andreopoulos

    Mohammad Adil Ansari, Genya Ishigaki, and William B. Andreopoulos. Fine-tuning large language models for environmental health and safety applications. In2024 Conference on AI, Science, Engineering, and Technology (AIxSET), pages 45–52, 2024

    work page 2024

  4. [4]

    A review of compliance with health and safety regula- tions and economic performance in small and medium construction enterprises.Edinburgh, UK, sn, pages 423–432, 2012

    Andrew Oyen Arewa and Peter Farrell. A review of compliance with health and safety regula- tions and economic performance in small and medium construction enterprises.Edinburgh, UK, sn, pages 423–432, 2012

    work page 2012

  5. [5]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  6. [6]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  7. [7]

    A survey on evaluation of large language models.ACM transactions on intelligent systems and technology, 15(3):1–45, 2024

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models.ACM transactions on intelligent systems and technology, 15(3):1–45, 2024

    2024

  8. [8]

    Vision language model for interpretable and fine-grained detection of safety compliance in diverse workplaces

    Zhiling Chen, Hanning Chen, Mohsen Imani, Ruimin Chen, and Farhad Imani. Vision language model for interpretable and fine-grained detection of safety compliance in diverse workplaces. Expert Systems with Applications, 265:125769, 2025

    work page 2025

Show all 72 references
  1. [9]

    Chatgpt goes to law school.J

    Jonathan H Choi, Kristin E Hickman, Amy B Monahan, and Daniel Schwarcz. Chatgpt goes to law school.J. Legal Educ., 71:387, 2021

    2021

  2. [10]

    Large legal fictions: Profiling legal hallucinations in large language models.Journal of Legal Analysis, 2024

    Matthew Dahl, Xinyu Lin, Harry Surden Green, et al. Large legal fictions: Profiling legal hallucinations in large language models.Journal of Legal Analysis, 2024. Forthcoming

    2024

  3. [11]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human langua...

    2019

  4. [12]

    Management of health, safety and environment in process industry.Safety Science, 46(6):908– 920, 2008

    Nijs Jan Duijm, C´ecile Fi´evez, Marko Gerbec, Ulrich Hauptmanns, and Myrto Konstandinidou. Management of health, safety and environment in process industry.Safety Science, 46(6):908– 920, 2008

    2008

  5. [13]

    Revisiting the traditional irac organisational structure for legal analysis: Towards a multidisciplinary approach.Nigerian LJ, 20:31, 2017

    Oghenemaro Festus Emiri, Ayuba Giwa, and Jonathan Ehusani. Revisiting the traditional irac organisational structure for legal analysis: Towards a multidisciplinary approach.Nigerian LJ, 20:31, 2017

    2017

  6. [14]

    Llm- supported safety annotation in high-risk environments

    Mohammad Eskandari, Murali Indukuri, Stephanie M Lukin, and Cynthia Matuszek. Llm- supported safety annotation in high-risk environments. InHRI 2025 Workshop VAM-HRI

    2025

  7. [15]

    An accuracy metric: Percentages, randomness, and probabilities.Journal of Data and Information Quality (JDIQ), 1(3):1–21, 2009

    Craig W Fisher, Eitel JM Lauria, and Carolyn C Matheus. An accuracy metric: Percentages, randomness, and probabilities.Journal of Data and Information Quality (JDIQ), 1(3):1–21, 2009

    2009

  8. [16]

    Fair balancing? evaluating llm-based privacy policy ethics assessments

    Vincent Freiberger and Erik Buchmann. Fair balancing? evaluating llm-based privacy policy ethics assessments. InProceedings of the 3rd European Workshop on Algorithmic Fairness (EWAF’24), 2024

    2024

  9. [17]

    Examining proposed uses of llms to produce or assess assurance arguments

    Mallory S Graydon and Sarah M Lehman. Examining proposed uses of llms to produce or assess assurance arguments. 2025. 10

    2025

  10. [18]

    The power of stories: Narrative priming shapes how llm agents collaborate and compete.arXiv preprint arXiv:2505.03961, 2025

    Gerrit Großmann, Larisa Ivanova, Sai Leela Poduru, Mohaddeseh Tabrizian, Islam Mesabah, David A Selby, and Sebastian J V ollmer. The power of stories: Narrative priming shapes how llm agents collaborate and compete.arXiv preprint arXiv:2505.03961, 2025

    2025 arXiv

  11. [19]

    Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models

    Neel Guha, Julian Nyarko, Daniel Ho, Christopher R ´e, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in Neur...

    2023

  12. [20]

    Enhancing visual-llm through prompt engineering and a two-stage retrieval- augmented generation algorithm for construction site safety compliance checking.Available at SSRN 5097308

    Xiaowen Guo, Peter Kok-Yiu WONG, Jack CP Cheng, Jeff Chak Fu Chan, Pak-Him Leung, and Xingyu Tao. Enhancing visual-llm through prompt engineering and a two-stage retrieval- augmented generation algorithm for construction site safety compliance checking.Available at SSRN 5097308

  13. [21]

    An integrated process-based hse management system: A case study

    Vahid Hajipour, Hamidreza Amouzegar, Ali Gharaei, Mohammad Saeed Gholami Abarghoei, and Sepideh Ghajari. An integrated process-based hse management system: A case study. Safety Science, 133:104993, 2021

    2021

  14. [22]

    Artificial intelligence in hse

    Dayanch Hojageldiyev. Artificial intelligence in hse. InAbu Dhabi International Petroleum Exhibition and Conference, page D012S120R002. SPE, 2018

    2018

  15. [23]

    Rulearena: A benchmark for llm rule-guided reasoning in real-world scenarios

    Yifan Jiang, Xinyi He, Ruiqi Wang, et al. Rulearena: A benchmark for llm rule-guided reasoning in real-world scenarios. InICLR 2025 Workshop on Foundation Models in Reasoning, 2025

    2025

  16. [24]

    Medcalc-bench: Evaluating large language models for medical calculations.Advances in Neural Information Processing Systems, 37:84730–84745, 2024

    Nikhil Khandekar, Qiao Jin, Guangzhi Xiong, Soren Dunn, Serina Applebaum, Zain Anwar, Maame Sarfo-Gyamfi, Conrad Safranek, Abid Anwar, Andrew Zhang, et al. Medcalc-bench: Evaluating large language models for medical calculations.Advances in Neural Information Processing System...

    2024

  17. [25]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  18. [26]

    How llms react to industrial spatio-temporal data? assessing hallucination with a novel traffic incident benchmark dataset

    Qiang Li, Mingkun Tan, Xun Zhao, Dan Zhang, Daoan Zhang, Shengzhao Lei, Anderson S Chu, Lujun Li, and Porawit Kamnoedboon. How llms react to industrial spatio-temporal data? assessing hallucination with a novel traffic incident benchmark dataset. InProceedings of the 2025 Conf...

    2025

  19. [27]

    Synthetic context generation for question generation.arXiv preprint arXiv:2406.13188, 2024

    Naiming Liu, Zichao Wang, and Richard Baraniuk. Synthetic context generation for question generation.arXiv preprint arXiv:2406.13188, 2024

    2024 arXiv

  20. [28]

    Understanding auc-roc curve.Towards data science, 26(1):220–227, 2018

    Sarang Narkhede. Understanding auc-roc curve.Towards data science, 26(1):220–227, 2018

    2018

  21. [29]

    An industrial application of a large language model to enhancing asset integrity and process safety management

    Benard Ongwae. An industrial application of a large language model to enhancing asset integrity and process safety management. 2025

    2025

  22. [30]

    Spiqa: A dataset for multimodal question answering on scientific papers.arXiv preprint arXiv:2407.09413, 2024

    Shraman Pramanick, Rama Chellappa, and Subhashini Venugopalan. Spiqa: A dataset for multimodal question answering on scientific papers.arXiv preprint arXiv:2407.09413, 2024

    2024 arXiv

  23. [31]

    Enhanced identification of equipment failures from descriptive accident reports using language generative model

    Unmesa Ray, Cristian Arteaga, Yonghan Ahn, and JeeWoong Park. Enhanced identification of equipment failures from descriptive accident reports using language generative model. Engineering, Construction and Architectural Management, 2024

    2024

  24. [32]

    Beacon: Benchmark for comprehensive rna tasks and language models.Advances in Neural Information Processing Systems, 37:92891–92921, 2024

    Yuchen Ren, Zhiyuan Chen, Lifeng Qiao, Hongtai Jing, Yuchen Cai, Sheng Xu, Peng Ye, Xinzhu Ma, Siqi Sun, Hongliang Yan, et al. Beacon: Benchmark for comprehensive rna tasks and language models.Advances in Neural Information Processing Systems, 37:92891–92921, 2024

    2024

  25. [33]

    Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Advancing multi-modal language models with a comprehen- sive dataset and benchmark for chain-of-thought reasoning.Advances in Neural Information Processing Systems,...

    2024

  26. [34]

    How to comply ai generated data with wells hse performance procedures and requirement in practice

    R Shirkhorshidi, N Norazman, MB Rosli, M Arriffin, and M Karbasian. How to comply ai generated data with wells hse performance procedures and requirement in practice. In SPE/IADC Middle East Drilling Technology Conference and Exhibition, page D011S006R002. SPE, 2023

    2023

  27. [35]

    Summarizing long regulatory documents with a multi-step pipeline

    Mika Sie, Ruby Beek, Michiel Bots, Sjaak Brinkkemper, and Albert Gatt. Summarizing long regulatory documents with a multi-step pipeline. InProceedings of the Natural Legal Language Processing Workshop 2024, pages 18–32, 2024

    2024

  28. [36]

    Legal reasoning.It’s all about IRAC, 2021

    Norman Otto Stockmeyer. Legal reasoning.It’s all about IRAC, 2021

    2021

  29. [37]

    Efficient unsupervised community search with pre-trained graph transformer.Proceedings of the VLDB Endowment, 17(9):2227–2240, 2024

    Jianwei Wang, Kai Wang, Xuemin Lin, Wenjie Zhang, and Ying Zhang. Efficient unsupervised community search with pre-trained graph transformer.Proceedings of the VLDB Endowment, 17(9):2227–2240, 2024

    2024

  30. [38]

    On llm-enhanced mixed-type data imputation with high-order message passing.arXiv preprint arXiv:2501.02191, 2025

    Jianwei Wang, Kai Wang, Ying Zhang, Wenjie Zhang, Xiwei Xu, and Xuemin Lin. On llm-enhanced mixed-type data imputation with high-order message passing.arXiv preprint arXiv:2501.02191, 2025

    2025 arXiv

  31. [39]

    Missing data imputation with uncertainty-driven network.Proceedings of the ACM on Management of Data, 2(3):1–25, 2024

    Jianwei Wang, Ying Zhang, Kai Wang, Xuemin Lin, and Wenjie Zhang. Missing data imputation with uncertainty-driven network.Proceedings of the ACM on Management of Data, 2(3):1–25, 2024

    2024

  32. [40]

    Professional readiness of llms in financial regulations? a report of regulations challenge at coling 2025

    Keyi Wang, Sarah Huang, Charlie Shen, Kaiwen He, Felix Tian, Jaisal Patel, Christina Dan Wang, Kairong Xiao, and Xiao-Yang Liu. Professional readiness of llms in financial regulations? a report of regulations challenge at coling 2025. InInternational Workshop on Multimodal Fin...

    2025

  33. [41]

    Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models

    Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee- Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. InProceedings of the 61st Annual Meeting of the Association for Computational Linguisti...

    2023

  34. [42]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. InThe Eleventh International Conference on Learning Representations

  35. [43]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  36. [44]

    Finben: A holistic financial benchmark for large language models.Advances in Neural Information Processing Systems, 37:95716–95743, 2024

    Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, et al. Finben: A holistic financial benchmark for large language models.Advances in Neural Information Processing Systems, 37:95716–95743, 2024

    2024

  37. [45]

    Smart construction sites: A promising approach to improving on-site hse management performance.Journal of Building Engineering, 49:104007, 2022

    Maozeng Xu, Xiuying Nie, Heng Li, Jack CP Cheng, and Zhongya Mei. Smart construction sites: A promising approach to improving on-site hse management performance.Journal of Building Engineering, 49:104007, 2022

    2022

  38. [46]

    isafe chatbot: Natural language processing and large language model driven construction safety learning through osha rules and video content delivery

    Syed Farhan Alam ZAIDI, Muhammad Sibtain ABBAS, Rahat HUSSAIN, Aqsa SABIR, KHAN Nasrullah, and YANG Jaehun. isafe chatbot: Natural language processing and large language model driven construction safety learning through osha rules and video content delivery. InInternational co...

    2024

  39. [47]

    Air-bench 2024: A safety benchmark based on regulation and policies specified risk categories

    Yi Zeng, Yu Yang, Andy Zhou, Jeffrey Ziwei Tan, Yuheng Tu, Yifan Mai, Kevin Klyman, Minzhou Pan, Ruoxi Jia, Dawn Song, et al. Air-bench 2024: A safety benchmark based on regulation and policies specified risk categories. InThe Thirteenth International Conference on Learning Re...

    2024

  40. [48]

    Laiw: A chinese legal large language models benchmark based on legal syllogism

    Ming Zhang, Zhen Liu, Jiahua Ma, et al. Laiw: A chinese legal large language models benchmark based on legal syllogism. InProceedings of the 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2024

    2024

  41. [49]

    Ontology-based semantic modeling of construction safety knowledge: Towards automated safety planning for job hazard analysis (jha)

    Sijie Zhang, Frank Boukamp, and Jochen Teizer. Ontology-based semantic modeling of construction safety knowledge: Towards automated safety planning for job hazard analysis (jha). Automation in construction, 52:29–41, 2015

    2015

  42. [50]

    Safetybench: Evaluating the safety of large language models.arXiv preprint arXiv:2309.07045, 2023

    Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. Safetybench: Evaluating the safety of large language models.arXiv preprint arXiv:2309.07045, 2023

    2023 arXiv

  43. [51]

    Lawbench: A benchmark for evaluating legal knowledge of large language models in chinese legal systems

    Zizheng Zheng, Liang Zhang, Yanbang Xu, et al. Lawbench: A benchmark for evaluating legal knowledge of large language models in chinese legal systems. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023

    2023

  44. [52]

    k shot of different [Scenario,Question,Options,Answer]

    Yujun Zhou, Jingdong Yang, Yue Huang, Kehan Guo, Zoe Emory, Bikram Ghosh, Amita Bedar, Sujay Shekar, Pin-Yu Chen, Tian Gao, et al. Labsafety bench: Benchmarking llms on safety issues in scientific labs.arXiv preprint arXiv:2410.14182, 2024. 13 A Evaluated LLMs In accordance wi...

  45. [56]

    Rule Conclusion: Present the final legal conclusion derived from the rule application. For each of these four IRAC components, generate a multiple choice question with only one correct answer (labeled as m1_question) For every question, provide: A correct answer (m1_answer). M...

  46. [57]

    Identify nuanced safety issues

  47. [58]

    Recall and articulate specific legal rules

  48. [59]

    Apply these rules with detailed legal analysis and inference

  49. [60]

    Reach a reasoned legal conclusion supported by specific regulatory references

  50. [61]

    region_name

    Critically evaluate adversarial answer options. Generate the complete set of questions, answers for each IRAC component and each question format. Court Case. We manually collect data from an online legal question-answering platform that contains a substantial number of court c...

  51. [62]

    Issue Spotting: Identify the critical safety or legal issue presented in the scenario

  52. [63]

    Rule Recall: State the relevant legal rule(s) or regulation(s) applicable to the identified issue

  53. [64]

    The explanation must include: - Specific inferences drawn from the facts

    Rule Application: Explain in detail how the legal rule applies to the given facts . The explanation must include: - Specific inferences drawn from the facts. - A detailed reasoning process that connects the facts to the rule. - A clear statement of the expected outcome based o...

  54. [65]

    For each of these four IRAC components, generate a multiple-choice question with only one correct answer (labeled as m1_question)

    Rule Conclusion: Present the final legal conclusion derived from the rule application. For each of these four IRAC components, generate a multiple-choice question with only one correct answer (labeled as m1_question). For every question, provide: - A correct answer (m1_answer)...

  55. [66]

    Correctness: The explanation must be factually accurate with no misstatements

  56. [67]

    Analysis: It must include detailed inferences and reasoning

  57. [68]

    Specificity: It must reference a specific, up-to-date HSE regulation (with name and clause number if applicable)

  58. [69]

    Additionally, include distractor options (e.g

    Uniqueness: It must be unique and tailored to the scenario. Additionally, include distractor options (e.g. trap option m1_trap_answer) with subtle errors. In the explanations, detail why the correct answer is valid and identify the misleading elements in distractors. Generate ...

    2024

  59. [70]

    It should sound like a real internal incident summary or audit finding

    Write the scenario in clear, well-organized prose. It should sound like a real internal incident summary or audit finding. Use a **natural, professional, and concise tone**

  60. [71]

    Do **not** mention any legal statute or regulation by name

  61. [72]

    Make the issue **non-obvious**: include some context that may relate to incorrect options, creating plausible confusion

  62. [73]

    Ensure the scenario includes at least one complicating detail (e.g., conflicting records, partial compliance, or unclear accountability)

  63. [74]

    Do **not** change the meaning of the correct answer, and do **not** modify the answer options

  64. [75]

    {question}

    Do not remove or alter any factual information such as the location, job roles, entities involved, or the outcome of the incident, but remove confidential information. The original question is: "{question}" The answer choices are: "{options}" The correct answer is option "{ans...

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.