Pith. sign in

REVIEW 4 major objections 5 minor 30 references

A RAG pipeline can automate regulatory compliance checks with 78% F1 and 85% recall.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A RAG pipeline with tuned chunking, retrieval depth, and in-context examples reports 78% F1 for automated compliance checking, but the evaluation has no held-out validation and a post-hoc No-Evidence-to-Non-Compliant reclassification.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Useful RAG engineering with a real deployment story, but the headline F1/recall is uninterpretable because the three-class metrics contradict the paper's own binary-label description. the 4 major comments →

arxiv 2608.02472 v1 pith:DL7XWHTK submitted 2026-08-03 cs.CL

CTRAG: An In-Context Retrieval-based Framework for Automated Compliance Checking using LLMs

Muhammad Roman , Karen Rafferty , Barry Devereux This is my paper
classification cs.CL
keywords automated compliance checkingretrieval-augmented generationin-context learningdocument chunkingregulatory controlsquestion answeringLLM evaluationNo-Evidence reclassification
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that regulatory compliance checking, traditionally a slow manual audit of company documents, can be largely automated with a retrieval-augmented generation (RAG) pipeline. It turns each regulatory control into a question, retrieves relevant chunks from company PDFs, and asks an LLM to answer whether the company passes. After tuning chunk size, retrieval depth, and in-context examples, the best configuration reports 78% F1 and 85% recall on 240 controls, with a 'No-Evidence' label recoded as non-compliant to avoid missing failures. If these numbers hold outside the single-organization test set, the approach could cut manual reviewer effort substantially — the paper reports about 60% reduction — and generalize to adjacent document-analysis tasks like policy gap analysis and contract review.

Core claim

On its own terms, the paper's central discovery is that retrieval configuration matters more than model choice or context size for compliance checking: small fixed-size chunks of 800 characters with four retrieved chunks (K=4), combined with curated in-context examples, consistently outperformed larger chunks and meta-chunking across five LLMs. The hardest class is Non-Compliant; direct prompting misses most failures, and the paper fixes this by reclassifying 'No-Evidence' answers as Non-Compliant, which raises recall to 85% at 78% F1 in the final deployment.

What carries the argument

The load-bearing mechanism is a question-answering framing of compliance: each regulatory control (e.g., 'Do incident procedures assign roles?') is posed to a RAG pipeline that retrieves the top-K chunks from company documents and asks an LLM to answer Pass/Fail with the retrieved text as evidence. Three components carry the argument: adaptive fixed-size chunking that keeps chunks small enough to avoid diluting relevant evidence; a retrieval depth K=4 that balances context and noise; and in-context learning examples that teach the model human decision rules, especially for indirect compliance via third-party services. The final piece is a conservative reclassification rule: when the model fi

Load-bearing premise

The paper assumes that the configuration that scored best on the single 240-control test set — including the No-Evidence reclassification — will perform as well in practice, without a separate held-out set, cross-validation, or significance testing to support that generalization.

What would settle it

Take the final configuration (Fixed800 chunks, K=4, curated in-context examples, No-Evidence reclassified as Non-Compliant) and run it on a held-out set of controls drawn from a second organization with independent expert labels; if recall falls well below 85% or F1 below 78% — or if experts judge many reclassified No-Evidence cases to be genuinely compliant — the deployed-performance claim is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • On the paper's reported numbers, an automated system could replace the bulk of manual compliance review, with about a 60% reduction in reviewer effort at a large professional services firm.
  • Small chunks with moderate retrieval depth beat larger or semantically segmented chunks on this task, suggesting that evidence granularity, not context volume, drives compliance accuracy.
  • In-context learning closes most of the gap between direct prompting and human annotators on nuanced indirect-compliance cases.
  • The No-Evidence-to-Non-Compliant recoding is what lifts recall from roughly 60-70% to 85%, so the reported high recall is contingent on this design decision.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 78% F1 / 85% recall figures are measured on the same 240 controls used to select the configuration; a fresh held-out evaluation across other organizations or regulatory domains is needed before treating them as stable deployment-level estimates.
  • The reclassification rule is a policy choice: in domains where missing documentation does not imply non-compliance, this rule would inflate false positives; the paper's recall gain may not transfer to such settings.
  • The reported monotonic decline in F1 with larger chunk size hints that the 'needle-in-a-haystack' effect is task-specific; for compliance queries requiring multi-clause or cross-document reasoning, larger or hierarchical chunks might still be necessary.
  • A testable extension would be to publish a small non-proprietary version of the control-to-document dataset (with anonymized docs) so other groups can benchmark chunking and ICL strategies on the same controls.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CTRAG, a Retrieval-Augmented Generation pipeline for automated compliance checking. It frames each regulatory control as a question, retrieves relevant passages from company documents using adaptive chunking and a hybrid retriever, and prompts an LLM with in-context examples to produce a compliance decision. Experiments on a proprietary dataset of 240 controls and 45 documents compare chunk sizes, retrieval depths, five LLMs, and direct prompting versus in-context learning. The authors report that the best configuration (GPT-4, 800-character chunks, K=4, with ICL) reaches a weighted F1 of 71.09% and, after reclassifying No-Evidence outputs as Non-Compliant, an overall F1 of 78.05% with 85.33% recall. They also claim a 60% reduction in manual reviewer effort in a Big Four deployment. The central evaluation, however, suffers from a fundamental inconsistency: the dataset is stated to have binary ground-truth labels (Pass/Fail), while the reported class-wise metrics include a No-Evidence class with nonzero precision and recall, which is arithmetically impossible under a binary label space. Additionally, the same evaluation set is used for configuration selection and final reporting, with no held-out validation or significance testing.

Significance. If the reported results were reliable, CTRAG would be a practically relevant contribution to automated compliance checking, an underexplored application of RAG with high real-world stakes. The paper addresses a genuine gap and includes useful systematic comparisons of chunking and retrieval depth. However, the significance is severely undermined by the evaluation issues: the headline F1 and recall figures are not reproducible from the stated label definitions, and the final configuration is selected on the same 240 controls used to compute the performance metrics. The limitations section candidly acknowledges the small, single-organization dataset and the lack of significance tests, but the core performance claim remains unsupported as reported.

major comments (4)
  1. [Section 4 vs Tables 2–7] The dataset is described as having binary ground-truth labels, Pass or Fail (Section 4). Under a binary ground truth, no instance has a true label of No-Evidence, so every No-Evidence prediction is a false positive; the No-Evidence class has zero true positives, and its precision and recall must be 0 or undefined. Yet Tables 2–7 report nonzero No-Evidence precision, recall, and F1 throughout; for example, Table 7 (GPT-4 Small) reports No-Evidence P=37.93, R=62.86, F1=47.31. This is mathematically impossible unless a third true label exists, but no such label is defined. Consequently, the weighted F1=71.09 in Table 7 and the post-reclassification F1=78.05 / recall=85.33 in Table 8 cannot be reproduced or interpreted from the stated label space. The abstract's central claim—'F1-score of 78% and a recall of 85%'—rests on an evaluation whose label definitions are internally inconsistent.
  2. [Section 6 and Section 7] The reported 'final deployed configuration' was obtained by selecting the chunk size, retrieval depth K=4, the ICL examples, and the No-Evidence-to-Non-Compliant reclassification rule using the same 240 controls on which the final metrics are computed. No held-out split, cross-validation, or significance test is performed; the conclusion explicitly defers statistical significance tests to future work. The headline performance is therefore the outcome of tuning on the evaluation set and is an optimistically biased estimate of deployed accuracy. The claim that the reported 78% F1 / 85% recall is a reliable deployed-level estimate is unsupported by the experimental design.
  3. [Section 6, reclassification rule] The final solution reclassifies all No-Evidence predictions as Non-Compliant, justified by the statement that absence of relevant information 'typically implies' non-compliance. This rule is not validated against cases where documents simply omit discussion, where retrieval failed, or where the evidence was not retrieved for reasons unrelated to actual compliance. The paper's own hit-rate analysis treats the model's No-Evidence output as a proxy for retrieval failure ('hit rate was calculated on the basis of the generated response where a No-Evidence is considered to be missing information'), so the distinction between retrieval failure, missing evidence, and true non-compliance is never established. The recall gain in Table 8 is thus an artifact of an unvalidated decision rule rather than a measured property of the system.
  4. [Section 7, deployment claim] The conclusion claims an 'approximately 60% reduction in manual reviewer effort' in a Big Four professional services deployment, but the paper provides no methodology, baseline definition, measurement protocol, or supporting data for this metric. Since the abstract presents reduced manual reviewer effort as a key benefit, this is a load-bearing claim that needs at least a brief description of how it was measured and compared to the existing process.
minor comments (5)
  1. [Abstract] 'CTRAG value' should be 'CTRAG's value' for grammatical correctness.
  2. [Section 4 and Figure 2] Minor typos: '45PDF documents' is missing a space; Figure 2's caption uses 'clients documents' instead of 'client's documents'.
  3. [Tables 2–8] Class-level counts (true positives, false positives, false negatives) are never reported, so the precision/recall/F1 values in Tables 2–8 cannot be independently verified. Reporting confusion matrices or per-class support would substantially improve reproducibility.
  4. [Section 6, hit rate] The hit-rate discussion ('approximately 22% using smaller chunks, compared to the actual hit rate of 80%') is vague: the reader is not told what 'actual hit rate' means, how it was measured, or why the lower hit rate is attributed to chunk size rather than the model's response behavior. This should be clarified.
  5. [Section 5.3] The paper mentions using LangChain RetrievalQA with 'stuff' and 'map_reduce' chains, but gives no details on prompt templates, temperature, or decoding settings. Reporting these settings is important for replicability of LLM-based results.

Circularity Check

3 steps flagged

Headline F1/recall are selected on the test set and rest on an undefined No-Evidence label; the abstract's central deployment claim is not independently derivable.

specific steps
  1. fitted input called prediction [Section 6 (ICL results, Tables 7–8)]
    "The configurations reported in Table 7 reflect a deliberate narrowing of the experimental space. ... For these experiments, we fixed the retrieval depth at K = 4 based on the findings from the direct-prompting experiments. ... Using this configuration, Fixed800 chunks yielded the strongest overall results across the compliance classes, particularly when paired with GPT-4."

    The same 240-control corpus is the only evaluation set ('the dataset is exclusively used for testing, with no portion allocated for training'). Chunk size, K, ICL examples, and the reclassification rule were selected by comparing their performance on this same set. Table 8 then reports the 'final deployed configuration' metrics. The headline F1=78% and recall=85% are therefore the best values found on the test data, not out-of-sample predictions: the reported performance is the fitted value by construction.

  2. other [Section 4 (Dataset) vs Tables 2–7]
    "For each control, the dataset also includes binary ground truth responses, labelled as 'Pass' or 'Fail,' indicating compliance or non-compliance ... only the binary 'Pass' or 'Fail' responses were used as the ground truth answer in this study. ... [Table 7, GPT-4 Small row:] No-Evidence P(%) R(%) F1(%) ... 37.93 62.86 47.31"

    Under a binary Pass/Fail ground truth, no true instance has label 'No-Evidence', so any No-Evidence prediction can only be a false positive: precision is 0 or undefined, recall is 0, and F1 is 0. Table 7 reports large nonzero No-Evidence precision, recall, and F1. The class-wise metrics, the weighted average in Table 7, and the Table 8 headline therefore assume a third ground-truth label that the paper's own dataset definition denies. The claimed 78% F1 / 85% recall are not derivable from the stated binary labels; they are defined only under an unstated label space.

  3. fitted input called prediction [Section 6, No-Evidence reclassification paragraph before Table 8]
    "In the final solution, however, No-Evidence classifications were reclassified as Non-Compliant, as the absence of relevant information typically implies that the vendor does not comply ... This reclassification approach significantly improved recall for the Non-Compliant class ... By addressing the No-Evidence cases in this manner, we captured many previously missed compliance fail cases, achieving a more robust and realistic compliance-checking solution."

    The recall gain is an arithmetic consequence of the post-hoc relabeling: every 'No-Evidence' prediction is converted to 'Non-Compliant', so previously missed Fail cases become true positives by rule rather than by retrieved evidence. The rule was introduced after inspecting the error pattern on the evaluation corpus, and there is no independent validation that 'absence of evidence' equals non-compliance. The 85% recall reported as the deployed performance is thus a property of a test-set-fitted relabeling, not a measured property of the retrieval-and-generation pipeline.

full rationale

The central claim in the abstract—'CTRAG achieving an F1-score of 78% and a recall of 85% in the final deployed configuration'—is not an independent estimate. The configuration was chosen by comparing configurations on the same 240-control set that produced the final numbers, with no held-out split, so the headline performance is selected-on-test rather than predicted. The reported recall is further inflated by a No-Evidence-to-Non-Compliant reclassification that was adopted after seeing which cases were missed; converting abstentions into non-compliance mechanically raises recall. In addition, Tables 2–7 report nonzero precision/recall/F1 for a 'No-Evidence' class even though Section 4 states that the ground truth is binary Pass/Fail only; such metrics are arithmetically impossible under the stated labels. This makes the reported evaluation internally incoherent as well as fitted. I found no load-bearing self-citation chain, no imported uniqueness theorem, and no ansatz-smuggling via the authors' prior work; the circularity arises from test-set fitting and an undefined label space, not from the references.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The central performance claim rests on multiple hyperparameters (chunk size, K, ICL examples, reclassification rule) that were tuned on the same evaluation set, plus strong domain assumptions about label reliability and the No-Evidence-to-Non-Compliant mapping.

free parameters (5)
  • Chunk size (Small=800 chars) = 800 characters
    Selected as the best-performing chunk size on the 240-control test set (Tables 7-8); no validation set.
  • Retrieval depth K = 4
    Fixed at K=4 based on exploratory results on the same test set (Section 6).
  • In-context learning examples = not specified
    Curated examples used in prompts; selection criteria are not described, and they are refined using error analysis on the same test set.
  • No-Evidence reclassification rule = No-Evidence → Non-Compliant
    Applied post hoc to improve Non-Compliant recall; acknowledged to trade precision for recall (Section 6, Conclusion).
  • Embedding model choice = text-embedding-3-large / ada-002
    Only two embedding models tested; the choice between them is not rigorously evaluated.
axioms (4)
  • domain assumption Ground-truth Pass/Fail labels produced by expert analysts are correct.
    No inter-annotator agreement measured (Conclusion); dataset is proprietary and used as the only evaluation set.
  • ad hoc to paper No-Evidence in retrieved context implies Non-Compliant.
    Applied in the final reclassification (Section 6); this is a domain judgment that may not hold generally and inflates recall.
  • domain assumption The 45 company PDFs and 240 controls are representative of the compliance-checking task at scale.
    Single organization, single domain; authors acknowledge broader testing needed (Conclusion).
  • ad hoc to paper The generated response's 'No-Evidence' label is a valid proxy for retrieval failure.
    The hit rate was computed from responses rather than comparing retrieved chunks to ground-truth evidence documents (Section 6).

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of CTRAG: An In-Context Retrieval-based Framework for Automated Compliance Checking using LLMs." pith.science (2026). https://pith.science/paper/DL7XWHTK

@misc{pith2026260802472,
  author       = {Pith},
  title        = {Pith review of: CTRAG: An In-Context Retrieval-based Framework for Automated Compliance Checking using LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DL7XWHTK}},
  note         = {Machine review of arXiv:2608.02472}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Trust is fundamental in modern regulatory ecosystems, and compliance checking plays a critical role in fostering that trust. Regulatory compliance verification is essential for businesses operating in highly controlled environments, as it ensures alignment with sector-specific guidelines across domains such as financial reporting, data privacy, and cybersecurity. Manual compliance testing, however, is often time-intensive and prone to inconsistencies, particularly when compliance depends indirectly on third-party services such as cloud providers, where vendors rely on external providers to meet regulatory standards. In this paper, we present CTRAG, a novel Retrieval-Augmented Generation (RAG) pipeline designed for automated compliance checking. CTRAG employs advanced strategies, including adaptive chunking, dynamic retrieval configurations, and in-context learning, to improve the precision and relevance of compliance assessments. By extracting control questions from regulatory texts and cross-referencing them with unstructured company documentation, CTRAG achieves highly accurate, document-informed compliance verification, even in cases of indirect compliance through third-party services. Empirical evaluations demonstrate significant improvements, with CTRAG achieving an F1-score of 78% and a recall of 85% in the final deployed configuration, ensuring minimal missed non-compliance cases while reducing manual reviewer effort in a real-world deployment. To validate CTRAG value, we developed and deployed a POC within a Big Four professional services firm, applying it to real-world cases and cross-checking results against manual compliance reports. These findings highlight CTRAG potential to streamline compliance workflows, mitigate risks, and enhance regulatory trust in complex, high-stakes environments.

Figures

Figures reproduced from arXiv: 2608.02472 by Barry Devereux, Karen Rafferty, Muhammad Roman.

Figure 1
Figure 1. Figure 1: Flow of third party compliance checking performed manually by a compliance tester and validated 2. RELATED WORK Automated compliance checking (ACC) has been an active area of re￾search for several decades, driven by the growing complexity of regula￾tions and the increasing cost of manual compliance assessment. Early work focused primarily on formal and rule-based approaches that sought to represent regulat… view at source ↗
Figure 2
Figure 2. Figure 2: RAG-based approach for compliance checking using clients documents and regulatory questions configuring each component to handle the complexity and specificity of compliance testing. By combining retrieval and generation, RAG enables the system to locate precise answers in documents and present them as coherent responses, reducing the need for manual compliance audits. This approach ensures consistency and… view at source ↗
Figure 3
Figure 3. Figure 3: Processing times across models and chunking strategies, with generation time (a–e) and chunking time (f) [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: F1 score performance after in-context learning across four LLMs and three chunk sizes [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Heatmap of F1 scores across models and chunking strategies, highlighting performance variations process. These findings suggest the approach is well suited to scal￾ing beyond the single-organization deployment evaluated here, and to adjacent document-analysis tasks such as policy gap analysis and contract review. The study has several limitations. The evaluation is based on 240 controls and 45 documents fr… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 3 linked inside Pith

  1. [1]

    Retrieval- 8–10 arXiv Preprint CTRAG: An In-Context Retrieval-based Framework for Automated Compliance Checking using LLMs augmentedgenerationforknowledge-intensivenlptasks,

    P.Lewis,E.Perez,A.Piktus,F.Petroni,V.Karpukhin,N.Goyal, H.Küttler,M.Lewis,W.-t.Yih,T.Rocktäscheletal.,“Retrieval- 8–10 arXiv Preprint CTRAG: An In-Context Retrieval-based Framework for Automated Compliance Checking using LLMs augmentedgenerationforknowledge-intensivenlptasks,”Ad- vancesinneuralinformationprocessingsystems,vol.33,pp.9459– 9474,2020

    2020

  2. [2]

    Lan- guagemodelsarefew-shotlearners,

    T.Brown,B.Mann,N.Ryder,M.Subbiah,J.D.Kaplan,P.Dhari- wal,A.Neelakantan,P.Shyam,G.Sastry,A.Askelletal.,“Lan- guagemodelsarefew-shotlearners,”Advancesinneuralinfor- mationprocessingsystems,vol.33,pp.1877–1901,2020

    1901

  3. [3]

    Themysteryof in-contextlearning: Acomprehensivesurveyoninterpretation andanalysis,

    Y.Zhou,J.Li,Y.Xiang,H.Yan,L.Gui,andY.He,“Themysteryof in-contextlearning: Acomprehensivesurveyoninterpretation andanalysis,”inProceedingsofthe2024ConferenceonEmpirical MethodsinNaturalLanguageProcessing,2024,pp.14365–14378

    2024

  4. [4]

    Businessprocesscompliancechecking: Currentstateand futurechallenges,

    M.e.Kharbili,A.K.A.d.Medeiros,S.Stein,andW.M.vander Aalst,“Businessprocesscompliancechecking: Currentstateand futurechallenges,”ModellierungbetrieblicherInformationssys- teme(MobIS2008),pp.107–113,2008

    2008

  5. [5]

    Are wedonewithbusinessprocesscompliance? stateoftheartand challengesahead,

    M.Hashmi, G.Governatori, H.-P.Lam, andM.T.Wynn, “Are wedonewithbusinessprocesscompliance? stateoftheartand challengesahead,”KnowledgeandInformationSystems,vol.57, no.1,pp.79–133,2018

    2018

  6. [6]

    Automated building in- formationmodelingcompliancecheckthroughalargelanguage model combined with deep learning and ontology,

    N. Chen, X. Lin, H. Jiang, and Y. An, “Automated building in- formationmodelingcompliancecheckthroughalargelanguage model combined with deep learning and ontology,”Buildings, vol.14,no.7,p.1983,2024

    1983

  7. [7]

    The promise of automated compli- ancechecking,

    R. Amor and J. Dimyadi, “The promise of automated compli- ancechecking,”Developmentsinthebuiltenvironment,vol.5,p. 100039,2021

    2021

  8. [8]

    Digitalapproaches toconstructioncompliancechecking: Validatingthesuitability ofanecosystemapproachtocompliancechecking,

    T.Beach,J.Yeung,N.Nisbet,andY.Rezgui,“Digitalapproaches toconstructioncompliancechecking: Validatingthesuitability ofanecosystemapproachtocompliancechecking,”Advanced EngineeringInformatics,vol.59,p.102288,2024

    2024

  9. [9]

    Asecureframeworkforcontinuouscomplianceacrossheteroge- neouspolicyvalidationpoints,

    T.Yanagawa,V.Agarwal,Y.Watanabe,L.Degenaro,andA.Sailer, “Asecureframeworkforcontinuouscomplianceacrossheteroge- neouspolicyvalidationpoints,”in2024IEEE17thInternational ConferenceonCloudComputing(CLOUD),2024,pp.176–182

    2024

  10. [10]

    Software-as-a-service security chal- lenges and best practices: A multivocal literature review,

    M.Humayun,M.Niazi,M.F.Almufareh,N.Z.Jhanjhi,S.Mah- mood, and M. Alshayeb, “Software-as-a-service security chal- lenges and best practices: A multivocal literature review,”Ap- pliedSciences,vol.12,no.8,p.3953,2022

    2022

  11. [11]

    A rule-based se- mantic approach for automated regulatory compliance in the construction sector,

    T. H. Beach, Y. Rezgui, H. Li, and T. Kasim, “A rule-based se- mantic approach for automated regulatory compliance in the construction sector,”Expert systems with applications, vol. 42, no.12,pp.5219–5231,2015

    2015

  12. [12]

    Leveragingartificialintelligenceforenhancingregulatorycom- plianceinthefinancialsector,

    V.Jain,A.Balakrishnan,D.Beeram,M.Najana,andP.Chintale, “Leveragingartificialintelligenceforenhancingregulatorycom- plianceinthefinancialsector,”InternationalJournalofComputer TrendsandTechnology,vol.72,no.5,pp.116–125,2024

    2024

  13. [13]

    Towardsautomatedregulatorycompliance verification in financial auditing with large language models,

    A. Berger, L. Hillebrand, D. Leonhard, T. Deußer, T. B. F. De Oliveira, T. Dilmaghani, M. Khaled, B. Kliem, R. Loitz, C.Bauckhageetal.,“Towardsautomatedregulatorycompliance verification in financial auditing with large language models,” in2023 IEEE International Conference on Big Data (BigData). IEEE,2023,pp.4626–4635

    2023

  14. [14]

    Semantic text classifica- tionforsupportingautomatedcompliancecheckinginconstruc- tion,

    D. M. Salama and N. M. El-Gohary, “Semantic text classifica- tionforsupportingautomatedcompliancecheckinginconstruc- tion,”JournalofComputinginCivilEngineering,vol.30,no.1,p. 04014106,2016

    2016

  15. [15]

    Semantic nlp-based informa- tionextractionfromconstructionregulatorydocumentsforau- tomated compliance checking,

    J. Zhang and N. M. El-Gohary, “Semantic nlp-based informa- tionextractionfromconstructionregulatorydocumentsforau- tomated compliance checking,”Journal of computing in civil engineering,vol.30,no.2,p.04015014,2016

    2016

  16. [16]

    Ontology-basedautomatedinforma- tion extraction from building energy conservation codes,

    P.ZhouandN.El-Gohary,“Ontology-basedautomatedinforma- tion extraction from building energy conservation codes,”Au- tomationinConstruction,vol.74,pp.103–117,2017

    2017

  17. [17]

    Building information modeling, naturallanguageprocessing,andartificialintelligenceforauto- matedcompliancechecking,

    R. Zhang and N. El-Gohary, “Building information modeling, naturallanguageprocessing,andartificialintelligenceforauto- matedcompliancechecking,”inResearchcompaniontobuilding informationmodeling. EdwardElgarPublishing,2022,pp.248– 267

    2022

  18. [18]

    The role of machine learning in automatedcodechecking-asystematicliteraturereview,

    M. Alnuzha and T. Bloch, “The role of machine learning in automatedcodechecking-asystematicliteraturereview,”Journal of Information Technology in Construction, vol. 30, pp. 22–44, 2025

    2025

  19. [19]

    Unpackingambiguityinbuild- ingrequirementstosupportautomatedcompliancechecking,

    Z.Zhang,L.Ma,andN.Nisbet,“Unpackingambiguityinbuild- ingrequirementstosupportautomatedcompliancechecking,” JournalofManagementinEngineering,vol.39,no.5,p.04023033, 2023

    2023

  20. [20]

    From complexity to clarity: Ai/nlp’s role in regulatory compliance,

    J. Jain, N. Dhanasekaran, and M. Diab, “From complexity to clarity: Ai/nlp’s role in regulatory compliance,” inFindingsof theAssociationforComputationalLinguistics: ACL2025,2025, pp.26629–26641

    2025

  21. [21]

    Leveraginglargelanguage models for BIM-based automated compliance checking,

    J.Dimyadi,R.Amor,andW.Solihin,“Leveraginglargelanguage models for BIM-based automated compliance checking,”Au- tomationinConstruction,vol.170,p.106707,2025

    2025

  22. [22]

    Asystematicliterature reviewofretrieval-augmentedgeneration: Techniques,metrics, andchallenges,

    A.Brown,M.Roman,andB.Devereux,“Asystematicliterature reviewofretrieval-augmentedgeneration: Techniques,metrics, andchallenges,”BigDataandCognitiveComputing,vol.9,no.12, p.320,2025

    2025

  23. [23]

    A survey onknowledge-orientedretrieval-augmentedgeneration,

    M. Cheng, Y. Luo, J. Ouyang, Q. Liu, H. Liu, L. Li, S. Yu, B. Zhang, J. Cao, J. Ma, D. Wang, and E. Chen, “A survey onknowledge-orientedretrieval-augmentedgeneration,”arXiv preprintarXiv:2503.10677,2025

    Pith/arXiv arXiv 2025

  24. [24]

    Meta-chunking: Learningtextsegmentationandsemanticcom- pletionvialogicalperception,

    J.Zhao,Z.Ji,Y.Feng,P.Qi,S.Niu,B.Tang,F.Xiong,andZ.Li, “Meta-chunking: Learningtextsegmentationandsemanticcom- pletionvialogicalperception,”arXivpreprintarXiv:2410.12788, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    LongRAG:Adual-perspectiveretrieval-augmentedgeneration paradigmforlong-contextquestionanswering,

    Q.Zhao,R.Wang,Y.Cen,D.Zha,S.Tan,Y.Dong,andJ.Tang, “LongRAG:Adual-perspectiveretrieval-augmentedgeneration paradigmforlong-contextquestionanswering,”inProceedingsof the2024ConferenceonEmpiricalMethodsinNaturalLanguage Processing. AssociationforComputationalLinguistics, 2024, pp.22600–22632

    2024

  26. [26]

    Rag4itops: Asupervisedfine-tunableandcomprehensiverag frameworkforitoperationsandmaintenance,

    T. Zhang, Z. Jiang, S. Bai, T. Zhang, L. Lin, Y. Liu, and J. Ren, “Rag4itops: Asupervisedfine-tunableandcomprehensiverag frameworkforitoperationsandmaintenance,”inProceedingsof the2024ConferenceonEmpiricalMethodsinNaturalLanguage Processing: IndustryTrack,2024,pp.738–754

    2024

  27. [27]

    Rirag: Regula- toryinformationretrievalandanswergeneration,

    T.Gokhan,K.Wang,I.Gurevych,andT.Briscoe,“Rirag: Regula- toryinformationretrievalandanswergeneration,”arXivpreprint arXiv:2409.05677,2024

    Pith/arXiv arXiv 2024

  28. [28]

    Enhancingregulatorycompliancethroughau- tomatedretrieval,reranking,andanswergeneration,

    K.Umar,H.Doğan,O.Özcan,I.Karakaya,A.Karamanlıoğlu, andB.Demirel,“Enhancingregulatorycompliancethroughau- tomatedretrieval,reranking,andanswergeneration,”inProceed- ingsofthe1stRegulatoryNLPWorkshop(RegNLP2025),2025,pp. 91–96. 9–10 CTRAG: An In-Context Retrieval-based Framework for Automated Compliance Checking using LLMs arXiv Preprint

    2025

  29. [29]

    RAVEN: In-context learning with retrieval- augmented encoder-decoder language models,

    J. Huang, W. Ping, P. Xu, M. Shoeybi, K. C.-C. Chang, and B. Catanzaro, “RAVEN: In-context learning with retrieval- augmented encoder-decoder language models,”Transactions onMachineLearningResearch,2024

    2024

  30. [30]

    Billion-scale similarity searchwithGPUs,

    J. Johnson, M. Douze, and H. Jégou, “Billion-scale similarity searchwithGPUs,”IEEETransactionsonBigData,vol.7,no.3, pp.535–547,2021. APPENDIX .1. Complete K-level breakdown of chunking and generation time Table 9.Chunking and Generation Times Across Models and Chunking Strategies Chunking Chunking Generation Times Avg. Strategy Time K=1 K=3 K=5 GPT-3.5 Tu...

    2021

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.