Pith. sign in

REVIEW 3 major objections 6 minor 38 references

From Regulation to Requirements: An Automated Requirement Derivation and Explanation Pipeline

T0 review · 3 major / 6 minor · reviewed 2026-07-11 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An automated pipeline can turn entire regulatory documents into system-agnostic software requirements with plain-language explanations that engineers without legal training can act on.

desk verdict Solid end-to-end RE pipeline with open tool and full-document eval; moderate correctness and non-legal ground truth are real but do not sink the contribution. read the letter →

arxiv 2607.04448 v1 pith:JYA5TU3W submitted 2026-07-05 cs.SE cs.AI

Pavithra PM Nair , Preethu Rose Anish This is my paper
classification cs.SEcs.AI
keywords requirementsengineeringregulatorycompliancenaturallanguageprocessinglargemodelsGDPREUAIActrequirementderivationtraceability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Translating dense legal text such as the GDPR or the EU AI Act into software requirements is still done largely by hand and is error-prone. This paper claims that process can be automated end-to-end: first identify which clauses actually imply observable system behaviour, then derive the minimum set of system-agnostic requirements together with plain-language explanations that stay traceable to the source. On the full clause sets of both regulations the identification stage reaches macro F1 scores of 0.82 and 0.78, beating a strong sentence-classifier baseline, while human raters give the derived requirements high completeness scores and the explanations near-ceiling clarity. A study with 25 practitioners shows the explanations significantly raise both comprehension and confidence to implement or test the requirements. The pipeline is released as an open tool that also supplies classification, use-case seeds, definition indexing and a full traceability matrix so teams can start compliance work from the regulation itself rather than from a blank page.

What carries the argument

The regulation-to-requirements pipeline: zero-shot identification of requirement-bearing clauses followed by a single joint generation step that emits the minimum necessary software requirements, their plain-language explanations, and typed cross-references, all from clause text alone.

What would settle it

Have a panel of legal experts re-annotate the same full GDPR and EU AI Act clause sets and re-score the derived requirements; if identification F1 falls below the SetFit baseline or explanation clarity drops once the outputs must satisfy real audit criteria, the central claim fails.

Watch

Extended reading notes

Core claim

A two-stage large-language-model pipeline that first filters every clause of a regulation for requirement-bearing content and then jointly derives system-agnostic software requirements plus plain-language explanations can produce outputs of high completeness and near-ceiling explanation clarity across the full GDPR and EU AI Act, and those explanations measurably improve practitioner comprehension and confidence.

Load-bearing premise

Both the labels for which clauses count as requirement-bearing and the quality scores for derived requirements and explanations rest solely on the judgement of requirements engineers who have no formal legal training.

Editorial extensions

If this is right

  • When a regulation is amended, re-running the pipeline plus the traceability matrix immediately surfaces which derived requirements are affected.
  • Requirements engineers without legal training can begin compliance work from an automated first draft instead of reading every clause manually.
  • The same regulation-agnostic prompts can be applied to newly enacted laws without custom configuration or re-training.
  • Plain-language explanations become a shared artefact that developers, testers and auditors can use when implementing or checking compliance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the identification stage’s known false-positive bias is tolerable, teams may prefer over-generation followed by human pruning over the risk of silently dropping obligations.
  • Linking the pipeline to existing requirements-management tools would turn it into a continuous compliance monitor that flags drift whenever source text changes.
  • Success on EU drafting conventions does not guarantee the same surface-cue strategy will work on rights- and exception-heavy common-law statutes; that is a direct next test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents Reg2Req, a two-stage LLM pipeline that (i) identifies requirement-bearing clauses in full regulatory documents and (ii) derives system-agnostic software requirements with plain-language explanations and typed cross-references. Evaluated on all 398 GDPR and 574 EU AI Act clauses, identification reaches macro F1 0.82/0.78 (outperforming a SetFit baseline); human ratings report high completeness (4.60/4.45), moderate correctness (3.74/3.54), and near-ceiling explanation clarity (4.92/4.94). An ablation argues the identification stage is necessary; a 25-practitioner within-subjects study finds explanations improve comprehension and confidence (p < 0.001). A public tool adds FR/NFR classification, use-case seeding, definition indexing, and a traceability matrix.

Significance. If the results hold under stronger validation, this is a useful systems contribution to regulatory RE: it treats requirement-bearing clause identification as an explicit, full-document RE stage (distinct from legal obligation mining), evaluates on complete clause sets rather than pre-selected articles, and pairs derivation with engineer-facing explanations whose benefit is measured in a controlled user study. Public release of Reg2Req, prompts, and annotated GDPR/AI Act datasets is a concrete community asset and strengthens reproducibility. The work is timely given expanding EU and global digital regulation, and the supporting workflow features (traceability matrix, definition indexing) address practical adoption barriers even if they are secondary contributions.

major comments (3)
  1. Table III (Section IV-C): requirement correctness is only moderate (3.74 GDPR / 3.54 EU AI Act) with the largest SDs, and the paper itself attributes failures to unresolved exceptions, conditions, and rights-granting clauses (e.g., GDPR-28-3/R5 restating an unresolved cross-reference). Completeness and clarity are high, but the central claim is production of correct, actionable software requirements from legal text. Either elevate a mitigation already listed as future work (e.g., retrieval over cross-referenced provisions) into the evaluated pipeline, or reframe claims more tightly as draft/seed requirements that require expert review, with quantitative characterization of when correctness fails.
  2. Sections IV-A1, IV-A2, and VII-B: ground-truth labels for both requirement-bearing identification and quality ratings were produced solely by requirements engineers without formal legal training (Krippendorff α ≈ 0.71–0.81 on overlaps). The paper correctly notes this yields an RE interpretation rather than legally authoritative labels, but that caveat undercuts the compliance-facing claim. A stratified legal-expert adjudication (even on a modest sample of edge cases: rights, exceptions, cross-reference-heavy clauses) is needed to secure construct validity of both the identification task and correctness scores.
  3. Table IV / Section IV-D (RQ3): the necessity of the upstream identification stage rests entirely on LLM-as-judge scores for the ablated condition. Spearman ρ 0.77–0.85 is reported only against full-pipeline human ratings; ablated outputs are larger and likely differently distributed (more non-requirement clauses forced through derivation). A human-rated sample of ablated outputs, or at least a correlation check stratified by clause type, is required before the large completeness drops (−1.23) can be treated as definitive evidence that the stage is necessary.
minor comments (6)
  1. Section V-A / Table VI: NFR recall on GDPR is 0.18 and overall AI Act accuracy is 66.7%; the text notes this but the UI and user-study framing still present FR/NFR classification as a ready decision-support feature—add a stronger on-tool caveat and report confusion patterns more prominently.
  2. Section VI / RQ5: supporting-feature utility was assessed from screenshots rather than live interaction; state this limitation next to Table VII, not only in threats.
  3. Figures 2–3: prompts are abbreviated in the main text; ensure the replication package path and exact model snapshots (GPT-5 / GPT-5.4) are cited at first use so readers can reproduce without hunting the Zenodo record.
  4. Related Work (II-B): the qualitative differentiation from XTRAREG is clear; a short side-by-side table of scope (full document vs. two articles), explanation audience, and configuration burden would help readers place the contribution faster.
  5. Section III-A: the decision to append lettered sub-points to parent paragraphs and to drop recitals is reasonable but should cite the drafting convention more tightly and note any known cases where recitals have been treated as quasi-operative in compliance practice.
  6. Throughout: standardize “requirement-bearing” hyphenation and ensure all α values report the exact metric (identification vs. each quality dimension) in one place for quick reference.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical pipeline results are scored against independent human annotations and a held-out baseline, not forced by construction.

full rationale

This is an empirical requirements-engineering / NLP systems paper, not a first-principles derivation. Requirement-bearing clause identification is evaluated as binary classification against labels produced by six external RE annotators (Krippendorff’s α=0.81 on a shared subset) and is compared to a SetFit baseline trained and tested on a stratified held-out split; the reported macro F1 values (0.82/0.78) are therefore not redefined by any fitted parameter of the pipeline itself. Derived-requirement correctness, completeness, and clarity, and explanation quality, are human Likert ratings of model outputs, not quantities recovered from a fit. The ablation (full pipeline vs. no identification stage) uses an LLM-as-judge proxy that was separately correlated with the same human ratings (Spearman ρ=0.77–0.85); that validation does not make the quality drop by construction. The user study measures before/after comprehension and confidence on held-out practitioners. Related-work citations (including XTRAREG) are used for contrast, not as load-bearing uniqueness theorems by the same authors. No self-definitional loop, fitted-input-as-prediction, or ansatz-via-self-citation appears in the evaluation chain. Score 0 is therefore the correct outcome.

Assumptions & free parameters 0 free parameters · 4 assumptions · 2 invented entities

The work is an empirical systems/RE paper, not a formal derivation. Load-bearing premises are methodological: the operational definition of ‘requirement-bearing’, the adequacy of non-lawyer RE annotators as ground truth, and the assumption that zero-shot GPT-5 with the engineered prompts generalizes beyond the two EU regulations and the 30-clause Data Act development set. No free parameters are fitted to produce the headline F1 or Likert scores; the three-type cross-reference taxonomy and system-agnostic framing are design choices rather than invented physical entities.

assumptions (4)
  • domain assumption A clause is requirement-bearing iff it specifies, grants or constrains an observable system behavior, interaction or property needed for compliance (independent of pure legal obligation).
    Stated in Section III-B and used as the sole labeling criterion for RQ1 ground truth; alternatives exist for rights/exception clauses.
  • domain assumption Requirements engineers without formal legal training can produce reliable ground-truth labels and quality ratings for legal-to-software translation.
    Annotation protocol in IV-A1/A2; authors acknowledge the RE (not legal) interpretation in threats VII-B.
  • ad hoc to paper Zero-shot GPT-5 with the iteratively engineered prompts (Figs. 2–3) is sufficiently consistent for both identification and derivation across full regulatory documents.
    Model and temperature choices fixed by organizational infrastructure; no temperature control exposed; development set of 30 Data Act clauses used for prompt tuning.
  • domain assumption Recitals and lettered sub-points can be safely excluded or folded into parent paragraphs without loss of system-level requirements.
    Preprocessing step III-A, justified by EU drafting conventions.
invented entities (2)
  • Requirement-bearing clause identification as an explicit, independently evaluable RE pipeline stage
    purpose: Separate legal-obligation analysis from software-requirement implication so that only the latter enters derivation.
    Authors claim no prior work frames this distinction as a first-class stage; it is the key architectural novelty.
  • Three-type surface-level cross-reference taxonomy (DEFINITION / EXCEPTION / NEUTRAL)
    purpose: Provide lightweight typed links for traceability without deep legal interpretation.
    Simplified adaptation of prior legal-metadata schemes (Sleimi et al.); introduced in III-C.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Regulation to Requirements: An Automated Requirement Derivation and Explanation Pipeline." pith.science (2026). https://pith.science/paper/JYA5TU3W

@misc{pith2026260704448,
  author       = {Pith},
  title        = {Pith review of: From Regulation to Requirements: An Automated Requirement Derivation and Explanation Pipeline},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JYA5TU3W}},
  note         = {Machine review of arXiv:2607.04448}
}
read the original abstract

Ensuring software compliance with regulations such as the General Data Protection Regulation (GDPR) and the Artificial Intelligence Act (EU AI Act) poses a significant challenge, as requirements engineers must translate complex legal text into actionable software requirements - a process that remains largely manual and error-prone in practice. We present an automated regulation-to-requirements pipeline that identifies requirement-bearing clauses in regulatory documents and derives system-agnostic software requirements, accompanied by plain-language explanations, traceable to their legal sources. We evaluate the pipeline on the full clause sets of the GDPR (398 clauses) and the EU AI Act (574 clauses). For requirement-bearing clause identification, the approach achieves macro-averaged F1 scores of 0.82 and 0.78, respectively, outperforming a SetFit-based baseline. Human evaluation shows high completeness (4.60 and 4.45) and correctness (3.74 and 3.54) of derived requirements, while explanation clarity scores are near-ceiling (4.92 and 4.94) on a 1-5 scale. We implement the approach in Reg2Req, a publicly released tool that further supports requirement classification, use case seeding, cross-reference analysis, definition indexing, and a traceability matrix to operationalize regulatory compliance in practice. A user study with 25 practitioners shows that the plain-language explanations significantly improve comprehension of derived requirements and confidence in acting on them (p < 0.001), and that all participants would use Reg2Req as a starting point for deriving software requirements from a regulation.

Figures

Figures reproduced from arXiv: 2607.04448 by the authors.

Figure 1
Figure 1. provides an overview of our approach, which comprises a preprocessing step followed by two sequential stages. In the preprocessing step, a regulatory document is parsed into a structured, clause-level representation. The first stage, requirement-bearing clause identification, filters this clause set to those that imply required system-level behavior for regulatory compliance. Here, system refers to the software￾base… view at source ↗
Figure 3
Figure 3. Prompt for requirement derivation, explanation generation, and cross [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 5
Figure 5. Traceability view for requirement GDPR-1-3-R1 in Reg2Req. The [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The Requirements tab in Reg2Req, filtered to show FR-labelled [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 6
Figure 6. Figure 6: Definition indexing in Reg2Req. Defined terms appearing in a clause [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 3 linked inside Pith

  1. [1]

    Systematic mapping study on requirements engineering for regulatory compliance of software systems,

    O. Kosenkov, P. Elahidoost, T. Gorschek, J. Fischbach, D. Mendez, M. Unterkalmsteiner, D. Fucci, and R. Mohanani, “Systematic mapping study on requirements engineering for regulatory compliance of software systems,”Information and Software Technology, vol. 178, p. 107622, 2025

    2025

  2. [2]

    Ai-enabled regulatory change analysis of legal requirements,

    S. Abualhaija, M. Ceci, N. Sannier, D. Bianculli, L. C. Briand, D. Zet- zsche, and M. Bodellini, “Ai-enabled regulatory change analysis of legal requirements,” in2024 IEEE 32nd International Requirements Engineering Conference (RE). IEEE, 2024, pp. 5–17

    2024

  3. [3]

    Regulatory compliance,

    H. Y .-F. Lim, “Regulatory compliance,” inArtificial Intelligence. Ed- ward Elgar Publishing, 2025, pp. 149–177

    2025

  4. [4]

    European Parliament and Council of the European Union, 2016, “Reg- ulation (EU) 2016/679 of the European Parliament and of the Council on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation),” Official Journal of the European Union, Tech. Rep. L 119, 201...

    2016

  5. [5]

    Reg- ulation (EU) 2024/1689 of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (Artificial Intelli- gence Act),

    European Parliament and Council of the European Union, 2024, “Reg- ulation (EU) 2024/1689 of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (Artificial Intelli- gence Act),” Official Journal of the European Union, Tech. Rep. L 1689, 2024, oJ L, 12.7.2024

    2024

  6. [6]

    Reg- ulation (EU) 2023/2854 of the European Parliament and of the Council on harmonised rules on fair access to and use of data (Data Act),

    European Parliament and Council of the European Union, 2023, “Reg- ulation (EU) 2023/2854 of the European Parliament and of the Council on harmonised rules on fair access to and use of data (Data Act),” Official Journal of the European Union, Tech. Rep. L 2854, 2023, oJ L, 22.12.2023

    2023

  7. [7]

    Reg- ulation (EU) 2022/2065 of the European Parliament and of the Council on a single market for digital services (Digital Services Act),

    European Parliament and Council of the European Union, 2022, “Reg- ulation (EU) 2022/2065 of the European Parliament and of the Council on a single market for digital services (Digital Services Act),” Official Journal of the European Union, Tech. Rep. L 277, 2022, oJ L 277, 27.10.2022

    2022

  8. [8]

    California consumer privacy act of 2018, as amended by the California Privacy Rights Act of 2020,

    State of California, “California consumer privacy act of 2018, as amended by the California Privacy Rights Act of 2020,” 2020, cal. Civ. Code §§ 1798.100–1798.199.100

    2018

Show all 38 references
  1. [9]

    The digital personal data protection act, 2023,

    Government of India, “The digital personal data protection act, 2023,” No. 22 of 2023, Gazette of India Extraordinary, Part II, Section 1, 2023, received assent on 11 August 2023

    2023

  2. [10]

    Advancing requirements engineering through generative ai: Assessing the role of llms,

    C. Arora, J. Grundy, and M. Abdelrazek, “Advancing requirements engineering through generative ai: Assessing the role of llms,” in Generative AI for Effective Software Development. Springer, 2024, pp. 129–148

    2024

  3. [11]

    GDPR: A culture of non- compliance?

    noyb – European Center for Digital Rights, “GDPR: A culture of non- compliance?” Online, 2024, available: https://noyb.eu. Accessed: May 21, 2026

    2024

  4. [12]

    Analyzing regulatory rules for privacy and security requirements,

    T. D. Breaux and A. I. Ant ´on, “Analyzing regulatory rules for privacy and security requirements,”IEEE Transactions on Software Engineering, vol. 34, no. 1, pp. 5–20, 2008

    2008

  5. [13]

    An automated framework for the extraction of semantic legal metadata from legal texts,

    A. Sleimi, N. Sannier, M. Sabetzadeh, L. C. Briand, M. Ceci, and J. Dann, “An automated framework for the extraction of semantic legal metadata from legal texts,”Empirical Software Engineering, vol. 26, no. 3, p. 43, 2021

    2021

  6. [14]

    Automating the extraction of rights and obligations for regulatory compliance,

    N. Kiyavitskaya, N. Zeni, T. D. Breaux, A. I. Ant ´on, J. R. Cordy, L. Mich, and J. Mylopoulos, “Automating the extraction of rights and obligations for regulatory compliance,” inProceedings of the 27th Inter- national Conference on Conceptual Modeling (ER 2008), ser. Lecture ...

    2008

  7. [15]

    GaiusT: Supporting the extraction of rights and obligations for regula- tory compliance,

    N. Zeni, N. Kiyavitskaya, L. Mich, J. R. Cordy, and J. Mylopoulos, “GaiusT: Supporting the extraction of rights and obligations for regula- tory compliance,”Requirements Engineering, vol. 20, no. 1, pp. 1–22, 2015

    2015

  8. [16]

    Compliance checking between business processes and business contracts,

    G. Governatori, Z. Milosevic, and S. Sadiq, “Compliance checking between business processes and business contracts,” inProceedings of the 10th IEEE International Enterprise Distributed Object Computing Conference (EDOC). IEEE, 2006

    2006

  9. [17]

    PrOnto: Privacy ontology for legal reasoning about personal data processing,

    M. Palmirani, M. Martoni, A. Rossi, C. Bartolini, and L. Robaldo, “PrOnto: Privacy ontology for legal reasoning about personal data processing,” inProceedings of the 17th IFIP International Electronic Government Conference (EGOV). Springer, 2018

    2018

  10. [18]

    A meta-model for mod- elling law-compliant requirements,

    A. Siena, A. Perini, A. Susi, and J. Mylopoulos, “A meta-model for mod- elling law-compliant requirements,” inProceedings of the International Workshop on Requirements Engineering and Law (RELAW), 2008

    2008

  11. [19]

    Nomos 2: Making laws explicit in requirements engineering,

    S. Ingolfo, A. Siena, and J. Mylopoulos, “Nomos 2: Making laws explicit in requirements engineering,” inProceedings of the CAiSE Forum, 2013

    2013

  12. [20]

    Legal goal-oriented require- ment language (Legal GRL) for modeling regulations,

    S. Ghanavati, D. Amyot, and A. Rifaut, “Legal goal-oriented require- ment language (Legal GRL) for modeling regulations,” inProceedings of the International Workshop on Modeling in Software Engineering (MiSE), 2014

    2014

  13. [21]

    Gracenote.ai: Legal generative AI for regulatory compliance,

    J. Ioannidis, J. Harper, M. S. Quah, and D. Hunter, “Gracenote.ai: Legal generative AI for regulatory compliance,” inProceedings of the Legal AI and Intelligent Agents Workshop (LegalAIIA), 2023

    2023

  14. [22]

    Rethinking legal compliance automation: Opportunities with large language models,

    S. Hassani, M. Sabetzadeh, D. Amyot, and J. Liao, “Rethinking legal compliance automation: Opportunities with large language models,” in Proceedings of the 32nd IEEE International Requirements Engineering Conference (RE), 2024, pp. 432–440

    2024

  15. [23]

    LLM-assisted extraction of regulatory requirements: A case study on the GDPR,

    S. Abualhaija, M. Ceci, N. Sannier, D. Bianculli, S. Lannier, M. Siclari, O. V oordeckers, and S. Tosza, “LLM-assisted extraction of regulatory requirements: A case study on the GDPR,” inProceedings of the 33rd IEEE International Requirements Engineering Conference (RE), 2025

    2025

  16. [24]

    The law of recitals in European Commu- nity legislation,

    T. Klimas and J. Vaiciukaite, “The law of recitals in European Commu- nity legislation,”ILSA Journal of International and Comparative Law, vol. 15, 2008

    2008

  17. [25]

    Introducing GPT-5,

    OpenAI, “Introducing GPT-5,” https://openai.com/index/ introducing-gpt-5/, 2025, accessed: 2025

    2025

  18. [26]

    Phoenix and M

    J. Phoenix and M. Taylor,Prompt Engineering for Generative AI: Future-Proof Inputs for Reliable AI Outputs. O’Reilly Media, 2024

    2024

  19. [27]

    Krippendorff,Content Analysis: An Introduction to Its Methodology, 2nd ed

    K. Krippendorff,Content Analysis: An Introduction to Its Methodology, 2nd ed. Thousand Oaks, CA: Sage Publications, 2004

    2004

  20. [28]

    Efficient few-shot learning without prompts,

    L. Tunstall, N. Reimers, U. E. S. Jo, L. Bates, D. Korat, M. Wasserblat, and O. Pereg, “Efficient few-shot learning without prompts,” 2022. [Online]. Available: https://arxiv.org/abs/2209.11055

    2022 arXiv

  21. [29]

    Sentence-BERT: Sentence embeddings using siamese BERT-networks,

    N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using siamese BERT-networks,” inProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2019. [Online]. Available: https://arxiv.o...

    2019 arXiv

  22. [30]

    Scikit-learn: Machine learning in Python,

    F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, J. Vander- plas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duch- esnay, “Scikit-learn: Machine learning in Python,”Journal of Machine L...

    2011

  23. [31]

    GPT-5.4 model,

    OpenAI, “GPT-5.4 model,” https://developers.openai.com/api/docs/ models/gpt-5.4, 2025, model snapshot: gpt-5.4-2026-03-05. Knowledge cutoff: August 2025

    2025

  24. [32]

    Judging LLM-as-a-judge with MT-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging LLM-as-a-judge with MT-bench and chatbot arena,” inAdvances in Neural Information Processing Systems (NeurIPS),

  25. [33]

    Available: https://arxiv.org/abs/2306.05685

    [Online]. Available: https://arxiv.org/abs/2306.05685

    arXiv

  26. [34]

    The proof and measurement of association between two things,

    C. Spearman, “The proof and measurement of association between two things,”The American Journal of Psychology, vol. 15, no. 1, pp. 72–101, 1904

    1904

  27. [35]

    Require- ments classification with interpretable machine learning and dependency parsing,

    F. Dalpiaz, D. Dell’Anna, F. B. Aydemir, and S. C ¸ evikol, “Require- ments classification with interpretable machine learning and dependency parsing,” inProceedings of the 27th IEEE International Requirements Engineering Conference (RE). IEEE, 2019, pp. 142–152

    2019

  28. [36]

    Cockburn,Writing Effective Use Cases, ser

    A. Cockburn,Writing Effective Use Cases, ser. Agile Software Devel- opment Series. Boston, MA: Addison-Wesley, 2001

    2001

  29. [37]

    Individual comparisons by ranking methods,

    F. Wilcoxon, “Individual comparisons by ranking methods,”Biometrics Bulletin, vol. 1, no. 6, pp. 80–83, 1945

    1945

  30. [38]

    How is ChatGPT’s behavior changing over time?

    L. Chen, M. Zaharia, and J. Zou, “How is ChatGPT’s behavior changing over time?”Harvard Data Science Review, vol. 6, no. 2, 2024

    2024

Pith tools

Reviewed July 11, 2026 · model on record in the stance chip above.