Pith. sign in

REVIEW 1 major objections 1 minor 1 cited by

The EVerest dataset supplies requirements, architecture, and code linked by fine-grained security labels.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-26 07:44 UTC pith:YCIFSFPZ

load-bearing objection The paper ships a new public dataset spanning requirements, architecture, and code with security labels from an EV charging project, but the manual annotations lack any reported validation or agreement metrics. the 1 major comments →

arxiv 2606.23197 v1 pith:YCIFSFPZ submitted 2026-06-22 cs.SE cs.CR

The EVerest Dataset for Secure Software Engineering

Sophie Corallo , Debora Grupp , Dominik Fuch{\ss} , Jan Keim , Frederik Reiche , Tobias Hey , Anne Koziolek This is my paper
classification cs.SE cs.CR
keywords security datasetrequirements engineeringsoftware architecturesecure software engineeringtraceabilityelectric vehicle chargingsecurity verificationopen source
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

End-to-end security verification needs datasets spanning requirements, architecture, and code with detailed security labels. No prior resource offers this full combination. The paper creates the EVerest dataset from an open-source electric vehicle charging stack. It contains 84 security requirements with annotations, 1,445 security elements, trace links, architecture models, source code, and documentation. This allows new studies on security classification, entity recognition, trace linking, and verification at multiple stages.

Core claim

The paper presents the EVerest dataset as a multi-artifact resource based on the EVerest open-source stack for electric vehicle charging stations. It includes 84 manually elicited security requirements annotated with security objectives, 1,445 fine-grained security elements such as components, entities, data, data flows, and states along with acceptance windows, coreferences, and architectural trace links, as well as the software architecture model, source code, and natural language documentation. The dataset supports research on security requirements classification, named entity recognition, architectural trace linking, and design-time or code-level security verification.

What carries the argument

The EVerest dataset, which integrates annotated security requirements, fine-grained security elements, and trace links across requirements, architecture, and code from a single industrial system.

Load-bearing premise

The authors' manual elicitation of 84 requirements and annotation of 1,445 security elements accurately and comprehensively captures the security aspects of the EVerest system without significant bias or omission.

What would settle it

Release or discovery of another public dataset that already combines requirements, architecture, and code with a comparable number of fine-grained security labels and trace links.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Researchers can train and test models for security requirements classification using linked multi-artifact data.
  • Architectural trace linking and named entity recognition methods gain a complete industrial test case with security annotations.
  • Design-time and code-level security verification approaches can be evaluated end-to-end against real requirements and implementation.
  • The same construction process can be replicated on other open-source or industrial software stacks.
  • Public release of the dataset and the disclosure of a fixed weakness (CWE-1295) demonstrate practical utility beyond research use.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The linked structure may enable new automated pipelines that propagate security labels from requirements through to implementation.
  • Domains with safety-critical embedded systems could adapt the same multi-artifact labeling approach.
  • Community updates to the code base or annotations could keep the dataset current over time.
  • Wider use of such datasets might encourage more systematic security practices across the full development lifecycle.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 1 minor

Summary. The manuscript presents the EVerest dataset based on an open-source EV charging system. It includes 84 manually elicited security requirements annotated with security objectives, 1,445 fine-grained security elements (components, entities, data flows, states, etc.) with acceptance windows, coreferences, and trace links, as well as the architecture model, source code, and documentation. The dataset supports research in security requirements classification, named entity recognition, architectural trace linking, and security verification. A security weakness (CWE-1295) was identified and fixed during creation, and the dataset is publicly available.

Significance. If the annotations prove reliable, this dataset would be significant for filling the gap in multi-artifact security datasets with fine-grained labels spanning requirements to code. The combination enables new research directions in secure software engineering. The public release and the disclosure of a real weakness are positive aspects that enhance its practical value.

major comments (1)
  1. [Dataset construction process] The description of the manual elicitation of the 84 requirements and annotation of the 1,445 security elements does not report inter-annotator agreement, external expert validation, or bias mitigation steps. This is load-bearing for the central claim that the dataset supplies a usable combination of artifacts with accurate fine-grained security labels, as the utility for the claimed research uses depends on label quality.
minor comments (1)
  1. [Abstract] The abstract could briefly mention any steps taken to ensure annotation quality to preempt concerns about manual labeling.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the detailed review and constructive comment. We address the single major comment point-by-point below and agree that additional transparency on the annotation process is warranted.

read point-by-point responses

  1. Referee: [Dataset construction process] The description of the manual elicitation of the 84 requirements and annotation of the 1,445 security elements does not report inter-annotator agreement, external expert validation, or bias mitigation steps. This is load-bearing for the central claim that the dataset supplies a usable combination of artifacts with accurate fine-grained security labels, as the utility for the claimed research uses depends on label quality.

    Authors: We agree that the current manuscript provides insufficient detail on the annotation process and does not report inter-annotator agreement (IAA), external validation, or explicit bias mitigation steps. The elicitation and annotation were performed manually by the paper authors following a structured procedure based on the EVerest architecture documentation and source code, using security objectives drawn from established standards. Because only a single primary annotator was used for the majority of elements, formal IAA statistics are not available. We will revise the manuscript to expand the Dataset Construction section with: (1) the precise annotation guidelines and decision rules applied, (2) the steps taken for internal consistency checking (iterative author review of subsets), (3) an explicit statement of the limitation regarding IAA and external validation, and (4) a discussion of how this affects claimed research uses. These additions will allow readers to better evaluate label quality without overstating the validation performed. revision: yes

Circularity Check

0 steps flagged

No circularity: dataset paper with no derivations, equations, or self-citation chains

full rationale

The paper constructs and describes a new dataset through manual elicitation and annotation of requirements and security elements from an existing open-source project. No mathematical derivations, fitted parameters, predictions, or uniqueness theorems are claimed. The central contribution is the dataset itself, whose value rests on the described creation process rather than any reduction to prior self-citations or internal definitions. No load-bearing steps reduce to inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The contribution rests on the manual annotation process and the representativeness of the single chosen open-source project; no free parameters, mathematical axioms, or new invented entities are introduced.

axioms (1)
  • domain assumption Manual elicitation and annotation by domain experts produces reliable fine-grained security labels suitable for research use.
    The dataset is built entirely through this process as described in the abstract.

pith-pipeline@v0.9.1-grok · 5728 in / 1154 out tokens · 19111 ms · 2026-06-26T07:44:27.599413+00:00 · methodology

0 comments
read the original abstract

End-to-end security verification, from requirements through architecture to code, requires datasets that span all three artifact types with fine-grained security labels. No existing dataset provides this combination. We present the EVerest dataset, a multi-artifact resource based on EVerest, an industry-driven open-source software stack for electric vehicle charging stations. The dataset includes 84 manually elicited security requirements annotated with security objectives, 1,445 fine-grained security elements (components, entities, data, data flows, states, etc.), acceptance windows, coreferences, and architectural trace links, as well as the EVerest software architecture model, source code, and natural language documentation. It enables research on security requirements classification, named entity recognition, architectural trace linking, and design-time or code-level security verification. During dataset creation, a real security weakness (CWE-1295) was identified, disclosed to the project maintainers, and subsequently fixed. The dataset is publicly available. A short video is available at https://youtu.be/pnn1uqpomvQ.

Figures

Figures reproduced from arXiv: 2606.23197 by Anne Koziolek, Debora Grupp, Dominik Fuch{\ss}, Frederik Reiche, Jan Keim, Sophie Corallo, Tobias Hey.

Figure 1
Figure 1. Figure 1: EVerest dataset: documentation, labeled security [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Req. Elicitation: Coarse-grained requirements are [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. xDECAF: An Extensible Data Flow Diagram Analysis Framework for Information Security

    cs.SE 2026-07 conditional novelty 4.0

    xDECAF is an extensible, open-source framework for architecture-based data flow analysis with a constraint DSL, web editor, and a catalog of 26 example models for information security.

Reference graph

Works this paper leans on

27 extracted references · 16 canonical work pages · cited by 1 Pith paper · 1 internal anchor

  1. [1]

    Benjamin Arp, Felix Schwickerath, Alexander Vogt, Tom Hüller, Nils Niehues, and Nicolas Boltz. 2026. xDECAF: An Extensible Data Flow Diagram Analysis Frame- work for Information Security. InIEEE/ACM ASE. Association for Computing Machinery, New York, NY, USA

    2026

  2. [2]

    Yizheng Chen, Zhoujie Ding, Lamya Alowain, Xinyun Chen, and David Wagner

  3. [3]

    arXiv:2304.00409 [cs.CR] Dataset: https://github

    DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection. arXiv:2304.00409 [cs.CR] Dataset: https://github. com/wagner-group/diversevul, version: commit 50a2c18

    work page arXiv

  4. [4]

    nfr [Data set],

    Jane Cleland-Huang, Sepideh Mazrouee, Huang Liguo, and Dan Port. 2007.nfr. doi:10.5281/zenodo.268542

    work page doi:10.5281/zenodo.268542 2007

  5. [5]

    2026.The EVerest Dataset for Secure Software Engineering

    Sophie Corallo, Debora Grupp, Dominik Fuchß, Jan Keim, Frederik Reiche, Tobias Hey, and Anne Koziolek. 2026.The EVerest Dataset for Secure Software Engineering. doi:10.5281/zenodo.20793741

    work page doi:10.5281/zenodo.20793741 2026

  6. [6]

    2018.Requirements data sets (user stories)

    Fabiano Dalpiaz. 2018.Requirements data sets (user stories). doi:10.17632/ 7zbk8zsd8y.1

    2018

  7. [7]

    2019.explainable-re/RE-2019-Materials v0

    Fabiano Dalpiaz, Davide Dell’Anna, Fatma Başak Aydemir, and Sercan Çevikol. 2019.explainable-re/RE-2019-Materials v0. doi:10.5281/zenodo.3309669

    work page doi:10.5281/zenodo.3309669 2019

  8. [8]

    Henning Femmer, Frank Houdek, Max Unterbusch, and Andreas Vogelsang. 2025. QuRE Dataset. doi:10.5281/zenodo.16049822

    work page doi:10.5281/zenodo.16049822 2025

  9. [9]

    Alessio Ferrari, Giorgio Oronzo Spagnolo, and Stefania Gnesi. 2017. PURE: A Dataset of Public Requirements Documents. In2017 IEEE 25th International Requirements Engineering Conference (RE). 502–505. doi:10.1109/RE.2017.29

    work page doi:10.1109/re.2017.29 2017

  10. [10]

    LF Energy Foundation. 2024. EVerest. https://lfenergy.org/projects/everest/

    2024

  11. [11]

    Yifan Huang, Weisong Sun, and Yubin Qu. 2025. A Vulnerability Code Intent Summary Dataset. arXiv:2504.08180 [cs.SE] Dataset version: commit 830b944

    work page arXiv 2025

  12. [12]

    2024.Riaz-Dataset

    iambackend. 2024.Riaz-Dataset. https://github.com/iambackend/thesis_code version: commit acd33ee

    2024

  13. [13]

    2018.Information technology — Security techniques — Information security management systems — Overview and vocabulary

    ISO/IEC. 2018.Information technology — Security techniques — Information security management systems — Overview and vocabulary. Technical Report ISO/IEC 27000:2018-02(E). ISO/IEC

    2018

  14. [14]

    2021.ReqExp: BERT-based ML Model for Extracting Software Requirements

    Vladimir Ivanov, Andrey Sadovykh, Alexander Naumchev, Kirill Yakovlev, and Alessandra Bagnato. 2021.ReqExp: BERT-based ML Model for Extracting Software Requirements. doi:10.5281/zenodo.4630687

    work page doi:10.5281/zenodo.4630687 2021

  15. [15]

    Fatemeh Khayashi, Behnaz Jamasb, Reza Akbari, and Pirooz Shamsinejadbabaki

  16. [16]

    arXiv:2211.05286 [cs.SE]

    Deep Learning Methods for Software Requirement Classification: A Perfor- mance Study on the PURE dataset. arXiv:2211.05286 [cs.SE]

    work page arXiv

  17. [17]

    2021.SecReq

    Eric Knauss, Siv Hilde Houmb, Shareeful Islam, Jan Jürjens, and Kurt Schneider. 2021.SecReq. doi:10.5281/zenodo.4530183

    work page doi:10.5281/zenodo.4530183 2021

  18. [18]

    Roxanne E. Miller. 2009.The Quest for Software Requirements. MavenMark Books, Oconomowoc, WI, USA

    2009

  19. [19]

    Sven Peldszus, Frederik Reiche, Kevin Hermann, Sophie Corallo, Thorsten Berger, and Robert Heinrich. 2026. Can I Check What I Designed? Mapping Security Design DSLs to Code Analyzers. arXiv:2605.07814 [cs.CR]

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  20. [20]

    Khashan, Naif Alzahrani, Javed Rashid, and Arif Ur Rahman

    Kiramat Rahman, Anwar Ghani, Osama A. Khashan, Naif Alzahrani, Javed Rashid, and Arif Ur Rahman. 2025. Req2Vec: Learning Distributed Representation of Non-Functional Software Requirements.IEEE Access13 (2025), 202906–202918. doi:10.1109/ACCESS.2025.3637025

    work page doi:10.1109/access.2025.3637025 2025

  21. [21]

    2025.[Dataset] NICE: Non-Functional Requirements Identification, Classification, and Explanation Using Small Language Models

    Gokul Rejithkumar and Preethu Rose Anish. 2025.[Dataset] NICE: Non-Functional Requirements Identification, Classification, and Explanation Using Small Language Models. doi:10.5281/zenodo.14590935

    work page doi:10.5281/zenodo.14590935 2025

  22. [22]

    Reussner, Steffen Becker, Jens Happe, Robert Heinrich, Anne Koziolek, Heiko Koziolek, Max Kramer, and Klaus Krogmann (Eds.)

    Ralf H. Reussner, Steffen Becker, Jens Happe, Robert Heinrich, Anne Koziolek, Heiko Koziolek, Max Kramer, and Klaus Krogmann (Eds.). 2016.Modeling and Simulating Software Architectures – The Palladio Approach. MIT Press

    2016

  23. [23]

    Maria Riaz, Jason King, John Slankas, and Laurie Williams. 2014. Hidden in plain sight: Automatically identifying security requirements from natural language artifacts. In2014 IEEE 22nd International Requirements Engineering Conference (RE). 183–192. doi:10.1109/RE.2014.6912260 Original dataset no longer available, dataset: https://github.com/iambackend/R...

    work page doi:10.1109/re.2014.6912260 2014

  24. [24]

    Computer Science3. 2023. https://www.kaggle.com/datasets/computerscience3/ public-requirementspure-dataset/data last accessed: 04-10-2026

    2023

  25. [25]

    2019.NFRLocator

    John Slankas. 2019.NFRLocator. https://github.com/RealsearchGroup/ NFRLocator Dataset version: commit 53de7dc

    2019

  26. [26]

    Wentao Wang, Kavya Reddy Mahakala, Arushi Gupta, Nesrin Hussein, and Yinglin Wang. 2019. Data on security requirements in open-source software projects.Data in Brief25 (2019), 103590. doi:10.1016/j.dib.2018.12.029

    work page doi:10.1016/j.dib.2018.12.029 2019

  27. [27]

    Masooma Yousuf and M.Asger M.Asger. 2015. Comparison of Various Require- ments Elicitation Techniques.International Journal of Computer Applications116 (04 2015), 8–15. doi:10.5120/20322-2408

    work page doi:10.5120/20322-2408 2015