Evaluating the Information Security Awareness of Smartphone Users
Pith reviewed 2026-05-25 17:02 UTC · model grok-4.3
The pith
ISA scores from mobile agents and network monitors correlate highly with users' success mitigating social engineering attacks on smartphones.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The authors present a framework that derives ISA scores from questionnaires, a mobile agent, and a network traffic monitor, then show in a study of 162 users that scores from the two objective sources correlate strongly with success against four classes of social engineering attacks while self-reported scores do not match observed behavior.
What carries the argument
The three-source framework that evaluates ISA for specific social engineering attack classes by combining subjective questionnaires with objective data from a mobile agent and network traffic monitor.
If this is right
- Self-reported user behavior differs significantly from actual observed behavior in security contexts.
- Objective monitoring data can produce ISA scores that predict real mitigation outcomes for specific attack classes.
- Evaluation methods must address differences between classes of social engineering attacks rather than using a single overall score.
- Frameworks relying solely on interviews or questionnaires are insufficient for accurate ISA assessment.
Where Pith is reading between the lines
- The approach could support real-time identification of vulnerable users within deployed mobile apps or enterprise networks.
- Security training programs could be validated or adjusted by measuring post-training changes in objective behavioral scores.
- The method opens the possibility of comparing ISA across device types or operating systems using the same objective tools.
Load-bearing premise
The four security challenges accurately resemble real-world social engineering attacks and the monitoring tools record representative behavior without study bias or user awareness changing their actions.
What would settle it
A replication study using different or more realistic challenges in which objective ISA scores show no correlation with participants' actual mitigation success.
Figures
read the original abstract
Information security awareness (ISA) is a practice focused on the set of skills, which help a user successfully mitigate a social engineering attack. Previous studies have presented various methods for evaluating the ISA of both PC and mobile users. These methods rely primarily on subjective data sources such as interviews, surveys, and questionnaires that are influenced by human interpretation and sincerity. Furthermore, previous methods for evaluating ISA did not address the differences between classes of social engineering attacks. In this paper, we present a novel framework designed for evaluating the ISA of smartphone users to specific social engineering attack classes. In addition to questionnaires, the proposed framework utilizes objective data sources: a mobile agent and a network traffic monitor; both of which are used to analyze the actual behavior of users. We empirically evaluated the ISA scores assessed from the three data sources (namely, the questionnaires, mobile agent, and network traffic monitor) by conducting a long-term user study involving 162 smartphone users. All participants were exposed to four different security challenges that resemble real-life social engineering attacks. These challenges were used to assess the ability of the proposed framework to derive a relevant ISA score. The results of our experiment show that: (1) the self-reported behavior of the users differs significantly from their actual behavior; and (2) ISA scores derived from data collected by the mobile agent or the network traffic monitor are highly correlated with the users' success in mitigating social engineering attacks.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes a framework for evaluating smartphone users' information security awareness (ISA) with respect to specific social engineering attack classes. It combines subjective questionnaires with objective behavioral data collected via a mobile agent and a network traffic monitor. A long-term study with 162 participants exposed to four security challenges reports two main findings: self-reported behavior differs significantly from observed behavior, and ISA scores derived from the mobile agent or network monitor are highly correlated with users' success at mitigating the attacks.
Significance. If the correlations are robust and the objective measures valid, the work offers a concrete improvement over purely subjective ISA assessment methods by linking scores directly to observed mitigation outcomes. The multi-source design and scale of the user study are positive features that could support more reliable security-awareness evaluation tools.
major comments (3)
- [§4] §4 (User Study): The paper provides insufficient detail on the exact procedure for deriving ISA scores from raw mobile-agent and network-monitor logs (e.g., which events are counted, how they are normalized, and any weighting scheme). This step is load-bearing for the central claim that these scores are “highly correlated” with mitigation success.
- [§5] §5 (Results and Statistical Analysis): No description is given of the correlation coefficient used, whether it is Pearson or Spearman, the handling of multiple comparisons across the three data sources, or any correction for participant attrition over the long-term study. These omissions prevent evaluation of the reported “high correlation” result.
- [§3.2] §3.2 (Security Challenges): The claim that the four challenges “resemble real-life social engineering attacks” lacks supporting validation data or pilot-study evidence. Without this, it is unclear whether the observed correlations generalize beyond the experimental setting or are artifacts of study-induced behavior.
minor comments (2)
- [Table 1] Table 1 and Figure 2: axis labels and legends are too small to read comfortably; consider enlarging or splitting into multiple panels.
- [§2] The related-work section cites several prior ISA questionnaires but does not compare their attack-class granularity with the four classes used here.
Simulated Author's Rebuttal
We thank the referee for the constructive comments. We address each major point below and will incorporate revisions to improve clarity and rigor.
read point-by-point responses
-
Referee: [§4] §4 (User Study): The paper provides insufficient detail on the exact procedure for deriving ISA scores from raw mobile-agent and network-monitor logs (e.g., which events are counted, how they are normalized, and any weighting scheme). This step is load-bearing for the central claim that these scores are “highly correlated” with mitigation success.
Authors: We agree that the current description of ISA score derivation is insufficient. The revised manuscript will expand §4 with a precise account of the events extracted from each log source, the normalization steps applied, and any weighting scheme used to produce the final scores. revision: yes
-
Referee: [§5] §5 (Results and Statistical Analysis): No description is given of the correlation coefficient used, whether it is Pearson or Spearman, the handling of multiple comparisons across the three data sources, or any correction for participant attrition over the long-term study. These omissions prevent evaluation of the reported “high correlation” result.
Authors: We acknowledge these statistical details are missing. In the revision we will specify the correlation method, describe the procedure for multiple comparisons, and report how attrition was addressed (including any sensitivity checks or corrections). revision: yes
-
Referee: [§3.2] §3.2 (Security Challenges): The claim that the four challenges “resemble real-life social engineering attacks” lacks supporting validation data or pilot-study evidence. Without this, it is unclear whether the observed correlations generalize beyond the experimental setting or are artifacts of study-induced behavior.
Authors: The challenges were modeled on documented real-world social-engineering vectors cited in the security literature. We did not conduct a separate pilot validation study. The revision will add explicit references to the source attack descriptions and include a limitations paragraph on generalizability. revision: partial
Circularity Check
No significant circularity; empirical correlation study
full rationale
The paper reports results from a long-term user study with 162 participants exposed to four security challenges. ISA scores are computed from three independent data sources (questionnaires, mobile agent, network monitor) and then correlated against observed mitigation success. No equations, fitted parameters renamed as predictions, self-citation chains, or ansatzes appear in the derivation of the central claims. The reported correlations are direct empirical outcomes rather than reductions to inputs by construction.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption The security challenges used resemble real-life social engineering attacks
Reference graph
Works this paper leans on
-
[1]
https://developer.android.com/guide/ topics/permissions/overview
Android permissions overview. https://developer.android.com/guide/ topics/permissions/overview
-
[2]
https://developers.google.com/gmail/api/
Gmail api. https://developers.google.com/gmail/api/
- [3]
-
[4]
https://www.trustedsec.com/ social-engineer-toolkit-set/
Social engineering toolkit. https://www.trustedsec.com/ social-engineer-toolkit-set/
- [5]
- [6]
-
[7]
The theory of planned behavior
Icek Ajzen. The theory of planned behavior. Organizational behavior and human decision processes , 50(2):179–211, 1991
work page 1991
-
[8]
Eirik Albrechtsen and Jan Hovden. Improving information security awareness and behaviour through dialogue, participation and collective reflection. an intervention study. Computers & Security, 29(4):432–445, 2010
work page 2010
-
[9]
Bluetooth R⃝ usage among students as an indicator of security awareness and feeling
Iosif Androulidakis and Gorazd Kandus. Bluetooth R⃝ usage among students as an indicator of security awareness and feeling. In ELMAR, 2011 Proceedings, pages 157–160. IEEE, 2011
work page 2011
-
[10]
Mobile phone security aware- ness and practices of students in budapest
Iosif Androulidakis and Gorazd Kandus. Mobile phone security aware- ness and practices of students in budapest. In Proceedings of the 6th International Conference on Digital Telecommunications , pages 17–22, 2011
work page 2011
-
[11]
A survey on saving personal data in the mobile phone
Iosif Androulidakis and Gorazd Kandus. A survey on saving personal data in the mobile phone. In Availability, Reliability and Security (ARES), 2011 Sixth International Conference on , pages 633–638. IEEE, 2011
work page 2011
-
[12]
A research model for investigating human behavior related to computer security
Kregg Aytes and Terry Conolly. A research model for investigating human behavior related to computer security. AMCIS 2003 Proceedings, page 260, 2003
work page 2003
-
[13]
Taxonomy of mobile users’ security awareness
Ron Bitton, Andrey Finkelshtein, Lior Sidi, Rami Puzis, Lior Rokach, and Asaf Shabtai. Taxonomy of mobile users’ security awareness. Computers & Security , 73:266–293, 2018
work page 2018
-
[14]
Research design and issues of validity
Marilynn B Brewer and William D Crano. Research design and issues of validity. Handbook of research methods in social and personality psychology, pages 3–16, 2000
work page 2000
-
[15]
A qualitative investigation of bank employee experi- ences of information security and phishing
Dan Conway, Ronnie Taib, Mitch Harris, Kun Yu, Shlomo Berkovsky, and Fang Chen. A qualitative investigation of bank employee experi- ences of information security and phishing. In Thirteenth Symposium on Usable Privacy and Security (SOUPS 2017), USENIX Association , pages 115–129, 2017
work page 2017
-
[16]
isam: an iphone stealth airborne malware
Dimitrios Damopoulos, Georgios Kambourakis, and Stefanos Gritzalis. isam: an iphone stealth airborne malware. In IFIP International Information Security Conference , pages 17–28. Springer, 2011
work page 2011
-
[17]
Phishing for user security awareness
Ronald C Dodge Jr, Curtis Carver, and Aaron J Ferguson. Phishing for user security awareness. computers & security , 26(1):73–80, 2007
work page 2007
-
[18]
Behavior ever follows intention?: A validation of the security behavior intentions scale (sebis)
Serge Egelman, Marian Harbach, and Eyal Peer. Behavior ever follows intention?: A validation of the security behavior intentions scale (sebis). In Proceedings of the 2016 CHI conference on human factors in computing systems, pages 5257–5261. ACM, 2016
work page 2016
-
[19]
Scaling the security wall: Developing a security behavior intentions scale (sebis)
Serge Egelman and Eyal Peer. Scaling the security wall: Developing a security behavior intentions scale (sebis). In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems , pages 2873–2882. ACM, 2015
work page 2015
-
[20]
Assessment of users’ information security behavior in smartphone networks
Mohammadjafar Esmaeili. Assessment of users’ information security behavior in smartphone networks . Eastern Michigan University, 2014
work page 2014
-
[21]
Official Journal of the European Union , L119:1–88, May 2016
Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation). Official Journal of the European Union , L119:1–88, May 2016
work page 2016
-
[22]
A survey of mobile malware in the wild
Adrienne Porter Felt, Matthew Finifter, Erika Chin, Steve Hanna, and David Wagner. A survey of mobile malware in the wild. In Proceedings of the 1st ACM workshop on Security and privacy in smartphones and mobile devices, pages 3–14. ACM, 2011
work page 2011
-
[23]
Using phishing experiments and scenario-based surveys to understand security behaviours in practice
Waldo Rocha Flores, Hannes Holm, Gustav Svensson, and Gran Er- icsson. Using phishing experiments and scenario-based surveys to understand security behaviours in practice. Information Management & Computer Security , 22(4):393–406, 2014
work page 2014
-
[24]
Security behavior ob- servatory: Infrastructure for long-term monitoring of client machines
Alain Forget, Saranga Komanduri, Alessandro Acquisti, Nicolas Christin, Lorrie F Cranor, and Rahul Telang. Security behavior ob- servatory: Infrastructure for long-term monitoring of client machines. Technical report, CARNEGIE-MELLON UNIV PITTSBURGH PA PITTSBURGH United States, 2014
work page 2014
-
[25]
Do or do not, there is no try: user engagement may not improve security outcomes
Alain Forget, Sarah Pearman, Jeremy Thomas, Alessandro Acquisti, Nicolas Christin, Lorrie Faith Cranor, Serge Egelman, and Marian Harbach. Do or do not, there is no try: user engagement may not improve security outcomes. In Twelfth Symposium on Usable Privacy and Security (SOUPS 2016) , pages 97–111, 2016
work page 2016
-
[26]
User modelling validation over the security awareness of digital natives
Vasileios Gkioulos, Gaute Wangen, and Sokratis K Katsikas. User modelling validation over the security awareness of digital natives. Future Internet, 9(3):32, 2017. 16
work page 2017
-
[27]
Security awareness of the digital natives
Vasileios Gkioulos, Gaute Wangen, Sokratis K Katsikas, George Kaval- lieratos, and Panayiotis Kotzanikolaou. Security awareness of the digital natives. Information, 8(2):42, 2017
work page 2017
-
[28]
Privacy Leakage in Mobile Computing: Tools, Methods, and Characteristics
Muhammad Haris, Hamed Haddadi, and Pan Hui. Privacy leakage in mobile computing: Tools, methods, and characteristics. arXiv preprint arXiv:1410.4978, 2014
work page internal anchor Pith review Pith/arXiv arXiv 2014
-
[29]
Mikko Hypponen. Malware goes mobile. Scientific American , 295(5):70–77, 2006
work page 2006
-
[30]
Tom N Jagatic, Nathaniel A Johnson, Markus Jakobsson, and Filippo Menczer. Social phishing. Communications of the ACM, 50(10):94–100, 2007
work page 2007
-
[31]
Phishing for phishing awareness
K Jansson and Rossouw von Solms. Phishing for phishing awareness. Behaviour & information technology , 32(6):584–593, 2013
work page 2013
-
[32]
Dissecting android malware: Characteri- zation and evolution
Xuxian Jiang and Yajin Zhou. Dissecting android malware: Characteri- zation and evolution. In 2012 IEEE Symposium on Security and Privacy, pages 95–109. IEEE, 2012
work page 2012
-
[33]
Using social psychology to implement security policies
ME Kabay, Bridgitt Robertson, Mani Akella, and DT Lang. Using social psychology to implement security policies. Computer Security Handbook, Sixth Edition , pages 50–1, 2002
work page 2002
-
[34]
Contextual usage patterns in smart- phone communication services
Juuso Karikoski and Tapio Soikkeli. Contextual usage patterns in smart- phone communication services. Personal and ubiquitous computing , 17(3):491–502, 2013
work page 2013
-
[35]
A conundrum of permis- sions: installing applications on an android smartphone
Patrick Gage Kelley, Sunny Consolvo, Lorrie Faith Cranor, Jaeyeon Jung, Norman Sadeh, and David Wetherall. A conundrum of permis- sions: installing applications on an android smartphone. In International Conference on Financial Cryptography and Data Security, pages 68–79. Springer, 2012
work page 2012
-
[36]
A framework for evaluating ict security awareness
Hennie A Kruger, Lynette Drevin, and Tjaart Steyn. A framework for evaluating ict security awareness. In ISSA, pages 1–11, 2006
work page 2006
-
[37]
A prototype for assessing information security awareness
Hennie A Kruger and Wayne D Kearney. A prototype for assessing information security awareness. computers & security , 25(4):289–296, 2006
work page 2006
-
[38]
Determinants of social desirability bias in sensitive surveys: a literature review
Ivar Krumpal. Determinants of social desirability bias in sensitive surveys: a literature review. Quality & Quantity, 47(4):2025–2047, 2013
work page 2025
-
[39]
School of phish: a real-world evaluation of anti-phishing training
Ponnurangam Kumaraguru, Justin Cranshaw, Alessandro Acquisti, Lor- rie Cranor, Jason Hong, Mary Ann Blair, and Theodore Pham. School of phish: a real-world evaluation of anti-phishing training. In Proceedings of the 5th Symposium on Usable Privacy and Security , page 3. ACM, 2009
work page 2009
-
[40]
Protecting people from phishing: the design and evaluation of an embedded training email system
Ponnurangam Kumaraguru, Yong Rhee, Alessandro Acquisti, Lor- rie Faith Cranor, Jason Hong, and Elizabeth Nunge. Protecting people from phishing: the design and evaluation of an embedded training email system. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 905–914. ACM, 2007
work page 2007
-
[41]
A survey on security for mobile devices
Mariantonietta La Polla, Fabio Martinelli, and Daniele Sgandurra. A survey on security for mobile devices. IEEE communications surveys & tutorials, 15(1):446–471, 2013
work page 2013
-
[42]
Agata McCormac, Dragana Calic, Kathryn Parsons, Tara Zwaans, Mar- cus Butavicius, and Malcolm Pattison. Test-retest reliability and internal consistency of the human aspects of information security questionnaire (hais-q). 2016
work page 2016
-
[43]
Delegate the smartphone user? security awareness in smartphone platforms
Alexios Mylonas, Anastasia Kastania, and Dimitris Gritzalis. Delegate the smartphone user? security awareness in smartphone platforms. Computers & Security , 34:47–66, 2013
work page 2013
-
[44]
As- sessing privacy risks in android: A user-centric approach
Alexios Mylonas, Marianthi Theoharidou, and Dimitris Gritzalis. As- sessing privacy risks in android: A user-centric approach. In Thomas Bauer, J ¨urgen Großmann, Fredrik Seehusen, Ketil Stølen, and Marc- Florian Wendland, editors, Risk Assessment and Risk-Driven Testing , pages 21–37, Cham, 2014. Springer International Publishing
work page 2014
-
[45]
Insights into user behavior in dealing with internet attacks
Kaan Onarlioglu, Utku Ozan Yilmaz, Engin Kirda, and Davide Balzarotti. Insights into user behavior in dealing with internet attacks. In NDSS, 2012
work page 2012
-
[46]
The human aspects of information security questionnaire (hais-q): two further validation studies
Kathryn Parsons, Dragana Calic, Malcolm Pattinson, Marcus Butavicius, Agata McCormac, and Tara Zwaans. The human aspects of information security questionnaire (hais-q): two further validation studies. Computers & Security, 66:40–51, 2017
work page 2017
-
[47]
Kathryn Parsons, Agata McCormac, Marcus Butavicius, Malcolm Pat- tinson, and Cate Jerram. Determining employee awareness using the human aspects of information security questionnaire (hais-q). Computers & Security, 42:165–176, 2014
work page 2014
-
[48]
Smartphone malware and its propagation modeling: A survey
Sancheng Peng, Shui Yu, and Aimin Yang. Smartphone malware and its propagation modeling: A survey. IEEE Communications Surveys & Tutorials, 16(2):925–941, 2014
work page 2014
-
[49]
Managing the risks of organizational accidents
James Reason. Managing the risks of organizational accidents . Rout- ledge, 2016
work page 2016
-
[50]
Asking for a friend: Evaluating response biases in security user studies
Elissa M Redmiles, Ziyun Zhu, Sean Kross, Dhruv Kuchhal, Tudor Dumitras, and Michelle L Mazurek. Asking for a friend: Evaluating response biases in security user studies. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security , pages 1238–1255. ACM, 2018
work page 2018
-
[51]
Decision making with the analytic hierarchy process
Thomas L Saaty. Decision making with the analytic hierarchy process. International journal of services sciences , 1(1):83–98, 2008
work page 2008
-
[52]
Internet security threat report 2018
Symantec. Internet security threat report 2018. URL https://www.symantec.com/content/dam/symantec/docs/reports/istr- 23-2018-en.pdf, 2018
work page 2018
-
[53]
Information security awareness: educating your users effectively
Mark E Thomson and Rossouw von Solms. Information security awareness: educating your users effectively. Information management & computer security , 6(4):167–173, 1998
work page 1998
-
[54]
Modifying smartphone user locking behavior
Dirk Van Bruggen, Shu Liu, Mitch Kajzer, Aaron Striegel, Charles R Crowell, and John D’Arcy. Modifying smartphone user locking behavior. In Proceedings of the Ninth Symposium on Usable Privacy and Security, page 10. ACM, 2013
work page 2013
-
[55]
All your droid are belong to us: A survey of current android attacks
Timothy Vidas, Daniel V otipka, and Nicolas Christin. All your droid are belong to us: A survey of current android attacks. In Woot, pages 81–90, 2011
work page 2011
-
[56]
Can people self-report security accurately?: Agreement between self-report and behavioral measures
Rick Wash, Emilee Rader, and Chris Fennell. Can people self-report security accurately?: Agreement between self-report and behavioral measures. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems , pages 2228–2232. ACM, 2017
work page 2017
-
[57]
Android permissions remys- tified: A field study on contextual integrity
Primal Wijesekera, Arjun Baokar, Ashkan Hosseini, Serge Egelman, David Wagner, and Konstantin Beznosov. Android permissions remys- tified: A field study on contextual integrity. In Proceedings of the 24th USENIX Conference on Security Symposium , pages 499–514, Berkeley, CA, USA, 2015. USENIX Association
work page 2015
-
[58]
F Yates. Lattice squares. The Journal of Agricultural Science, 30(4):672– 687, 1940. 17 APPENDICES A. The Security Questionnaire What is the likelihood of you to perform the following actions? Never Unlikely Medium Likelihood Very Likely Always
work page 1940
- [59]
- [60]
- [61]
- [62]
- [63]
- [64]
- [65]
- [66]
- [67]
- [68]
- [69]
- [70]
- [71]
- [72]
- [73]
- [74]
- [75]
- [76]
- [77]
- [78]
- [79]
- [80]
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.